Introduction to Riak

Click here to load reader

  • date post

    15-Jan-2015
  • Category

    Technology

  • view

    3.362
  • download

    5

Embed Size (px)

description

 

Transcript of Introduction to Riak

  • 1. Introduction to Riak Python hack-a-thon 2011.10.15Yosuke Hara {twiter_id, yosukehara} Software DeveloperSaturday, October 15, 111

2. Who am I ?Saturday, October 15, 112 3. twitter-id: yosukehara RIA Distributed StorageSaturday, October 15, 113 4. Whats Riak ?bashoSaturday, October 15, 11 4 5. KVS + ExtraSaturday, October 15, 11 5 6. Basho Bitcaskor LevelDB (Google, from 1.0)Saturday, October 15, 116 7. Architecture Erlang/OTP Runtime Client APIsHTTPProtocol BuffersErlang local client Request Coordinationgetputdelete map-reduceconsistent hashing handoff gossipmembership node-liveness bucketsRiak Core vnode master vnodes Workersstorage backend Riak KVSaturday, October 15, 117 8. The horizontal scaling bitsSaturday, October 15, 11 8 9. Inspired Amazons Dynamo (2007)KVS (Key-Value Storage) + ExtraMasterless, P2P-ReplicationConsistent HashingFailover - Quorums, Hinted handoffEventual ConsistencySaturday, October 15, 11 9 10. Inspired Amazons Dynamohttp://www.allthingsdistributed.com/2007/10/amazons_dynamo.html The reliability and scalability of a system is dependent on how its application state is managed. - Werner VogelsSaturday, October 15, 1110 11. KVS (Key-Value-Store) Simple operations - get, put, delete Value is mostly opaque (some metadata) Extras MapReduce (depends on Javascript) Links Full-text search (Optional - Riak Search) Secondary Indexes (from ver 1.0)Saturday, October 15, 11 11 12. P2P - MasterlessSaturday, October 15, 11 12 13. P2P - Masterless NOT SPOFSaturday, October 15, 1112 14. Consistent Hashing Virtual Node ID = SHA1( $KEY )Virtual Node Hash Ring 2^160 / 32Node-0Node-1Node-2Node-3Saturday, October 15, 11 13 15. Consistent Hashing Virtual Node ID = SHA1( $KEY ) Virtual Node Logical subdivision of the cluster Hash Ring 2^160 / 32Node-0Node-1Node-2Node-3Saturday, October 15, 11 13 16. For Parallelism# of VNodes == Maximum Concurrent Reqs For Rebalancing the ClusterSmallest block that can be shifted to a newnode For ResilianceThe system restarts failed VNodesSaturday, October 15, 1114 17. Quorum Every request contacts all replicas of key N : number of replicas (default 3) R : read quorum W : write quorumAble to adjust to Consistency LevelSaturday, October 15, 1115 18. Distributed Data(1) PUT File (3) if # of replicas needed for a successful Write > 2 # of replicas = 3 (4) return ok # of successful WRITEs = 2 Node-0 Node-1 Node-2 Node-3 (2) Replicate - Router on each nodeSaturday, October 15, 1116 19. Hinted Handoff X (1) PUT FileX (3) if # of replicas neededX for a successful Write > 2 # of replicas = 3 X(4) return okX# of successful WRITEs = 2X Node-0Node-1 X XNode-2(2) ReplicateNode-3 X (5) RecoverSaturday, October 15, 11 17 20. Eventual ConsistencySaturday, October 15, 1118 21. CAP Theorem Consistency AvailabilityPartition ToleranceSaturday, October 15, 1119 22. The doc-database bitsSaturday, October 15, 11 20 23. Store objects as JSON or any formatLink between objectsNo SQL - Query with Javascript Map-ReduceSaturday, October 15, 1121 24. The operations-friendly bitsSaturday, October 15, 1122 25. Web-shaped storageStore data in its original formatUsing HTTP Load balancing CachesNo special-nodes - grow horizontallySaturday, October 15, 1123 26. DEMO bashoSaturday, October 15, 11 24 27. PUTcurl -X PUT -H "x-riak-index-service_bin: news" -H "x-riak-index-range_int: 201110151300" -H "Content-Type: application/json" -d {"operation":"W", "key":"news/photo/20111015_small.jpg","service":"news","range":"201110151300"} http://127.0.0.1:8098/riak/news/log_001 KEY VALUE For Secondary IndexSaturday, October 15, 1125 28. GETcurl http://127.0.0.1:8098/demo/news/log_001 | python -mjson.tool{"key": "news/photo/20111015_small.jpg","operation": "W","range": "201110151300","service": "news"} KEY VALUE For Secondary IndexSaturday, October 15, 1126 29. Seconday Index (2I)curl http://127.0.0.1:8098/buckets/demo/index/service_bin/news| python -mjson.tool{ "keys": [ "log_001", "log_002" ]} KEY VALUE For Secondary IndexSaturday, October 15, 11 27 30. Map Reducecurl -H "Content-Type: application/json" http://127.0.0.1:8098/mapred --data @-