Cassandra at Umbel

download Cassandra at Umbel

of 52

  • date post

    03-Jul-2015
  • Category

    Software

  • view

    142
  • download

    1

Embed Size (px)

description

Umbel presents on their internal bitmap index that is backed by Cassandra.

Transcript of Cassandra at Umbel

  • 1. Cassandra at UmbelSeptember 20141

2. 2Travis TurnerChief Data Scientist, Co-Foundertravis@umbel.com 3. 3UmbelEmpower companies to convert people-baseddata into addressable, actionable relationships.! 4. 4Cassandra at Umbel1. Segmentation of people-based data2. Pilosa3. Cassandra Persistent Storage 5. SDKsCaptureS3Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 5 6. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 6 7. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 7 8. 8PilosaDecentralized, DistributedBitmap Index & Query Engine 9. SDKsCaptureS3PilosaUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 9 10. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 10 11. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 11 12. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 12 13. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 13 14. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 14 15. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 15 16. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 16 17. 1 1 0 1 0 1 1 0 0 0 1 1 0 1 1 1 0 0 0 011 0 1 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 101 1 0 1 1 0 0 1 0 1 1 0 1 0 1 0 0 1 0 011 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 1 0 101 0 1 0 1 1 0 1 0 1 1 0 1 1 1 0 1 1 0 011 0 1 1 0 1 0 0 0 0 0 1 1 1 0 1 1 1 1 101 1 1 0 1 0 1 1 1 0 1 0 1 1 0 1 01 1 1 11 1 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 0 1 100 0 0 0 0 0 00 0 0 0 0 0 01 1 1 1 1 1 1 1 1 1 1 0 101 1 1 1 1 1 1 0 1 1 1 01 1Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 17 18. 1 1 0 1 0 1 1 0 0 0 1 1 0 1 1 1 0 0 0 011 0 1 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 101 1 0 1 1 0 0 1 0 1 1 0 1 0 1 0 0 1 0 011 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 1 0 101 0 1 0 1 1 0 1 0 1 1 0 1 1 1 0 1 1 0 011 0 1 1 0 1 0 0 0 0 0 1 1 1 0 1 1 1 1 101 1 1 0 1 0 1 1 AND1 0 1 0 1 1 0 1 01 1 1 11 1 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 0 1 100 0 0 0 0 0 00 0 0 0 0 0 01 1 1 1 1 1 1 1 1 1 1 0 101 1 1 1 1 1 1 0 1 1 1 01 1Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 18 19. 1 1 0 1 0 1 1 0 0 0 1 1 0 1 1 1 0 0 0 011 0 1 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 101 1 0 1 1 0 0 1 0 1 1 0 1 0 1 0 0 1 0 011 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 1 0 101 0 1 0 1 1 0 1 0 1 1 0 1 1 1 0 1 1 0 011 0 1 1 0 1 0 0 0 0 0 1 1 1 0 1 1 1 1 101 1 1 0 1 0 1 1 OR1 0 1 0 1 1 0 1 01 1 1 11 1 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 0 1 100 0 0 0 0 0 00 0 0 0 0 0 01 1 1 1 1 1 1 1 1 1 1 0 101 1 1 1 1 1 1 0 1 1 1 01 1Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 19 20. 1 1 0 1 0 1 1 0 0 0 1 1 0 1 1 1 0 0 0 011 0 1 1 1 0 0 1 0 1 0 1 0 1 0 1 0 0 1 101 1 0 1 1 0 0 1 0 1 1 0 1 0 1 0 0 1 0 011 0 0 1 0 1 0 1 1 1 0 0 1 0 1 0 1 1 0 101 0 1 0 1 1 0 1 0 1 1 0 1 1 1 0 1 1 0 011 0 1 1 0 1 0 0 0 0 0 1 1 1 0 1 1 1 1 101 1 1 0 1 0 1 1 OR 1 0 1 0 NOT1 1 0 1 01 1 1 11 1 0 1 1 1 0 1 0 1 1 0 1 1 0 1 1 0 1 100 0 0 0 0 0 00 0 0 0 0 0 01 1 1 1 1 1 1 1 1 1 1 0 101 1 1 1 1 1 1 0 1 1 1 01 1Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 20 21. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 21 22. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 22 23. Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 23 24. 264 0Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 24 25. Slice 26465,536 bitsUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 25 26. Slice 26465,536 bitsFragmentUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 26 27. Slice 26465,536 bits0 1 2 3 31FragmentUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 27 28. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 28 29. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 29 30. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits) profile id: 150,0000 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)0010 0100 1001 1111 0000Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 30 31. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 216Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 31 32. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 2Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 32 33. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 0000Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 33 34. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 0000 211Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 34 35. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 35 36. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9001 1111 0000Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 36 37. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9001 1111 0000 26Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 37 38. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9001 1111 00000 0111 26= 7Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 38 39. Slice 26465,536 bits0 1 2 3 31ChunkFragment(2048 bits)0 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)profile id: 150,0000010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9001 1111 00000 0111 26= 711 0000 = 48Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 39 40. 40Cassandra Persistent Storagev 2.1 41. 0 1 2 3 31Chunk(2048 bits) profile id: 150,0000 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)0010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9001 1111 00000 0111 26= 711 0000 = 48CREATE TABLE bitmap(bitmap_id bigint,db varchar,frame varchar,slice int,chunkkey int,blockindex int,block bigint,PRIMARY KEY((bitmap_id, db, frame, slice), chunkkey, blockindex))Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 41 42. PQL: set(88, d, 0, 150000) cqlsh:pilosa> select * from bitmap;!bitmap_id | db | frame | slice | chunkkey | blockindex | block | filter-----------+------+-------+-------+----------+------------+-----------------+--------88 | test | d | 2 | -1 | 0 | 1 | 088 | test | d | 2 | 9 | 7 | 281474976710656 | 0!(2 rows)0 1 2 3 31Chunk(2048 bits) profile id: 150,0000 12 34 56 78 910 1112 1314 1516 1718 1920 2122 2324 2526 2728 2930 31Block (64 bits)0010 0100 1001 1111 0000 2160010 = 20100 1001 1111 00000 1001 211= 9001 1111 00000 0111 26= 711 0000 = 48Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 42 43. SliceUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 43 44. SliceFrameUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 44 45. FrameFrame Types! Default Time-based Top-nUmbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 45 46. fooDefaultset(88, foo, 150000)88150000Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 46 47. bar.tTime-basedset(88, bar.t, 150000, 2014-09-18T19:00)!fn(88, 2014-09-18T19:00) => [88.y, 88.m, 88.d, 88h]bitmap_id: 64 bitsid: 44 bitsdate/hour: 20 bits (60 years with hours, 2010-2070)Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 4788.y15000088.m88.d88.h 48. bar.tTime-basedget(88, bar.t, 2014-08-01T00:00, 2014-09-04T03:00)88.m.2014-0888.d.2014-09-0188.d.2014-09-0288.d.2014-09-0388.h.2014-09-04T0088.h.2014-09-04T0188.h.2014-09-04T02Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 4888.y15000088.m88.d88.h 49. brands.nTop-nset(88, brands.n, 150000)! only one fragment sorted by count configurable limit (50,000) compares count and loads from Cassandra88150000Umbel | Unmask your data. 2014 Umbel Corp. All Rights Reserved www.umbel.com 49 50. brands.nTop-ncqlsh:pilosa> select * from b