An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect,...
-
date post
19-Dec-2015 -
Category
Documents
-
view
220 -
download
0
Transcript of An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect,...
![Page 1: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/1.jpg)
An overview of InfiniteGraph,the distributed graph database.
Darren WoodChief Architect, InfiniteGraph
![Page 2: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/2.jpg)
For today’s discussion…
• InfiniteGraph– Background / History– Architecture– What we are working on
• Consistency Models• Challenges/Requirements of Distibution• Data model – Schema vs Schema-less
Copyright © InfiniteGraph
![Page 3: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/3.jpg)
Building on Objectivity/DB
• Derived from Objectivity/DB core– Distributed Object Database– Native core (broad platform support) with C++,
Java, C# and Python Bindings– Thousands of deployments, many are 24x7x365– Markets : VLDB, Data Fusion / Metadata, Complex
Object Models, Relationship Analytics
Copyright © InfiniteGraph
![Page 4: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/4.jpg)
Relationship Analytics Experience
• Relationship Analytics– Large part of Objy/DB Government business– Significant growth in the commercial space– Many OEM opportunities– Mostly based on graph theory– Hand coded algorithms
Copyright © InfiniteGraph
![Page 5: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/5.jpg)
Needed something more…
• Objectivity Provides– Powerful distributed object storage– Flexible class based persistence– Enterprise DB features
• But….– Lots of custom coding– Complex deployment– No generic algorithm framework
Copyright © InfiniteGraph
![Page 6: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/6.jpg)
Enter InfiniteGraph
• Simple Graph focused API• Automated distribution and deployment• Mostly configuration driven• Java class based persistence• Property model support• Asynchronous navigation• Indexing framework
Copyright © InfiniteGraph
![Page 7: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/7.jpg)
Some code…
Copyright © InfiniteGraph
Vertex alice = myGraph.addVertex(new Person(“Alice”)); Vertex bob = myGraph.addVertex(new Person(“Bob”)); Vertex carlos = myGraph.addVertex(new Person(“Carlos”)); Vertex charlie = myGraph.addVertex(new Person(“Charlie”));
alice.addEdge(new Meeting(“Denver”, “5-27-10”), bob);bob.addEdge(new Call(timestamp), carlos);carlos.addEdge(new Payment(100000.00), charlie);bob.addEdge(new Call(timestamp), charlie);
Alice Carlos CharlieBobMeets Calls Pays
Calls
![Page 8: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/8.jpg)
Basic Architecture
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
BlueprintsUser Apps
Objectivity/DB Distributed Database
Session / TX ManagementPlacement
![Page 9: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/9.jpg)
Copyright © InfiniteGraph
![Page 10: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/10.jpg)
Targeting Large Graphs
• Graphs grow quickly– Billions of phone calls / day in US– Emails, social media events, IP Traffic– Financial transactions
• Some analytics require navigation of large sections of the graph
• Must distribute data and go parallel
Copyright © InfiniteGraph
![Page 11: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/11.jpg)
Consistency Models
• Trading off full consistency for performance• Relax locking, allow stale reads• Gain 100x edge ingest rates• Great for “social graph” applications• These tend to have relaxed consistency
requirements• API allows choice per operation
Copyright © InfiniteGraph
![Page 12: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/12.jpg)
Pipelining
Copyright © InfiniteGraph
IG Core/API
ConfigurationNavigation Execution
Management Extensions
Session / TX Management
Placement(MDP)
Placement(Pipelining)
V1V1
V2V2
V3V3
E12E12
E23E23
Pipeline
Manager
Sta
ging
Con
tain
ers P
ipeline Containers
E(1->2)
E(3->1)
E(2->3)
E(2->1)
E(2->3)E(3->1)
E(1->2)
E(3->2)
E(1->2)
E(2->3)
E(3->1)
E(2->1)
E(2->3)
E(3->1)
E(3->2)
E(1->2)
![Page 13: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/13.jpg)
Distributing Navigation
• Graph algorithms naturally branch• Breaking up the process is relatively simple• Orchestrating it is more challenging
Copyright © InfiniteGraph
Alice
Carlos CharlieBobMeets Calls Pays
Dave EveChuckCallsLives With
Meets
![Page 14: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/14.jpg)
Distributed API
Application(s)
Partition 1 Partition 3Partition 2 Partition ...n
Processor Processor Processor Processor
Partitioned Graphs are Ugly
Copyright © InfiniteGraph
![Page 15: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/15.jpg)
Making it scale…
• A distributed data tier supports parallel IO• A distributed cache/processor tier for
processing “navlets”• Ability to deal with remote data reads (fast)• Navlets hit processor/cache instances based
on consistent hashing of their start vertex
Copyright © InfiniteGraph
![Page 16: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/16.jpg)
Flexible Data Models
• Schema isn’t for everyone– Great for relatively static models– Support for inheritance helps– So does auto-evolution
• Looking to support document style elements– Support for ad-hoc attributes– Ability to mix both approaches
Copyright © InfiniteGraph
![Page 17: An overview of InfiniteGraph, the distributed graph database. Darren Wood Chief Architect, InfiniteGraph.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d405503460f94a198f8/html5/thumbnails/17.jpg)
Other Projects
• Blueprints / Gremlin support• Native Visualizer and Management Console• Built in Qualifiers• More Indexing Enhancements• Algorithms• Large scale samples
Copyright © InfiniteGraph