Graphs for enterprise architects

Jesus Barrasa @barrasadvMark Needham @markhneedham

Presentation Outline

‣ Context: what are graph DBs & Neo4j ?‣ Graph Dbs features matching EA’s demands‣ Cypher‣ Architectural patterns‣ Integration, adoption, partners and the

Neo4j ecosystem‣ Q&A

Case Studies

Introducing Neo4j

‣ Most popular graph database...

...

‣ ...and in top 20 DBMS platforms!

2000 2003 2007 2009 2011 2013 2014 20152012

GraphConnect, first conference for graph DBs

First Global 2000

Customer

Introduced first and only

declarative query language for

property graph

Published O’Reilly book

on Graph Databases

$11M Series A from Fidelity,

Sunstoneand Conor

$11M Series B from Fidelity,

Sunstoneand Conor

CommercialLeadership

First native

graph DB in 24/7

production

Invented property

graph model

Contributed first graph DB to open

source

$2.5M SeedRound from

Sunstone and Conor

Funding

Extended graph data model to labeled

property graph

150+ customers

50K+ monthlydownloads

500+ graph DB eventsworldwide

$20M Series C led by

Creandum, with Dawn and

existing investors

TechnicalLeadership

Neo4j: The Graph Database Leader

Graph?

Graph!

Demands on EA

Time to market

FlexibilityOperational Efficiency

Flexibility

‣ Data is the model ‣ Optional schema ‣ Multi-dimensional queries

Flexibility

‣ Data is the model ‣ Optional schema ‣ Multi-dimensional queries

Whiteboard model = physical model

Whiteboard model = physical model

Whiteboard model = physical model

Flexibility

‣ Data is the model ‣ Optional schema ‣ Multi-dimensional queries

Optional Schema

CREATE CONSTRAINT ON (book:Book)

ASSERT book.isbn IS UNIQUE

CREATE CONSTRAINT ON (book:Book)

ASSERT exists(book.isbn)

Flexibility

‣ Data is the model ‣ Optional schema ‣ Multi-dimensional queries

Operational efficiency

‣ Minutes to millisecond performance• Savings on hardware/software costs

‣ Horizontal + Vertical scaling‣ Mature DB

• In production for >10y, over 200 enterprise

subscription customers

‣ Professional support, services & training

Time to Market

‣ Agile/incremental delivery‣ Reduced Business-IT gap‣ Lower modeling impedance

Time to Market

‣ Agile/incremental delivery‣ Reduced Business-IT gap‣ Lower modeling impedance

Agile/incremental delivery

‣ Graph model is naturally adaptive• Build the model for the questions that we need to

answer• Evolve your model as new requirements arise

‣ No need to design the whole thing up front

Time to Market

‣ Agile/incremental delivery‣ Reduced Business-IT gap‣ Lower modeling impedance

Reduced Business-IT gap

‣ Intuitiveness of the graph model‣ Code easier to read, better communication

between teams‣ Faster ramp-up for new project members

Time to Market

‣ Agile/incremental delivery

‣ Reduced Business-IT gap

‣ Lower modeling impedance

Relational vs Graph models

Relational Model Graph Model

KNOWS

KNOWS

KNOWS

ANDREAS

TOBIAS

MICA

DELIA

Person FriendPerson-Friend

ANDREASDELIA

TOBIAS

MICA

Cypher: The Graph Query Language

LOVES

Dan Ann

Cypher: The Graph Query Language

MATCH (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} )

LOVES

Dan Ann

NODE NODERELATIONSHIP

Cypher: The Graph Query Language

MATCH (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} )

LOVES

Dan Ann

LABEL PROPERTY

NODE NODE

LABEL PROPERTY

RELATIONSHIP

Cypher: The Graph Query Language

Find all direct reports and how many people they manage, up to 3 levels down

Cypher: The Graph Query Language

Find all direct reports and how many people they manage, up to 3 levels down

SQL Query

Cypher: The Graph Query Language

MATCH (boss)-[:MANAGES*0..3]->(sub), (sub)-[:MANAGES*1..3]->(report)WHERE boss.name = “John Doe”RETURN sub.name AS Subordinate, count(report) AS Total

Find all direct reports and how many people they manage, up to 3 levels down

Cypher Query

SQL Query

Cypher: The Graph Query Language

‣ A query language designed for connectedness‣ Impact on projects

• Less time writing / optimising /debugging queries• Improved code maintainability

‣ Impact on teams• Improved communication• Faster ramp-up for new project members

Patterns in Neo4j architectures

‣ Primary Store• Reads + Write

‣ Secondary Store• Mostly read, scheduled batch updates

‣ ODS / MDS• Mostly read, scheduled batch updates + integration

logic

Primary Store case: Telenor

‣ Identity and access management problem. Need to compute resource authorization in real time

Was taking up to 20 min for large customers -> precalculation & cache -> stale data

Original architecture

500 requests / sec

Middleware services

Backend Backend Backend

Channel Channel Channel42 channels

35 systems

New architecture

500 requests / sec

Middleware services

Backend Backend Backend

Channel Channel Channel42 channels

35 systems

Did it work?

‣ Response times reduced to seconds and milliseconds

‣ Code maintainability improved. Access rules in Cypher as graph patterns.

"The Neo4j graph database gives us drastically improved performance and a simple language to query our connected data" – Sebastian Verheughe, Architect & Developer

Who uses Neo4j like this?

‣ UBS using it to control access to systems• Also migrated from RDBMS based solution• The Value

‣ Compliance‣ Faster onboarding‣ Real-time provisioning‣ Real-time deprovisioning

‣ AYI by Snap Interactive• Needed third degree connections in social network

Secondary Store case : Shutl/eBay

Wanted to come up with quotes for delivery slots.

It took 2 seconds to come up with a quote for a single delivery slot and they wanted to extend the service to show 20 potential delivery slots

Original architecture

Requests quote

Rails monolith Relational

Solution

Broke application up into services e.g. quoting, booking, feedback.

Key goal was to improve the speed of quoting and neo4j backed this service

New architecture

Requests quote

Quoting service Relational

Booking service

Feedback service

Nightly job to populate

ETL

Did it work?

‣ Quoting for 20 time windows down from 82,000 ms to 80ms

‣ Code complexity much reduced

Did it work?

‣ Quoting for 20 time windows down from 82,000 ms to 80ms

‣ Code complexity much reduced

“Our Neo4j solution is literally thousands of times faster than the prior MySQL solution, with queries that require 10-100 times less code. At the same time, Neo4j allowed us to add functionality that was previously not possible.”

Volker Pacher, Member of Technical Staff, eBay

Who uses Neo4j like this?

‣ World’s largest courier service• They noticed that they would not have been able to

deliver packages by Christmas 2013 !!• Hierarchical routing system (RDBMS backed)• Replaced it with Neo4j using lateral routing • Value:

‣ Minimized time‣ Maximized usage of road network

ODS / MDS case: Gov. Agency

‣ The graph integrates data from multiple sources• ODS: Additional operations on the integrated data

• MDS: Provide reference data to other operational

systems

‣ European government agencies

Traditional ODS/MDS architecture

RelationalODS

Data Source 1

Data Source 2

Semi / unstructured data source

ETL

Operational BI

RelationalMDS

EDW

Operational Systems

Graph based ODS/MDS architecture

ETL

Real time connected data analysis and

exploration

EDW

Operational Systems

Data Source 1

Data Source 2

Semi / unstructured data source

Who uses Neo4j like this?

‣ Schleich: Toy manufacturer. Germany

• PDM holding a MPD from: component manufacturers,

suppliers, different countries regulations• Rich and complex data used across entire value chain.• Previous arch: RDBMS backed unfit for Schleich’s

requirements for flexibility, perf and ease of operation‣ Adidas

• Metadata Store• Used to drive content personalization on their web site

Neo4j: Integration surfaces

‣ Cypher integration• JDBC• Transactional REST endpoint

‣ Java integration• Extensions / Procedures

‣ Import Tool‣ Batch Inserter

Tran

sact

iona

l

Non

Transactional

Adopting Neo4j

‣ Start with PoC‣ Bootcamp‣ First project

• an obviously graphy problem

‣ Multiple projects

Community vs Enterprise

Community vs Enterprise

The Neo4j Ecosystem

Graph ComputeMazerunner Spark Integrationhttps://github.com/neo4j-contrib/neo4j-mazerunner

Datastore IntegrationCassandra https://github.com/neo4j-contrib/neo4j-cassandra-connector

MongoDBhttps://github.com/neo4j-contrib/neo4j_doc_manager

Elastichttps://github.com/neo4j-contrib/neo4j-elasticsearch

Cloud HostingAWShttp://neo4j.com/developer/guide-cloud-deployment/

Graphene DBhttp://www.graphenedb.com/

Graph Storyhttp://graphstory.com/

Neo4j on Dockerhttps://hub.docker.com/r/neo4j/neo4j/

Graph Visualisation

Popoto.JShttp://www.popotojs.com/

Linkurious.jshttps://github.com/Linkurious/linkurious.js

Keylineshttp://cambridge-intelligence.com/keylines/neo4j

App Frameworks

Structrhttps://structr.org/

Graph Awarehttp://graphaware.com/products/

DB MigrationsLiquigraphhttp://fbiville.github.io/liquigraph

Business IntelligenceQlikView http://irregular-bi.tumblr.com/tagged/neo4j

DB IntrospectionSylva DBhttp://sylvadb.com/

NeoProfilerhttps://github.com/moxious/neoprofiler

‣ Training courses• Graph fundamentals• Graph modeling • SI training

‣ Meetups• introductory/modeling/use case specific

‣ Consulting

Getting the team up to speed

Partners

Lots of ways to learn Neo4j

Adoption by selected verticalsFinancialServices Communications Health &

Life SciencesHR &

RecruitingMedia &

PublishingSocialWeb

Industry & Logistics

Entertainment Consumer Retail Information ServicesBusiness Services

How customers use Neo4jNetwork &

Data Center Master DataManagementSocial Recom–

mendationsIdentity &

AccessSearch &Discovery GEO

Q&A

‣ Questions?• Jesus jesus.barrasa@neo4j.com• Mark mark.needham@neo4j.com