Case Study: Real-time Recommendations with a Graph Database

Realtime Recommendations

with a Graph Database

Andreas Kollegger

#neo4j @akollegger

Wednesday, August 24, 11

Andreas Kollegger --[known_as]--> @akollegger --[talks_about]--> #neo4j

What’s the plan?

๏Graph Database 101

๏Case Study: Viadeo

๏Finding new Friends

๏Recommending new Sites

2



3



Obey!

3



Obey!

๏tweet using #neo4j

3



Obey!

๏tweet using #neo4j

๏I am @akollegger

3


NOSQL Categories

4

Size

Data Complexity


NOSQL Categories

4

Size

Data Complexity

Key-ValueStore


NOSQL Categories

4

Size BigTable

Clones

Data Complexity

Key-ValueStore


NOSQL Categories

4

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases


NOSQL Categories

4

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases

GraphDatabases


NOSQL Categories

4

Size BigTable

Clones

Data Complexity

Key-ValueStore

DocumentDatabases

GraphDatabases

90%of

usecases

(this is still billionsof nodes &

relationships)



Graph DB 101(Reading a graph)

5



A graph database?

6



A graph database?

6

๏no: not for storing charts & graphs, or vector artwork



A graph database?

6


๏yes: for storing data that is structured as a graph



A graph database?

6



• remember linked-list, tree?



A graph database?

6




• graphs are the generalized connected data structure



A graph database?

6





๏whiteboard friendly - no data mangling required



A graph database?

6





๏whiteboard friendly - no data mangling required

๏“A traditional relational database may tell you the average salary of everyone in this conference,

but a graph database will tell you who is most likely to buy you a beer.”



7

Q: What are graphs good for?



7

๏Recommendations

๏Business intelligence

๏Social computing

๏Geospatial

๏MDM

๏Systems management

๏Genealogy

๏Time series data

๏Product catalogue

๏Web analytics

๏Scientific computing (especially bioinformatics)

๏ Indexing your slow RDBMS

๏And much more!

Q: What are graphs good for?

A: highly-connected data



G=(V,E)

8

๏A (Graph) -records data in- (Nodes)

๏A (Graph) -records data in- (Relationships)

๏(Nodes) are -organized by- (Relationships)

๏(Nodes & Relationships) -have- (Properties)

๏ Instead of a Data Model, you have a Data Graph

๏Structure is defined by the Relationships you create

Graph

records data in

Nodes Relationships

Properties

have

organized by

have

records data in



Query a graph with a traversal

๏A (Traversal) -navigates- a (Graph)

๏A (Traversal) -identifies- (Paths)

๏(Paths) -order- (Nodes)

๏Traversals reveal localized data

๏Find data that is relevant to other data

9

Nodes Relationships

organized by

Traversal

identifies

Paths

order

navigatesGraph

records data in records data in



Indexes look-up Nodes or Relationships

๏An (Index) -maps to either-

(Nodes or Relationships)

๏An (Index) is a -special- (Traversal)

๏ Indexes find starting points for a traversal

๏Can be queried with simple constraints

10

Nodes Relationships

Properties

have

organized by

have

Index

maps from

maps to either maps to either

Traversalspecial



Neo4j is a Graph Database

๏A (Graph Database) -manages-

a (Graph) and related (Indexes)

๏A Graph Database:

• Full ACID Transactions

•High-Availability (read scaling)

• 32 Billion Nodes, 32 Billion Relationships, 64 Billion Properties

• Server with REST API

•Or, Embeddable as a Java Library

11

Graph Indexes

GraphDatabase

manages manages

lookup



Neo4j Features

๏Neo4j is stable

• In 24/7 operation since 2003

๏Neo4j is under active development

• open source, with a vibrant community

๏High performance graph operations

• high performance? like what? ...

12



Example: Social Network “path exists”๏~1,000 persons

๏average 50 friends per person

๏pathExists(a,b) limited to depth 4

๏caches warmed up to eliminate disk I/O

13

# persons query time

Relational database

Neo4j

Neo4j

1,000 2000ms

1,000 2ms

1,000,000 2ms

EmilAndreas

MarcusGabriella

Madeline

Peter

Jacob

Joanne



Case Study: Viadeo

14



Case Study: Viadeo๏Viadeo - 35 million user professional network

14




๏the past tech

14




๏the past tech

• recommendations running on MySQL as a batch job

14




๏the past tech


• took an hour, then success happened.. and it took two days

14




๏the past tech



๏now using Neo4j

14




๏the past tech



๏now using Neo4j

• recommendations run in real-time

14




๏the past tech



๏now using Neo4j


•what does a recommendation look like?

14




๏the past tech



๏now using Neo4j


•what does a recommendation look like?

๏Check out: http://www.youtube.com/watch?v=sLlbtoCfciE

14


http://www.youtube.com/watch?v=sLlbtoCfciE

http://www.youtube.com/watch?v=sLlbtoCfciE


A closer look at recommendations

15



Recommendations 2 ways...

16




๏Gremlin - graph scripting

16





•an algorithmic approach

16






๏Cypher - declarative querying

16







•a “pattern matching” approach

16








๏Stig - functional graph query

16








๏Stig - functional graph query

•ok, not yet...16



Gremlin

17



Gremlin๏a graph scripting DSL

17




๏part of the TinkerPop stack

17





๏try it out:

17





๏try it out:

• g - the graph itself

17





๏try it out:


• g.v(0) - node 0

17





๏try it out:


• g.v(0) - node 0

• g.v(0).in - nodes connected to Node 0

17





๏try it out:


• g.v(0) - node 0


• g.v(1).out(“KNOWS”) - nodes connected by “KNOWS”

17





๏try it out:


• g.v(0) - node 0



๏details about Gremlin: https://github.com/tinkerpop/gremlin

17


https://github.com/tinkerpop/gremlin





๏try it out:


• g.v(0) - node 0



๏details about Gremlin: https://github.com/tinkerpop/gremlin

๏props to @twarko for creating it

17





Cypher

18



Cypher๏a pattern-matching query language (SQL-ish)

18




๏developed in-house

18





๏try it out:

18





๏try it out:

• start a=(0) return a - get node 0

18





๏try it out:


• start a=(1) match (a)-->(b) return b - traverse from node 1

18





๏try it out:



• start a=(1) match (a)--()--(c) return c - friends of friends

18





๏try it out:




๏details about Cypher:

18





๏try it out:




๏details about Cypher:

• http://docs.neo4j.org/chunked/stable/cypher-query-lang.html

18


http://docs.neo4j.org/chunked/stable/cypher-query-lang.html

http://docs.neo4j.org/chunked/stable/cypher-query-lang.html


The Social Graphas a Data Graph

19



The Usual Social Graph, is also data

20

name:Andreas

employer:Neo Technology

worked_at

body: presenting at NoSQL Now! like right now!

status

name: Markoknows

url: http://www.amazon.com/Nikon-14-24mm-2-8G-Nikkor-Angle/dp/B000VDCTCI/

likes

body: mocking up a simplified social graph

status

url: http://www.imdb.com/title/tt1475582/

likes

knows

name: Peter

employer:Xilinx

worked_at



The Usual Social Graph, is also data

20

name:Andreas


worked_at


status

name: Markoknows


likes


status


likes

knows

name: Peter

employer:Xilinx

worked_at

me



with the Usual “status tweets”

21

name:Andreas


worked_at


status

name: Markoknows


likes


status


likes

knows

name: Peter

employer:Xilinx

worked_at



and the Usual Suspects

22

name:Andreas


worked_at


status

name: Markoknows


likes


status


likes

knows

name: Peter

employer:Xilinx

worked_at



and the Usual Business

23

name:Andreas


worked_at


status

name: Markoknows


likes


status


likes

knows

name: Peter

employer:Xilinx

worked_at



and the Usual Likes

24

name:Andreas


worked_at


status

name: Markoknows


likes


status


likes

knows

name: Peter

employer:Xilinx

worked_at



Making Friends

25



My Expanded Ego Network

26

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows



Me & My Friends

27

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows



Friends of my Friends

28

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows



Friend of a Friend of a Friend?

29

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows



Friend of a Friend of a Friend?

29

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows

Whatever.



Query for Friends of Friends

30



Query for Friends of Friends// Gremlin - andreas to friends to their friends, counted

30



Query for Friends of Friends// Gremlin - andreas to friends to their friends, countedandreas.out('KNOWS').out('KNOWS').name.groupCount(m)

30




// Cypher - match from andreas to friends to their friends

30




// Cypher - match from andreas to friends to their friendsstart andreas=(1) match (andreas)-[:KNOWS]->(friends)

30





-[:KNOWS]->(foafs) return foafs

30






// result: Emil, Stephen, Allison (& Peter?)

30






// result: Emil, Stephen, Allison (& Peter?)

30

Is that good enough?



Ranking my Friends’ Friends (intuitively)

31

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows




31

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows

1




31

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows

1

2




31

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows

1

2

3




31

name:Andreas

name: Markoknows

knows

name: Peter

name: Emil

knows

name: Stephen

knows

knows

name: Delia

knows

knows

name: Tiberius

knows

name: Allisonknows

knows

knows

1

2

3

?



Query for Ranked extended FoaF

32



Query for Ranked extended FoaF// Gremlin - a bit more complex

32



Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights

32



Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights// - weights decrease with each “hop” outward

32



Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights// - weights decrease with each “hop” outward// - extended hop, to include Stephen

32



Query for Ranked extended FoaF// Gremlin - a bit more complex// - requiring “memory” and weights// - weights decrease with each “hop” outward// - extended hop, to include Stephenandreas.out('KNOWS').aggregate(x).out('KNOWS').except(x).name

32




.groupCount(m){it}{it+1.0}.back(2)

32




.groupCount(m){it}{it+1.0}.back(2).out('KNOWS').except(x).name.groupCount(m){it}{it+0.5}

32





==>Emil=2.0

32





==>Emil=2.0==>Allison=2.0

32





==>Emil=2.0==>Allison=2.0==>Stephen=1.0

32





==>Emil=2.0==>Allison=2.0==>Stephen=1.0==>Tiberius=0.5

32



Like, like this

33



What else might I like?

34

name:Andreas

name: Markoknows


likes:+1


likes:+1

knows

name: Peter

url: http://theoatmeal.com/comics/state_web_summer/


url: http://browsertoolkit.com/fault-tolerance.png


url: http://lensbaby.com/optics-pinhole.php

likes:+1

likes:+2

likes:+1

likes:+1

likes:+2

likes:+3



Query for Things to Like

35



Query for Things to Like// Cypher - websites my friends like

35



Query for Things to Like// Cypher - websites my friends likestart andreas=(1) match (andreas)-[:KNOWS]->(friend)

35




-[:LIKES]->(website) return website.url

35





+------------------------------------------------+

35





+------------------------------------------------+| website.url |

35





+------------------------------------------------+| website.url |+------------------------------------------------+

35





+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png |

35


http://browsertoolkit.com/fault-tolerance.png





+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 |

35




http://www.imdb.com/title/tt0436992





+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php |

35






http://lensbaby.com/optics-pinhole.php





+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ |

35








http://theoatmeal.com/comics/state_web_summer/





+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 |

35















+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 || http://browsertoolkit.com/fault-tolerance.png |

35

















+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 || http://browsertoolkit.com/fault-tolerance.png |+------------------------------------------------+

35

















+------------------------------------------------+| website.url |+------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png || http://www.imdb.com/title/tt0436992 || http://lensbaby.com/optics-pinhole.php || http://theoatmeal.com/comics/state_web_summer/ || http://www.imdb.com/title/tt0133093 || http://browsertoolkit.com/fault-tolerance.png |+------------------------------------------------+6 rows, 4 ms

35
















36



Query for Things to Like// ordered by average rating

36



Query for Things to Like// ordered by average ratingstart andreas=(1) MATCH (andreas)-[:KNOWS]->(friend)

36




-[l:LIKES]->(website) return website.url, AVG(l.plus)

36




-[l:LIKES]->(website) return website.url, AVG(l.plus) ORDER BY AVG(l.plus) DESC

36





+--------------------------------------------------------------+

36





+--------------------------------------------------------------+| website.url | avg(l.plus) |

36





+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+

36





+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 |

36







+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 |

36









+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 |

36











+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 |

36













+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 || http://browsertoolkit.com/fault-tolerance.png | 1.0 |

36















+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 || http://browsertoolkit.com/fault-tolerance.png | 1.0 |+--------------------------------------------------------------+

36















+--------------------------------------------------------------+| website.url | avg(l.plus) |+--------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 || http://www.imdb.com/title/tt0436992 | 2.0 || http://www.imdb.com/title/tt0133093 | 2.0 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 || http://browsertoolkit.com/fault-tolerance.png | 1.0 |+--------------------------------------------------------------+5 rows, 5 ms

36














37



Query for Things to Like// ordered by number of ratings, then average rating

37



Query for Things to Like// ordered by number of ratings, then average ratingstart andreas=(1) MATCH (andreas)-[:KNOWS]->(friend)

37




-[l:LIKES]->(website) return website.url, AVG(l.plus), count(*) ORDER BY count(*) DESC, AVG(l.plus) DESC

37





+-------------------------------------------------------------------------+

37





+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |

37





+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+

37





+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 |

37







+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 |

37









+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 |

37











+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |

37













+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 | 1 |

37















+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 | 1 |+-------------------------------------------------------------------------+

37















+-------------------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-------------------------------------------------------------------------+| http://browsertoolkit.com/fault-tolerance.png | 1.0 | 2 || http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 || http://theoatmeal.com/comics/state_web_summer/ | 1.0 | 1 |+-------------------------------------------------------------------------+5 rows, 7 ms

37














38



Query for Things to Like// ordered, grouped, rated higher than 1

38



Query for Things to Like// ordered, grouped, rated higher than 1start andreas=(1) MATCH (andreas)-[:KNOWS]->(friend)

38




-[l:LIKES]->(website)

38




-[l:LIKES]->(website) WHERE l.plus>1

38





return website.url, AVG(l.plus), count(*)

38





return website.url, AVG(l.plus), count(*) ORDER BY count(*) DESC, AVG(l.plus) DESC

38






+-----------------------------------------------------------------+

38






+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |

38






+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+

38






+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 |

38








+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 |

38










+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |

38












+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |+-----------------------------------------------------------------+

38












+-----------------------------------------------------------------+| website.url | avg(l.plus) | count(*) |+-----------------------------------------------------------------+| http://lensbaby.com/optics-pinhole.php | 3.0 | 1 || http://www.imdb.com/title/tt0436992 | 2.0 | 1 || http://www.imdb.com/title/tt0133093 | 2.0 | 1 |+-----------------------------------------------------------------+3 rows, 91 ms

38









What else would you recommend? ๏Continue to elaborate the graph, and the algorithm

•more variables reveal more nuances in results

๏Rank influence of each user

• PageRank, Centrality, pick-an-algorithm

•Modulate weights by user rank

39



Questions?

๏learn more at http://neo4j.org

๏join the mailing list

๏join a meetup, or start one and we’ll come to you

40


http://neo4j.org

http://neo4j.org

Case Study: Real-time Recommendations with a Graph Database

Technology

Transcript of Case Study: Real-time Recommendations with a Graph Database