Post on 15-Apr-2017
Apache GiraphLarge-scale graph processing done better
Data Mining Class
Sapienza, University of Rome
A. Y. 2016 - 2017
Basic concepts Let’s start Get our hands dirty
Hi!Simone Santacrocesantacroce.1542338@studenti.uniroma1.it
https://it.linkedin.com/in/simone-santacroce-272739134
Manuel Coppotellicoppotelli.1540732@studenti.uniroma1.it
https://it.linkedin.com/in/manuelcoppotelli
George Adrian Munteanumunteanu.1540833@studenti.uniroma1.it
https://it.linkedin.com/in/george-adrian-munteanu-707744134
Lorenzo Marconimarconi.1494505@studenti.uniroma1.it
https://www.linkedin.com/in/lorenzo-marconi-1a2580105
Antonio La Torrealatorre182@hotmail.it
https://www.linkedin.com/in/antonio-la-torre-768738134
Lucio Burliniburlini.1705432@studenti.uniroma1.it
https://www.linkedin.com/in/lucio-burlini-827739134
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Agenda
1 Basic concepts• Graphs in the real world• Challenges on graphs• MapReduce• Giraph
2 Let’s start• Out-Degree & In-Degree
3 Get our hands dirty• Simple PageRank
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Agenda
1 Basic concepts• Graphs in the real world• Challenges on graphs• MapReduce• Giraph
2 Let’s start• Out-Degree & In-Degree
3 Get our hands dirty• Simple PageRank
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Graphs 101
• Graph: representation of a setof objects G =< V ,E >
• Captures pairwise relationshipsbetween objects
• Can have directions, weights,. . .
Apache Giraph
Basic concepts Let’s start Get our hands dirty
A computer network
Apache Giraph
Basic concepts Let’s start Get our hands dirty
A road map
Apache Giraph
Basic concepts Let’s start Get our hands dirty
The web
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Social networks
• Both physical and Internet mediated
• Users are vertices
• Any kind of interaction generates edges
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Graph are huge!
∼ 50B pages
∼ 1.1B users
∼ 570M users
∼ 530M users
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Graph are nasty
• Graph needs processing
• Each vertex depends on its neighbors, recursively
• Recursive problems are nicely solved iteratively
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Graph are nasty
• Graph needs processing
• Each vertex depends on its neighbors, recursively
• Recursive problems are nicely solved iteratively
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Graph are nasty
• Graph needs processing
• Each vertex depends on its neighbors, recursively
• Recursive problems are nicely solved iteratively
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Graph are nasty
• Graph needs processing
• Each vertex depends on its neighbors, recursively
• Recursive problems are nicely solved iteratively
So what?
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Why not MapReduce?1
MapReduce is the current standard to manage big sets of data forintensive computing.
Repeat N times . . .1https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
MapReduce Drawbacks
• Each job is executed N times
• Job bootstrap
• Mappers send values and structure
• Extensive IO at input, shuffle & sort, output
Disk I/O and Job scheduling quickly dominate the algorithm
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s Pregel2
• Especially developed for large scale graph processing
• Intuitive API that let’s you “think like a vertex”
• Bulk Synchronous Parallel (BSP) as execution model
• Fault tolerance by checkpointing
2https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p135-malewicz.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s Pregel2
• Especially developed for large scale graph processing
• Intuitive API that let’s you “think like a vertex”
• Bulk Synchronous Parallel (BSP) as execution model
• Fault tolerance by checkpointing
2https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p135-malewicz.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s Pregel2
• Especially developed for large scale graph processing
• Intuitive API that let’s you “think like a vertex”
• Bulk Synchronous Parallel (BSP) as execution model
• Fault tolerance by checkpointing
2https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p135-malewicz.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s Pregel2
• Especially developed for large scale graph processing
• Intuitive API that let’s you “think like a vertex”
• Bulk Synchronous Parallel (BSP) as execution model
• Fault tolerance by checkpointing
2https://www.cs.cmu.edu/~pavlo/courses/fall2013/static/papers/p135-malewicz.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Giraph
Apache Giraph
Basic concepts Let’s start Get our hands dirty
The Story
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Think like a vertex
• Each vertex has an id, a value, a list of adjacent neighbors andcorresponding edge values
• Vertices implement algorithms by sending messages• Messages are delivered at the start of each superstep
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Bulk Synchronous Parallel (BSP)
• Master-Slave architecture
• Batch oriented processing
• Computation happens in-memory
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Advantages
• No locks: message-based communication
• No semaphores: global synchronization
• Iteration isolation: massively parallelizable
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Architecture
Single Map-only Job
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Jobs Schema
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Other things
Aggregators
• Mechanism for global communication and global computation
• Global value calculated in superstep t available in t + 1
• Pre-defined (e.g. sum, max, min) or user-definable functions3
Combiners
• User-defined function3 for messages before being sent or delivered
• Similar to Hadoop ones
• Saves on network or memory
Checkpointing
• Store work to disk at user-defined intervals (isn’t always evil)
• Restart on failure
3The function has to be both commutative and associative
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Other things
Aggregators
• Mechanism for global communication and global computation
• Global value calculated in superstep t available in t + 1
• Pre-defined (e.g. sum, max, min) or user-definable functions3
Combiners
• User-defined function3 for messages before being sent or delivered
• Similar to Hadoop ones
• Saves on network or memory
Checkpointing
• Store work to disk at user-defined intervals (isn’t always evil)
• Restart on failure
3The function has to be both commutative and associative
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Other things
Aggregators
• Mechanism for global communication and global computation
• Global value calculated in superstep t available in t + 1
• Pre-defined (e.g. sum, max, min) or user-definable functions3
Combiners
• User-defined function3 for messages before being sent or delivered
• Similar to Hadoop ones
• Saves on network or memory
Checkpointing
• Store work to disk at user-defined intervals (isn’t always evil)
• Restart on failure3The function has to be both commutative and associative
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Agenda
1 Basic concepts• Graphs in the real world• Challenges on graphs• MapReduce• Giraph
2 Let’s start• Out-Degree & In-Degree
3 Get our hands dirty• Simple PageRank
Apache Giraph
Basic concepts Let’s start Get our hands dirty
LongLongNullTextInputFormat
org.apache.giraph.io.formats.LongLongNullTextInputFormat
If there is ad edge from Node 1 to Node 2 thenNode 2 appears in the neighbor list of Node 1
<NODE1 ID> <SPACE> <NEIGHBOR1 ID> <SPACE> <NEIGHBOR2 ID> ...
<NODE2 ID> <SPACE> <NEIGHBOR1 ID> <SPACE> <NEIGHBOR2 ID> ...
...
Apache Giraph
Basic concepts Let’s start Get our hands dirty
IdWithValueTextOutputFormat
org.apache.giraph.io.formats.IdWithValueTextOutputFormat
For each node print the Node ID and the Node Value
<NODE1 ID> <TAB> <NODE1 VALUE>
<NODE2 ID> <TAB> <NODE2 VALUE>
...
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Demo
Demo code
https://github.com/manuelcoppotelli/giraph-demo
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Agenda
1 Basic concepts• Graphs in the real world• Challenges on graphs• MapReduce• Giraph
2 Let’s start• Out-Degree & In-Degree
3 Get our hands dirty• Simple PageRank
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s PageRank4
• The success factor of Google’s search engine
• A graph algorithm computing the “importance” of webpages
◦ Important pages have a lot of links from other important pages◦ Look at the structure of the underlying network
• Ability to conduct web scale graph processing
4http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s PageRank4
• The success factor of Google’s search engine• A graph algorithm computing the “importance” of webpages
◦ Important pages have a lot of links from other important pages◦ Look at the structure of the underlying network
• Ability to conduct web scale graph processing
4http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s PageRank4
• The success factor of Google’s search engine• A graph algorithm computing the “importance” of webpages
◦ Important pages have a lot of links from other important pages
◦ Look at the structure of the underlying network
• Ability to conduct web scale graph processing
4http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s PageRank4
• The success factor of Google’s search engine• A graph algorithm computing the “importance” of webpages
◦ Important pages have a lot of links from other important pages◦ Look at the structure of the underlying network
• Ability to conduct web scale graph processing
4http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Google’s PageRank4
• The success factor of Google’s search engine• A graph algorithm computing the “importance” of webpages
◦ Important pages have a lot of links from other important pages◦ Look at the structure of the underlying network
• Ability to conduct web scale graph processing
4http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Simple PageRank
• Recursive definition
PageRanki+1(v) =1 − d
N+ d ·
∑u→v
PageRanki (u)
O(u)
• Where:◦ d: damping factor; which percentage of the PageRank must be
transferred to the neighbors. Usually 0.85◦ N: total number of pages◦ O: out-degree; total number of link within a page
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Simple PageRank
• Recursive definition
PageRanki+1(v) =1 − d
N+ d ·
∑u→v
PageRanki (u)
O(u)
• Where:◦ d: damping factor; which percentage of the PageRank must be
transferred to the neighbors. Usually 0.85◦ N: total number of pages◦ O: out-degree; total number of link within a page
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Simple PageRank Example
1.0
1.0
1.0
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Simple PageRank Example
1.0
1.0
1.0
0.5
0.5
1
1
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Simple PageRank Example
1 · 0.85 + 0.1
5/3
0.5 · 0.85 + 0.15/3
1.5 · 0.85 + 0.15/3
0.5
0.5
1
1
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Simple PageRank Example
0.43
0.21
0.64
Apache Giraph
Basic concepts Let’s start Get our hands dirty
JsonLongDoubleFloatDoubleVertexInputFormat
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
Express both nodes and edges information using JSON arrays
[<vertex id>, <vertex value>,
[
[<dest vertex id>, <edge value>],
...
]
]
NoticeFore more in/out formats visit https://github.com/apache/giraph/tree/trunk/giraph-core/src/main/java/org/apache/giraph/io/formats
Apache Giraph
Basic concepts Let’s start Get our hands dirty
DemoDemo code
https://github.com/manuelcoppotelli/giraph-demo
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Q? & A!
Apache Giraph
Basic concepts Let’s start Get our hands dirty
Thank you for your attention
Contact us for any questions or problem
Demo code
https://github.com/manuelcoppotelli/giraph-demo
Homework
https://github.com/manuelcoppotelli/giraph-homework
Apache Giraph