GRAPH-BASED DATA INTEGRATION AND ANALYSIS WITH … · 2017-04-06 · Hadoop-based framework for...

www.scads.de

GRAPH-BASED DATA INTEGRATION AND ANALYSIS WITH GRADOOP

ERHARD RAHM,MARTIN JUNGHANNS, ANDRÉ PETERMANN, ERIC PEUKERT

Project period: 4 years (10/2014 – 09/2018), option for +3 more years after evaluation

Many involved research groups + many associated partners

Focal point for new research activities

Specialists from computer and domain sciences

NATIONAL BIG DATA COMPETENCE CENTER

2

Max Planck Institute of Molecular Cell Biology and Genetics

www.scads.de

COLLABORATION WITH APPLICATION SCIENCES

Big Data Life Cycle Management und Workflows

Efficient Big Data Architecture

Data Quality/Data Integration Visual AnalysisKnowledge Extraction

Life‐Sciences

Material Sciences

Digital Humanities

Environmental and Traffic Sciences

Business Data

Service‐Center

3www.scads.de

W.E. NagelE.Rahm

W. Lehner

K.-P. Fähnrich M. Bogdan C. Rother G. Scheuermann

S. Gumhold

P. Stadler G. Heyer

Life Sciences Gene‐Sequence Alignment (Consulting, Measurement, Parallelization), Myers‐Bit‐

Vector with dedicated hardware Parallelization of Gene‐Imputation‐Tasks UFZ‐Mass‐Spectromy‐Data WorkflowsDigital Humanities Text repository services for Digital Humanities (CTS) OCR‐Workflow on HPC (in progress)Environmental/Traffic sciences Deep Learning & structure recognition in spatial planningMaterial and Engineering Sciences Multi‐Scale VisualisationBusiness Data Condition Monitoring of Renewable Energy Power Plants Real‐Estate Data Fusion Deduplication of Person Data

APPLICATION SCIENCES

COMPUTER SCIENCE RESEARCH

Big Data Life Cycle Management und Workflows

Efficient Big Data Architecture

Data Quality/Data Integration Visual AnalysisKnowledge Extraction

Life‐Sciences

Material Sciences

Digital Humanities

Environmental and Traffic Sciences

Business Data

Service‐Center

5www.scads.de

W.E. NagelE.Rahm

W. Lehner

K.-P. Fähnrich M. Bogdan C. Rother G. Scheuermann

S. Gumhold

P. Stadler G. Heyer

Big Data Lifecycle and Workflows KNIME & HPC Integration

Data Quality and Integration Privacy‐Preserving Data Matching Hadoop‐based deduplication (Dedoop) Graph‐based data integration Holistic data integration

Visual Analysis Porting CV‐Algorithms to GPU Improve Visualization of Large Particle Data Muliti‐Scale Visualization for Big Data

Knowledge Extraction Deep Learning for Structure Recognition in Spatial Planning

Big Data Infrastructure Hardening computation infrastructure (Security) Flexible Cluster management Big Data Framework Execution & Monitoring on HPC Time Series Management and Forecasting Geotemporal Data Storage

Graph‐based data analysis (Gradoop)

COMPUTER SCIENCE RESEARCH (METHODS)

,

“GRAPHS ARE EVERYWHERE”

7

,


Alice

Bob

Eve

Dave

Carol

Mallory

Peggy

Trent

8

,


Alice

Bob

Eve

Dave

Carol

Mallory

Peggy

Trent

9

“GRAPHS ARE HETEROGENEOUS”

Alice

Bob

AC/DC

Dave

Carol

Mallory

Peggy

Metallica

∪ , ∪

10

0.2

0.28

0.26

0.33

0.25

0.26

“GRAPHS CAN BE ANALYZED”

Alice

Bob

AC/DC

Dave

Carol

Mallory

Peggy

Metallica

3.62.82

∪ , ∪

11

“GRAPHS CAN BE ANALYZED“

Assuming a social network1. Determine subgraph2. Find communities3. Filter communities4. Find common subgraph

12

all V challenges (volume, variety, velocity, veracity) ease-of-use high cost-effectiveness powerful but easy to use graph data model support for heterogeneous, schema-flexible vertices and edges support for collections of graphs (not only 1 graph) powerful graph operators

graph-based integration of many data sources

versioning and evolution (dynamic /temporal graphs)

interactive, declarative graph queries

scalable graph mining

comprehensive visualization support

GRAPH DATA ANALYTICS: REQUIREMENTS

13

COMPARISON

14

Graph Database SystemsNeo4j, OrientDB

data model rich graph models(PGM)

focus queries

query language yes

graph analytics no

scalability vertical

Workflows no

persistency yes

dynamic graphs / versioning

no

data integration no

visualization (yes)

COMPARISON (2)

15


Graph Processing Systems(Pregel, Giraph)


genericgraph models

focus queries analytic

query language yes no

graph analytics no yes

scalability vertical horizontal

Workflows no no

persistency yes no


no no

data integration no no

visualization (yes) no

COMPARISON (3)

16



Distributed DataflowSystems (Flink Gelly, Spark GraphX)


genericgraph models

genericgraph models

focus queries analytic analytic

query language yes no no

graph analytics no yes yes

scalability vertical horizontal horizontal

Workflows no no yes

persistency yes no no


no no no

data integration no no no

visualization (yes) no no

An end-to-end framework and research platform forefficient, distributed and domain independent graph

data management and analytics.

WHAT‘S MISSING?

17

Data Volume and Problem Complexity

Ease

-of-u

se

Graph Processing Systems

Graph Databases

Graph Dataflow Systems Gelly

18

Intro Graph Analytics Graph data Requirements Graph database vs graph processing systems

Gradoop architecture and data integration

Extended Property Graph Model (EPGM) Data organization and operators Implementation

Performance Evaluation

Summary/Outlook

AGENDA

19

Hadoop-based framework for graph data management and analysis persistent graph storage in scalable distributed store (Hbase) utilization of powerful dataflow system (Apache Flink) for parallel,

in-memory processing

Extended property graph data model (EPGM) operators on graphs and sets of (sub) graphs support for semantic graph queries and mining

Declarative specification of graph analysis workflows Graph Analytical Language - GrALa

End-to-end functionality graph-based data integration, data analysis and visualization

Open-source implementation: www.gradoop.org

GRADOOP CHARACTERISTICS

20

integrate data from one or more sources into a dedicated graph storewith common graph data model

definition of analytical workflows from operator algebra

result representation in meaningful way

END-TO-END GRAPH ANALYTICS

Data Integration Graph Analytics Visualization

21

HIGH LEVEL ARCHITECTURE

HDFS/YARNCluster

HBase Distributed Graph Store

Extended Property Graph Model

Flink Operator Implementations

Data Integration

Flink Operator Execution

Workflow Declaration

Visual

GrALa DSLRepresentation

Data flow

Control flow

Graph Analytics Representation

22

BIIIG: Business Intelligence on Integrated Instance Graphs

Heterogeneous data sources are integrated within an instance graph by preserving original relationships between data objects transactional and master data

Largely automated extraction of metadata and instance data andtransformation into graphs fusion of matching entities and relations

Extraction of subgraphs (business transaction graphs) related to interrelated business activities

Analysis of graphs/subgraphs with aggregation queries, pattern mining etc.

GRAPH-BASED BUSINESS INTELLIGENCE

23

DATA INTEGRATION AND ANALYSIS WORKFLOW

24

„Business Intelligence on Integrated Instance Graphs (BIIIG)“ (PVLDB 2014)

Business Transaction Graphs

(3) SubgraphIsolation

(2) Graph integration

Integrated Instance Graph

Domain expert

metadata

(1) Graph transformation

Dat

a So

urce

s

SAMPLE INSTANCE GRAPH

25

SCREENSHOT OF NEO4J IMPLEMENTATION

26





Summary/Outlook

AGENDA

27

includes PGM as special case

support for collections of logical graphs / subgraphs can be defined explicitly can be result of graph algorithms / operators

support for graph properties

powerful operators on both graphs and graph collections

Graph Analytical Language – GrALa domain-specific language (DSL) for EPGM flexible use of operators with application-specific UDFs plugin concept for graph mining algorithms

EXTENDED PROPERTY GRAPH MODEL (EPGM)

28

• Vertices and directed Edges

29

• Vertices and directed Edges• Logical Graphs

30

• Vertices and directed Edges• Logical Graphs• Identifiers

1 3

4

5

21 2

3

4

5

1

2

31

• Vertices and directed Edges• Logical Graphs• Identifiers• Type Labels

1 3

4

5

21 2

3

4

5Person Band

Person

Person

Band

likes likes

likes

knows

likes

1|Community

2|Community

32

• Vertices and directed Edges• Logical Graphs• Identifiers• Type Labels• Properties

1 3

4

5

21 2

3

4

5Personname : Aliceborn : 1984

Bandname : Metallicafounded : 1981

Personname : Bob

Personname : Eve

Bandname : AC/DCfounded : 1973

likessince : 2014

likessince : 2013

likessince : 2015

knows

likessince : 2014

1|Community|interest:Heavy Metal

2|Community|interest:Hard Rock

33

Operators

34

Operators

Unary BinaryAlgorithms

* auxiliary

Graph

Collection

LogicalG

raph

Aggregation

Pattern Matching

Transformation

Grouping Equality

Call *

Combination

Overlap

Exclusion

Equality

Union

IntersectionDifference

Gelly Library

BTG Extraction

Frequent Subgraphs

Limit

Selection

DistinctSort

Apply *Reduce *Call *

Adaptive Partitioning

Subgraph

35

Combination

Overlap

Exclusion

LogicalGraph graph3 = graph1.combine(graph2);LogicalGraph graph4 = graph1.overlap(graph2);LogicalGraph graph5 = graph1.exclude(graph2);

BASIC BINARY OPERATORS

1 34

52

3

1 2

1 34

52

12 4

5

3

36

udf = (graph => graph[‘vertexCount’] = graph.vertices.size())graph3 = graph3.aggregate(udf)

AGGREGATION

1 34

52

3

1 34

52

3 | vertexCount: 5

UDF

37

LogicalGraph graph4 = graph3.subgraph((vertex => vertex[:label] == ‘green’))LogicalGraph graph5 = graph3.subgraph((edge => edge[:label] == ‘blue’))LogicalGraph graph6 = graph3.subgraph(

(vertex => vertex[:label] == ‘green’),(edge => edge[:label] == ‘orange’))

SUBGRAPH

3

1 34

52

3

4

1 2

5

35

2UDF

UDF

UDF 3

6

1 2

38

GraphCollection collection = graph3.match(“(:Green)‐[:orange]‐>(:Orange)”);

PATTERN MATCHING

3

1 34

52 Pattern

4 5

1 34

2

Graph Collection

39

LogicalGraph grouped = graph3.groupBy([:label], // vertex keys[:label]) // edge keys

LogicalGraph grouped = graph3.groupBy([:label], [COUNT()], [:label], [MAX(‘a’)])

GROUPING

Keys

3

1 34

52

+Aggregate

3

a:23 a:84

a:42

a:12

1 34

52

a:13

a:21

4

count:2 count:3

max(a):42

max(a):84

max(a):13 max(a):21

6 7

4

6 7

40

SAMPLE GRAPH

[0] Tagname : Databases

[1] Tagname : Graphs

[2] Tagname : Hadoop

[3] Forumtitle : Graph Databases

[4] Forumtitle : Graph Processing

[5] Personname : Alicegender : fcity : Leipzigage : 23

[6] Personname : Bobgender : mcity : Leipzigage : 30

[7] Personname : Carolgender : fcity : Dresdenage : 30

[8] Personname : Davegender : mcity : Dresdenage : 42

[9] Personname : Evegender : fcity : Dresdenage : 35speaks : en

[10] Personname : Frankgender : mcity : Berlinage : 23IP: 169.32.1.3

0

1

2

3

4

5

6 7 8 9

10

11 12 13 14

15

16

17

18 19 20 21

22

23

knowssince : 2014

knowssince : 2014

knowssince : 2013

hasInterest

hasInterest hasInterest

hasInterest

hasModeratorhasModeratorhasMember hasMember

hasMember hasMember

hasTag hasTaghasTag hasTag

knowssince : 2013

knowssince : 2014

knowssince : 2014

knowssince : 2015

knowssince : 2015

knowssince : 2015

knowssince : 2013

GROUPING: TYPE LEVEL (SCHEMA GRAPH)

vertexGrKeys = [:label]edgeGrKeys = [:label]sumGraph = databaseGraph.groupBy(vertexGrKeys, [COUNT()], edgeGrKeys, [COUNT()])

[11] Person

count : 6

[12] Forum

count : 2

[13] Tag

count : 3

hasMembercount : 4

knowscount : 10

hasInterestcount : 4

hasTagcount : 4

hasModeratorcount : 2

24

26

28

27

25

42

personGraph = databaseGraph.subgraph((vertex => vertex[:label] == ‘Person’),(edge => edge[:label] == ‘knows’))

vertexGrKeys = [:label, “city”]edgeGrKeys = [:label]sumGraph = personGraph.groupBy(vertexGrKeys, [COUNT()], edgeGrKeys, [COUNT()])

GROUPING: PROPERTY-SPECIFIC

1 3

[11] Person

city : Leipzigcount : 2

[12] Person

city : Dresdencount : 3

[13] Person

city : Berlincount : 1

24

25

26

27

28

knowscount : 3

knowscount : 1 knows

count : 2

knowscount : 2

knowscount : 2

43

personGraph = databaseGraph.subgraph((vertex => vertex[:label] == ‘Person’),(edge => edge[:label] == ‘knows’))

vertexGrKeys = [:label, “city”]edgeGrKeys = [:label]sumGraph = personGraph.groupBy(vertexGrKeys, [COUNT()], edgeGrKeys, [COUNT()])

GROUPING: PROPERTY-SPECIFIC

1 3

[11] Person

city : Leipzigcount : 2

[12] Person

city : Dresdencount : 3

[13] Person

city : Berlincount : 1

24

25

26

27

28

knowscount : 3

knowscount : 1 knows

count : 2

knowscount : 2

knowscount : 2

44

GraphCollection filtered = collection.select((graph => graph[‘vertexCount’] > 4));

SELECTION

UDF

vertexCount > 4

1 | vertexCount: 5

2 | vertexCount: 4

0 23

41

5 7 86

1 | vertexCount: 5

0 23

41

45

GraphCollection frequentPatterns = collection.callForCollection(new TransactionalFSM(0.5))

CALL (E.G. FREQUENT SUBGRAPHS)

FSM

Threshold: 50%

1

0 1 23

4

5 6 78

9

1013

14

2

3

11 12

15 16

17 18

19 20

4

5

6

21 2322

25 2624

7

8

46

Implementation

47

GRAPH REPRESENTATION

Id Label Properties Graphs

Id Label Properties SourceId TargetId Graphs

EPGMGraphHead

EPGMVertex

EPGMEdge

Id Label Properties POJO

POJO

POJO

DataSet<EPGMGraphHead>

DataSet<EPGMVertex>

DataSet<EPGMEdge>


EPGMVertex

GradoopId := UUID128‐bit

String PropertyList := List<Property>Property := (String, PropertyValue)PropertyValue := byte[]

GradoopIdSet := Set<GradoopId>

48

Id Label Properties

1 Community {interest:Heavy Metal}

2 Community {interest:Hard Rock}


1 Person {name:Alice, born:1984} {1}

2 Band {name:Metallica,founded:1981} {1}

3 Person {name:Bob} {1,2}

4 Band {name:AC/DC,founded:1973} {2}

5 Person {name:Eve} {2}

Id Label Source Target Properties Graphs

1 likes 1 2 {since:2014} {1}

2 likes 3 2 {since:2013} {1}

3 likes 3 4 {since:2015} {2}

4 knows 3 5 {} {2}

5 likes 5 4 {since:2014} {2}

likessince : 2014

likessince : 20131 3

4

5

2



Personname : Aliceborn : 1984


Personname : Bob

Personname : Eve

Bandname : AC/DCfounded : 1973likes

since : 2015

knows


3

4

5

DataSet<EPGMGraphHead>

DataSet<EPGMVertex> DataSet<EPGMEdge>

GRAPH REPRESENTATION: EXAMPLE

49

// input: firstGraph (G[1]), secondGraph (G[2])

1: DataSet<GradoopId> graphId = secondGraph.getGraphHead()2: .map(new Id<G>());3: 4: DataSet<V> newVertices = firstGraph.getVertices()5: .filter(new NotInGraphBroadCast<V>())6: .withBroadcastSet(graphId, GRAPH_ID);7:8: DataSet<E> newEdges = firstGraph.getEdges()9: .filter(new NotInGraphBroadCast<E>())

10: .withBroadcastSet(graphId, GRAPH_ID)11: .join(newVertices)12: .where(new SourceId<E>().equalTo(new Id<V>())13: .with(new LeftSide<E, V>())14: .join(newVertices)15: .where(new TargetId<E>().equalTo(new Id<V>())16: .with(new LeftSide<E, V>());

Exclusion

OPERATOR IMPLEMENTATION

likessince : 2013


4

5

2



Personname : Aliceborn : 1984


Personname : Bob

Personname : Eve

Bandname : AC/DCfounded : 1973likes

since : 2015

knows


3

4

5

50

IMPLEMENTATION OF GRAPH GROUPING ( PROC. BTW2017)

GroupBy(1,2,3) +GC + GR* + MapAssign edges to groupsCompute aggregatesBuild super edges

Filter + MapExtract super vertex tuplesBuild super vertices

GroupBy(1) + GroupReduce*Assign vertices to groupsCompute aggregatesCreate super vertex tuplesForward updated group members

V

E

MapExtractattributes

Filter + Map Extract group membersReduce memory footprint

Join*Replace Source/TargetIdwith corresponding super vertex id

MapExtractattributes

*requires worker communication

V1 V2

V3

V‘

E1 E2 E‘

51

ITERATIVE COMPUTATION OF FREQUENT SUBGRAPHS

52

n‐edge3‐edge2‐edge1‐edge

result

collecting intermediate iteration results

searchspace

1‐edgeR C F

2‐edgeG R C F

3‐edgeG R C F

n‐edgeG R C F

G : grow frequent patternsR : report supported patternsC : count global frequencyF : filter by min frequency





Summary/Outlook

AGENDA

53

TEST WORKFLOW: SUMMARIZED COMMUNITIES

http://ldbcouncil.org/

1. Extract subgraph containing only Persons and knows relations

2. Transform Persons to necessary information

3. Find communities using Label Propagation

4. Aggregate vertex count for each community

5. Select communities with more than 50K users

6. Combine large communities to a single graph

7. Group graph by Persons location and gender

8. Aggregate vertex and edge count of grouped graph

54

TEST WORKFLOW: SUMMARIZED COMMUNITIES

https://git.io/vgozj

1. Extract subgraph containing only Persons

and knows relations

2. Transform Persons to necessary information

3. Find communities using Label Propagation

4. Aggregate vertex count for each community

5. Select communities with more than 50K users

6. Combine large communities to a single graph

7. Group graph by Persons location and gender

8. Aggregate vertex and edge count of grouped graph

55

BENCHMARK RESULTS

Dataset # Vertices # Edges

Graphalytics.1 61,613 2,026,082

Graphalytics.10 260,613 16,600,778

Graphalytics.100 1,695,613 147,437,275

Graphalytics.1000 12,775,613 1,363,747,260

Graphalytics.10000 90,025,613 10,872,109,028

16x Intel(R) Xeon(R) 2.50GHz (6 Cores) 16x 48 GB RAM 1 Gigabit Ethernet Hadoop 2.6.0 Flink 1.0-SNAPSHOT

0

200

400

600

800

1000

1200

1 2 4 8 16

Runtim

e [s]

Number of workers

RuntimeGraphalytics.100

1

2

4

8

16

1 2 4 8 16

Speedu

p

Number of workers

SpeedupGraphalytics.100 Linear

56

BENCHMARK RESULTS 2

Dataset # Vertices # Edges

Graphalytics.1 61,613 2,026,082

Graphalytics.10 260,613 16,600,778

Graphalytics.100 1,695,613 147,437,275

Graphalytics.1000 12,775,613 1,363,747,260

Graphalytics.10000 90,025,613 10,872,109,028

1

10

100

1000

10000

Runtim

e [s]

Datasets

16x Intel(R) Xeon(R) 2.50GHz (6 Cores) 16x 48 GB RAM 1 Gigabit Ethernet Hadoop 2.6.0 Flink 1.0-SNAPSHOT

57

EVALUATION OF GROUPING: SCALABILITY

Speedup for grouping on type Runtime for grouping on type

58





Summary/Outlook

AGENDA

59

Big Graph Analytics Hadoop-based graph processing frameworks based on generic graphs Spark/Flink: batch/streaming-oriented workflows (rather than interactive OLAP) graph collections not generally supported generally missing: graph-based data integration, built-in support for dynamic graph data

GraDoop (www.gradoop.org) open-source infrastructure for entire processing pipeline: graph acquisition, storage,

integration, transformation, analysis (queries + graph mining), visualization extended property graph model (EPGM) with powerful operators (e.g., grouping, pattern

matching) and support for graph collections leverages Hadoop ecosystem Apache HBase for permanent graph storage Apache Flink to implement operators

ongoing implementation

SUMMARY

60

COMPARISON

61



DistributedDataflow Systems (Flink Gelly, Spark GraphX)

data model rich graphmodels (PGM)

genericgraph models

genericgraph models

Extended PGM

focus queries analytic analytic analytic

query language yes no no no

graph analytics no yes yes yes

scalability vertical horizontal horizontal horizontal

Workflows no no yes yes

persistency yes no no yes


no no no no

data integration no no no (yes)

visualization (yes) no no limited

Graph-based data integration unified approach for knowledge graphs and regular data graphs holistic data integration for many sources

Graph analytics automatic optimization of analysis workflows optimized graph partitioning approaches visualization of graphs and analysis results interactive graph analytics dynamic graph data

OUTLOOK / CHALLENGES

62

M. Junghanns, A. Petermann, K. Gomez, E. Rahm: GRADOOP - Scalable Graph Data Management and Analytics with Hadoop. Tech. report (Arxiv), Univ. of Leipzig, 2015

M. Junghanns, A. Petermann, N. Teichmann, K. Gomez, E. Rahm: Analyzing Extended Property Graphs with Apache Flink. Proc. ACM SIGMOD workshop on Network Data Analytics (NDA), 2016

M. Junghanns, A. Petermann, M. Neumann, E. Rahm: Management and Analysis of Big Graph Data: Current Systems and Open Challenges. In: Big Data Handbook (eds.: S. Sakr, A. Zomaya) , Springer, 2017 (to appear)

M. Junghanns, A. Petermann, E. Rahm: Distributed Grouping of Property Graphs with GRADOOP. In: Proc. BTW, March 2017 (to appear)

A. Petermann, M. Junghanns, S. Kemper, K. Gomez, N.Teichmann, E. Rahm: Graph Mining for Complex Data Analytics. Proc. ICDM 2016 (Demo paper)

A. Petermann, M. Junghanns, R. Müller, E. Rahm: BIIIG : Enabling Business Intelligence with Integrated Instance Graphs. Proc. 5th Int. Workshop on Graph Data Management (GDM 2014)

A. Petermann, M. Junghanns, R. Müller, E. Rahm: Graph-based Data Integration and Business Intelligence with BIIIG. Proc. VLDB Conf., 2014

A. Petermann, M. Junghanns, R. Müller, E. Rahm: FoodBroker - Generating Synthetic Datasets for Graph-BasedBusiness Analytics. Proc. 5th Int. Workshop on Big Data Benchmarking (WBDB), 2014

A. Petermann; M. Junghanns: Scalable Business Intelligence with Graph Collections. it - Information Technology Special Issue: Big Data Analytics, 2016

REFERENCES

63

GRAPH-BASED DATA INTEGRATION AND ANALYSIS WITH … · 2017-04-06 · Hadoop-based framework for...

Documents

Transcript of GRAPH-BASED DATA INTEGRATION AND ANALYSIS WITH … · 2017-04-06 · Hadoop-based framework for...