EMF Compare 2

Scaling to Millions

Mikaël Barbero 26 March 2013

EMF Compare 2.0

http

://id

ealis

tpro

paga

nda.

blog

spot

.fr/2

010/

04/im

agina

ry-fo

unda

tions

-mak

e-di

ffere

nce.

htm

l

Released mid-september 2012

Sponsored by Ericssonunder Polarsys and Modeling Platform

Working Group (MPWG)

Goal: Performance

http

://w

ww.

flick

r.com

/pho

tos/

9781

230@

N06

/227

0505

202

But also simpler API, better understanding of differences for end-users...

Should relate to number of differences, not to the size

of the model

Brand New UI Better integration with Team/Compare

Demo

Because it worths a thousand words

http://www.flickr.com/photos/viggum/2417430161/

How does it works? in a Nutshell

Match

Two Strategies

Two Strategies

Technical IDXMI:id, resource URI + URI fragment,

your own...

Two Strategies

Technical IDXMI:id, resource URI + URI fragment,

your own...

Content basedHeuristics, String distance

Differencing

http://www.flickr.com/photos/brianjmatis/3209203494/

Rationalization

Longest Common Subsequencehttp://www.youtube.com/watch?v=RUckZMzqUcw

Dependencies

http://www.flickr.com/photos/qilin/2262821682/

Requirements between differences

Package 1

State machine 1

Addition of ‘Package 1’ within ‘something’

Addition of ‘State Machine 1’ within ‘Package 1’

requires the merge of

Equivalences

http://www.bavarianfootballworks.com/2012/9/24/3384502/preview-fc-bayern-vs-vfl-wolfsburg

Equivalence between differences

Class 1

Property 1

Setting ‘Property 1’ to reference ‘Associtation 1’

Setting ‘Association 1’ to reference ‘Property 1’

the merge is equivalent to

Association 1

ass.ends

‘ass.’ and ‘ends’ are opposite references

Conflicts

http://www.theology21.com/2010/07/27/are-you-talking-to-me-the-art-of-misunderstanding-and-theological-double-talk/

Match

Difference

Equivalence

Requirement

Conflict

Summary

How to scale to Millions?

http://www.flickr.com/photos/somemixedstuff/2403249501/

A graph of EObjects

A «logical» EMF model

A graph of EObjects

In Resources

A

G

D

B

C

F

E

A graph of EObjects

Cross-resources links

A

G

D

B

C

F

E

A graph of Resources

A

G

D

B C

F

E

Left (local)

Right (remote)

Ancestor

A

G

D

B C

F

E

A

G

D

B C

F

E

A

G

D

B C

F

E

Ancestor

A

G

D

B C

F

E

A

G

D

B C

F

E

A

G

D

B C

F

E

Left (local)

Right (remote)

Ancestor

A

G

D

B C

F

E

A

G

D

B C

F

E

A

G

D

B C

F

E

Left (local)

Right (remote)

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

To be Loaded

To be Unloaded

Already browsed

Loading ThreadUnloading Thread

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

B G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

B G

A

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

B G

A

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G

A

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G

A

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G

A

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

A

A

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

A B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

A B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

G C

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

E D

B

A B

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

E D

B C G

A B C G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

E D

B C G

A B C G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

D

G

A B C G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

D

G

A B C G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

D

G

A B C G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

D

G

A B C G

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

D F

G E

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

D F

G E

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

F

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

F

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

F

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

F

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

F

A B C G E

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D F

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D F

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D F

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

F

A B C G E D F

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D F

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Multi-ThreadedConcurrent Resolve All

A B C

DE

F

G

A B C G E D F

Loading ThreadUnloading Thread

To be Loaded

To be Unloaded

Already browsed

Why concurrent loading?

The HDD is the bottleneck, right?

Scheduling is crucial

Between loading and unloading threads to avoid high-memory consumption (our goal)

Left (local)

Right (remote)

Ancestor

A

G

D

B C

F

E

A

G

D

B C

F

E

A

G

D

B C

F

E

Ancestor

A

G

D

B C

F

E

A

G

D

B C

F

E

A

G

D

B C

F

E

Left (local)

Right (remote)

Ancestor

A

G

D

B C

F

E

A

G

D

B C

F

E

A

G

D

B C

F

E

Left (local)

Right (remote)

Summary

Summary

Build the graph of resources

Summary

Build the graph of resources

Concurrent resolve all

Summary

Build the graph of resources

Find which resources has

changed

Concurrent resolve all

Summary

Build the graph of resources

Find which resources has

changed

Concurrent resolve all

Binary comparison of resources

Summary

Build the graph of resources

Find which resources has

changedCompare resource

with differences

Concurrent resolve all

Binary comparison of resources

Summary

Build the graph of resources

Find which resources has

changedCompare resource

with differences

Concurrent resolve all

Binary comparison of resources

Match, Diff, etc. without resolving

proxies

Kill two pigs with one (angry) bird

http

://ex

trem

ehdw

allpa

pers

.com

/wp-

cont

ent/u

ploa

ds/2

012/

08/a

ngry

-bird

s-w

allpa

per-h

d-.jp

g

Lightweight How light?

http://www.flickr.com/photos/roberts87/2646193801/

Fast How fast?

http://www.flickr.com/photos/themonnie/2495892146/

Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)

10

100

1000

10000

Small Medium Large

Number of fragments (resources)

10K

100K

1000K

10000K

Small Medium Large

Number of EObjects

10M

100M

1000M

Small Medium Large

Disk Usage (XMI files size)

Sample Models The numbers (x2 in 2-ways, x3 in 3-ways)

10

100

1000

10000

Small Medium Large

Number of fragments (resources)

10K

100K

1000K

10000K

Small Medium Large

Number of EObjects

10M

100M

1000M

Small Medium Large

Disk Usage (XMI files size)

The promised millions

CPU Usage

Some changes in 1 resource

30

60

90

120

1 10 100

Tim

e (s

ec)

number of changes

Small Medium Large

CPU Usage

1 change in some fragments

30

60

90

120

1 5 10 20

Tim

e (s

ec)

number of changed fragments

Small Medium Large

Memory Usage

Max Heap (-Xmx512m)

0

150

300

450

600

Small Medium Large

Max

Hea

p (M

iB)

What’s Next?

http://www.flickr.com/photos/exploratorium/4689346802/in/set-72157624060408411http://www.flickr.com/photos/exploratorium/4680503314/in/set-72157624060408411http://www.flickr.com/photos/exploratorium/4689347618/in/set-72157624060408411

Time distribution

0 %

25 %

50 %

75 %

100 %

SmallNominal

Large

Model Resolving Scope Narrowing MatchingDifferencing Post-Process

Index Cross-Resources Links

http://www.flickr.com/photos/manchesterlibrary/5425247171/

Eclipse Papyrus supporthttp://www.flickr.com/photos/manchesterlibrary/5425247171/

Porting Eclipse Papyrus support

A B C

DE

F

G

To be Loaded

To be Unloaded

C

Already browsed

Loading ThreadUnloading Thread

Protect the Integrity of the Model When committing, updating...

Generic Content Based Match Engine Decently Fast

http://nn.wikipedia.org/wiki/Fil:Postcards_and_magnifying_glass.jpg

Merge Wizard Help to Resolve Conflict

http://cdn.memegenerator.net/instances/400x/20451927.jpg

EMF Compare 2.1 With Kepler (Eclipse 4.3), June 2013

http://www.space.com/17376-sharing-the-light-of-two-suns.html

What you should remember

New UI

Blazing Fast and Lightweight

More to come ;)

http://www.slideshare.net/mikaelbarbero/http://www.eclipse.org/emf/compare/