DryadInc: Reusing work in large-scale computations
Lucian Popa*+, Mihai Budiu+, Yuan Yu+, Michael Isard+
+ Microsoft Research Silicon Valley* UC Berkeley
Problem Statement
…
…
Goal: Reuse (part of) prior computations to: - Speed up the current job- Increase cluster throughput- Reduce energy and costs
Outputs
Inputs
Distributed Computation
Append-only data
Propose Two Approaches
1. IDEReuse IDEntical computations from the past(like make or memoization)
2. MERDo only incremental computation on the new data and MERge results with the previous ones(like patch)
Context• Implemented for Dryad
– Dryad Job = Computational DAG• Vertex: arbitrary computation + inputs/outputs• Edge: data flows
Simple Example: Record Count
C
I2
C
AAdd
Outputs
Inputs (partitions)
Count
I1
IDE – IDEntical ComputationRecord Count
C
I2
C
AAdd
Outputs
Inputs (partitions)
Count
I1
First executionDAG
IDE – IDEntical Computation
Second executionDAG
Record Count
C
I2
C
AAdd
Outputs
Inputs (partitions)
Count
I1 I3
C
New Input
IDE – IDEntical Computation
Second executionDAG
Record Count
C
I2
C
AAdd
Outputs
Inputs (partitions)
Count
I1 I3
C
Identical subDAG
IDE – IDEntical Computation
IDE Modified DAG
Replaced with Cached Data
Replace identical computational subDAG with edge data cached from previous execution
AAdd
Outputs
Inputs (partitions)
Count
I3
C
IDE – IDEntical Computation
IDE Modified DAG
Use DAG fingerprints to determine if computations are identical
AAdd
Outputs
Inputs (partitions)
Count
I3
C
Replace identical computational subDAG with edge data cached from previous execution
Semantic Knowledge Can Help
C
I2
C
A
I1
Reuse Output
Semantic Knowledge Can Help
C
I2
C
A
I1
C
I3
A Merge (Add)
Previous Output
Incremental DAG
MER – MERgeable Computation
C
I2
C
A
I1
C
I3
A Merge (Add)
Automatically Inferred
Automatically Built
User-specified
MER – MERgeable Computation
C
I2
C
A
I1
A
C
I2
C
A
I1 I3
C
Empty
Save to CacheIncremental DAG – Remove Old Inputs
Merge Vertex
IDE in practice
6 input DAG
IDE 6 (4+2) input DAG
9 input DAG
MER 9 (5+4) input DAG
MER in practice
EmptyEmpty
Evaluation – Running time
0.260.520.781.041.31.561.822.082.342.62.863.123.383.643.94.164.424.684.945.25.465.725.986.246.56.767.027.287.547.88.068.328.588.849.19.360000000000019.629.8810.1410.410.6610.9211.1811.4411.711.9612.2212.4812.741313.2613.5213.7814.0414.314.5614.8215.0815.3415.615.8616.1216.3816.6416.917.1617.4217.6817.9418.218.4618.7218.9819.2419.519.7620.0220.2820.5420.821.0621.3221.5821.8422.122.3622.6222.8823.1423.423.6623.9224.1824.4424.724.9625.2225.4825.742626.2626.5226.7827.0427.327.5627.8228.0828.3428.628.8629.1229.3829.6429.930.1630.4230.6830.9431.231.4631.7231.9832.2432.532.7633.0233.2833.5433.834.0634.3234.5834.8435.135.3635.6235.8836.1436.436.6636.9237.1837.4437.737.9638.2238.4838.743939.2639.5239.7840.0440.340.5640.8241.0841.3441.641.8642.1242.3842.6442.943.1643.4243.6843.9444.244.4644.7244.9845.2445.545.7646.0246.2846.5446.847.0647.3247.5847.8448.148.3648.6248.8849.1449.449.6649.9250.1850.4450.750.9651.2251.4851.745252.2652.5252.7853.0453.353.5653.8254.0854.3454.654.8655.1255.3855.6455.956.1656.4256.6856.9457.257.4657.7257.9858.2458.558.7659.0259.2859.5459.860.0660.3260.5860.8461.161.3661.6261.8862.1462.462.6662.9263.1863.4463.763.9664.2264.4864.746565.2665.5265.7866.0466.366.5666.8267.0867.3467.667.8668.1268.3868.6468.969.1669.4269.6869.9470.270.4670.7270.9871.2471.571.7672.0272.2872.5472.873.0673.3273.5873.8474.174.3674.6274.8875.1475.475.6675.9276.1876.4476.776.9677.2277.4877.747878.2678.5278.7879.0479.379.5679.8280.0880.3480.680.8681.1281.3881.6481.982.1682.4282.6882.9483.283.4683.7283.9884.2484.584.7685.0285.2885.5485.886.0686.3286.5886.8487.187.3687.6287.8888.1488.488.6688.9289.1889.4489.789.9690.2290.4890.749191.2691.5291.7892.0492.392.5692.8293.0893.3493.693.8694.1294.380
200
400
600
800
1000
1200
1400
1600
no cache identical
incremental
data size (GB)
Job
Run
ning
Tim
e (s
) No Cache
IDE
MER
Word Histogram Application – 8 nodes
Discussion
• MapReduce: just a particular case– IDE reuses the output of Mappers – MER requires combined Reduce function
• Combine IDE with MER: benefits don’t add up– IDE can be used for the incremental DAG at MER
• More semantic knowledge: further opportunities– Generate merge function automatically– Improve incremental DAG
Conclusions & Questions
• Problem: reuse work in distributed computations on append-only data
• Two methods:– INC – reuse IDEntical past computations
• No user effort– MER – MERge past results with new ones
• Small user effort, potentially larger gains
• Implemented for Dryad
Backup Slides
Architecture
Modify DAG before run
Dryad Job Manager
Cache Server
IDE/MER Rerun Logic
Update cache after runRUN
Use fingerprints to identify identical computational DAGs• Guarantee that both computation and data are unchanged
First DAG
Store to Cache: <hash, outputs>
Second DAG
IDE – IDEntical Computation
cached
Use a heuristic to identify the vertices to cache
First DAG Second DAG
IDE – IDEntical Computation
…
…Hash Distribute (Map)
Merge Sort
Count (Reduce)
Sort
Outputs
Inputsn
m
Word Histogram Application
Evaluation – Running time
0.260.520.781.041.31.561.822.082.342.62.863.123.383.643.94.164.424.684.945.25.465.725.986.246.56.767.027.287.547.88.068.328.588.849.19.360000000000019.629.8810.1410.410.6610.9211.1811.4411.711.9612.2212.4812.741313.2613.5213.7814.0414.314.5614.8215.0815.3415.615.8616.1216.3816.6416.917.1617.4217.6817.9418.218.4618.7218.9819.2419.519.7620.0220.2820.5420.821.0621.3221.5821.8422.122.3622.6222.8823.1423.423.6623.9224.1824.4424.724.9625.2225.4825.742626.2626.5226.7827.0427.327.5627.8228.0828.3428.628.8629.1229.3829.6429.930.1630.4230.6830.9431.231.4631.7231.9832.2432.532.7633.0233.2833.5433.834.0634.3234.5834.8435.135.3635.6235.8836.1436.436.6636.9237.1837.4437.737.9638.2238.4838.743939.2639.5239.7840.0440.340.5640.8241.0841.3441.641.8642.1242.3842.6442.943.1643.4243.6843.9444.244.4644.7244.9845.2445.545.7646.0246.2846.5446.847.0647.3247.5847.8448.148.3648.6248.8849.1449.449.6649.9250.1850.4450.750.9651.2251.4851.745252.2652.5252.7853.0453.353.5653.8254.0854.3454.654.8655.1255.3855.6455.956.1656.4256.6856.9457.257.4657.7257.9858.2458.558.7659.0259.2859.5459.860.0660.3260.5860.8461.161.3661.6261.8862.1462.462.6662.9263.1863.4463.763.9664.2264.4864.746565.2665.5265.7866.0466.366.5666.8267.0867.3467.667.8668.1268.3868.6468.969.1669.4269.6869.9470.270.4670.7270.9871.2471.571.7672.0272.2872.5472.873.0673.3273.5873.8474.174.3674.6274.8875.1475.475.6675.9276.1876.4476.776.9677.2277.4877.747878.2678.5278.7879.0479.379.5679.8280.0880.3480.680.8681.1281.3881.6481.982.1682.4282.6882.9483.283.4683.7283.9884.2484.584.7685.0285.2885.5485.886.0686.3286.5886.8487.187.3687.6287.8888.1488.488.6688.9289.1889.4489.789.9690.2290.4890.749191.2691.5291.7892.0492.392.5692.8293.0893.3493.693.8694.1294.380
200
400
600
800
1000
1200
1400
1600no cache
identical
incremental
identical 20ext
incremental 20ext
data size (GB)
Job
Run
ning
Tim
e (s
)
No CacheIDE (4 ext)MER (4 ext)IDE (20 ext)MER (20 ext)
Results similar if 20 more inputs appended ~ 1.2GB
Top Related