Hive, Presto, and Spark on

TPC-DS benchmark

Dongwon Kim, PhDSK Telecom

Contents• Experimental setup• Experimental results

[Experimental setup] TPC-DS dataset and query • Hive

• Entirely depend on github.com/hortonworks/hive-testbench• Distributed data generator

• A small dataset (100GB)• A large dataset (1TB)

• DDLs• External table declaration• Partitioned table declaration (ORC)

• 66 queries provided (out of 99 TPC-DS queries)

• Presto• Use hive-hadoop2 connector to read the same partitioned table• Use the same query

• Spark• Connected to Hive MetaStore to read the same partitioned table• Use the same query

[Experimental setup] Cluster setup• A single master node with 5 slave nodes

• Two 12-core processes with total 48 hyper threads• 128GB main memory• 10 HDDs

• Hadoop 2.7.3• Hive 2.1.1 + Tez 0.8.4

• A LLAP worker on a node uses 192 cores and 80GB• Presto 0.162

• A Presto worker on a node uses 192 cores and 80GB• distributed-joins-enabled = false

• Spark 2.0.2• 4 Spark executors on a node uses 192 cores and 80GB

[Experimental setup] Performance monitoring tool

• github.com/eastcirclek/swimlane-graphs• Hive/Presto/Spark task swimlane graph + Ganglia resource utilization

graph• To observe the main cause of performance bottleneck

[Experimental results] Characteristics of each engine• Hive

• Improve significantly through LLAP• Good for both small and large workload• Especially good for IO-bound workloads

• Spark• Improve CPU performance through Whole Stage Code Generation• Especially good for CPU-bound workloads• Does not outperform Hive and Presto for IO-bound workloads

• Presto• Pipelined execution to reduce unnecessary disk IOs• Good for simple queries• Works okay only when data is fit into memory

[Experimental results] Query execution time (100GB)

with query72 without query72Pairwise comparison reduction in

sum of running times Pairwise comparison reduction in sum of running times

Spark > Hive 26.3 %(1668s 1229s) Hive > Spark 19.8 %

(1143s 916s)

Hive > Presto 55.6 %(2797s 1241s) Hive > Presto 50.2 %

(982s 489s)

Spark > Presto

62.0 %(2932s 1114s)

Spark > Presto

5.2%(1116s 1057s)

Spark > Hive >>> Presto Hive > Spark >= Presto

Reversed

Gap reducedsignificantly

Hive with LLAP is good even for small workload* When comparing each pair of engines, I count queries that are completed by both of two engines.

[Experimental results] Query execution time (1TB)

with query72 without query72Pairwise comparison reduction in

sum of running times Pairwise comparison reduction in sum of running times

Hive > Spark 28.2 %(6445s 4625s) Hive > Spark 41.3 %

(6165s 3629s)

Hive > Presto 56.4 %(5567s 2426s) Hive > Presto 25.5 %

(1460s 1087s)

Spark > Presto

29.2 %(5685s 4026s)

Presto > Spark

58.6%(3812s 1578s)

Hive > Spark >>> Presto Hive > Presto > Spark

Reversed

* When comparing each pair of engines, I count queries that are completed by both of two engines.

Hive with LLAP is good for large, I/O-bound workloadPresto works okay if data fit into memory

[Experimental results] Curse of Query72 on Hive and Presto

72 89 43 63 19 3 51 52 42 55 82 29 17 25 49 39 91 40 21 13 12 73 96 48 20 85 34 84 79 32 7 26 45 27 46 88 68 15 97 92 93 66 87 24 28 56 71 83 60 76 31 64 740

200400600800

100012001400160018002000

Presto Spark

100GB dataset

72 89 43 63 19 3 42 52 55 25 49 29 17 75 82 40 88 31 13 71 91 56 85 60 68 28 46 48 26 7 51 66 21 34 27 20 45 12 87 73 79 15 32 96 84 76 39 93 92 970

200400600800

100012001400160018002000

Presto Hive

72226751979295823993219496847312187932 3 9120439852428963451555344827 7 269046681319858740667628885060291756587154254931800

200400600800

100012001400160018002000

Spark Hive

72 75 91 49 13 26 88 71 40 66 56 60 68 7 31 27 48 79 87 46 45 73 15 34 84 20 21 12 39 32 96 76 28 51 92 85 930

5001000150020002500300035004000

Presto Hive

726782929795225121122015261996392784 3 131828434552 7 344842465532738987636890799176664071935094568554605825173129884974800

5001000150020002500300035004000

Spark Hive

72 26 13 91 21 15 20 12 27 7 84 39 45 96 48 46 34 40 71 68 66 73 87 92 97 79 32 51 28 76 85 56 93 83 60 24 49 88 31 740

5001000150020002500300035004000

Presto Spark

[Experimental results] Curse of Query72 on Hive and Presto 1TB dataset

[Experimental results] Curse of Query72 on Presto and Hive

Presto SparkHive

High CPU utilizationfor a long time(looks like “plateau”)

No plateau observed thanks to

WholeStageCodeGen

plateau

w/ WholeStageCodeGen w/o WholeStageCode-Gen

CPU

Network

Disk

[Experimental results] Whole Stage Code Generation

Presto Hive

Without WholeStageCodeGeneration,“plateau” is observed like Presto and Hive

10th stage takes much longer without WholeStageCodeGeneration

plateau

[Experimental results] Whole Stage Code Generation

75 71 64 83 73 40 56 55 43 85 89 63 84 27 92 82 28 93 91 12 96 32 7 66 20 45 98 42 68 52 87 90 94 79 3 48 26 60 46 34 67 51 15 29 65 76 50 31 49 80 19 18 21 58 17 13 97 22 95 54 25 39 88 24 721

10

100

1000

w/ WSCG w/o WSCG

100GB dataset

[Experimental results] Performance of Hive w/ and w/o LLAP

67 51 70 97 92 68 46 42 71 73 90 34 39 48 84 52 43 63 98 13 3 7 79 66 20 26 27 96 21 45 89 80 12 56 15 32 40 18 49 19 55 60 54 87 31 76 75 82 22 88 28 58 94 93 95 721

10

100

1000

10000

llap container

LLAP shows improvement over container-based Tez for most queries

100GBdataset

1TBdataset

80 84 96 79 94 51 91 31 56 67 27 32 95 46 72 39 19 48 90 52 21 45 15 3 34 55 42 20 12 97 43 60 63 22 26 76 75 83 88 68 70 89 65 28 13 87 40 58 18 85 25 17 29 93 50 92 66 73 711

10

100

1000

10000

llap container

All queries : 78.3% reduc-tionAll except 72 : 36.1% reduc-tion

All queries : 44.9% reduc-tionAll except 72 : 27.9% reduc-tion

[Experimental results][Query 75] without and with LLAPwithout LLAP without LLAP

with LLAP

with LLAP

[Experimental results][Query 93] without and with LLAPwithout LLAP without LLAP with LLAP

with LLAP

[Experimental results][Query 94] without and with LLAPwithout LLAP without LLAP with LLAP

with LLAP

[Experimental results][Query 93] Difference pattern of resource utilization

Network CPU

Presto Hive

CPU Network

Spark

The end