Analyzing Query Performance and Attributing Blame forContentions in a Cluster Computing Framework

Prajakta Kalmegh, Shivnath Babu, Sudeepa RoyDepartment of Computer Science, Duke University

308 Research Drive Duke Box 90129, Durham, NC 27708-0129

{pkalmegh,shivnath,sudeepa}@cs.duke.edu

AbstractThere are many approaches is use today to either preventor minimize the impact of inter-query interactions on ashared cluster. Despite these measures, performance is-sues due to concurrent executions of mixed workloads stillprevail causing undue waiting times for queries. Analyz-ing these resource interferences is thus critical in order toanswer time sensitive questions like who is causing myquery to slowdown in a multi-tenant environment. Moreimportantly, dignosing whether the slowdown of a queryis a result of resource contentions caused by other queriesor some other external factor can help an admin narrowdown the many possibilities of performance degradation.This process of investigating the symptoms of resourcecontentions and attributing blame to concurrent queries isnon-trivial and tedious, and involves hours of manuallydebugging through a cycle of query interactions.

In this paper, we present ProtoXplore - a Proto orfirst system to eXplore contentions, that helps administra-tors determine whether the blame for resource bottleneckscan be attibuted to concurrent queries, and uses a method-ology called Resource Acquire Time Period (RATP) toquantify this blame towards contentious sources accu-rately. Further, ProtoXplore builds on the theory ofexplanations and enables a step-wise deep explorationof various levels of performance bottlenecks faced by aquery during its execution using a multi-level directedacyclic graph called Proto-Graph. Our experimentalevaluation uses ProtoXplore to analyze the interac-tions between TPC-DS queries on Apache Spark to showhow ProtoXplore provides explanations that help indiagnosing contention related issues and better managinga changing mixed workload in a shared cluster.

1 IntroductionThere is a growing demand in the industry for autonomousdata processing systems [10]. Some critical roadblocksneed to be addressed in order to make progress towardsachieving the goal of automation. One of these road-blocks is in ensuring predictable query performance in amulti-tenant system. “ Why is my SQL query slow?” This

question is nontrivial to answer in many systems. The au-thors have seen firsthand how enterprises use an army ofsupport staff to solve problem tickets filed by end userswhose queries are not performing as they expect. An enduser can usually troubleshoot causes of slow performancethat arise from her query (e.g., when the query did notuse the appropriate index, data skew, change in executionplans, etc.). However, often the root cause of a poorly-performing ‘victim query’ is resource contention causedby some other ‘culprit queries’ in a multi-tenant system[36, 12, 13, 5, 9]. Diagnosing such causes is hard and timeconsuming. Today, cluster administrators have to manu-ally traverse through such intricate cycles of query inter-actions to identify how resource interferences can causeperformance bottlenecks.

Should the solution be prevention, cure, or both? Fea-tures like query isolation at the resource allocation levelprovide one approach to address the problem of unpre-dictable query performance caused by contention. Forexample, an admin tries to reduce conflicts by partition-ing resources among tenants using capped capacities [3],reserving shares [17] of the cluster, or dynamically regu-lating offers to queries based on the configured schedulingpolicies like FAIR [32] and First-In-First-Out (FIFO). De-spite such meticulous measures, providing performanceisolation guarantees is still challenging since resourcesare not governed at a fine-granularity. Moreover, the al-locations are primarily based on only a subset of the re-sources (only CPU in Spark [35], CPU and Memory inYarn [31]) leaving the requirements for other shared re-sources unaccounted for. For example, two queries thatare promised equal shares of resources get an equal op-portunity to launch their tasks in Spark. However, thereare no guarantees on the usages of other resources likememory, disk IO, or network bandwidth for competingqueries. For instance, if a task uses more resources dueto skewed input data on a single host, this can result inhigher resource waiting times for tasks of other querieson this host compared to others.

1

arX

iv:1

708.

0843

5v2

[cs

.DC

] 2

9 M

ay 2

018

However, in practice, an approach based solely on pre-vention will not solve the problem. Real-life query work-loads are a mix of very diverse types of queries. For exam-ple, long-running ETL (Extract-Transform-Load) batchqueries often co-exist with short interactive Business In-telligence (BI) queries. Analytical SQL queries are sub-mitted along with machine learning, graph analytics, anddata mining queries. It is hard to forecast what the re-source usage of any query will be and to control what mixof queries will run at any point of time. In addition, theend users who create and submit the queries may havevery different skill levels. All end users may not be skilledenough to submit well-tuned queries. Thus, a practical ap-proach to provide predictable query performance shouldsupplement ‘prevention’ techniques with techniques for‘diagnosis and cure’.

Our Contributions: This paper focuses on the latter.We present ProtoXplore, - a Proto or first systemto eXplore the resource interferences between concur-rent queries that provides a framework to attribute blamefor contentions systematically. ProtoXplore’s foun-dations are based on the theory of providing explanations.While there have been several attempts to drill-down tothe systemic root causes when a query slows down, webelieve that ProtoXplore is a first attempt towards us-ing symptoms of low-level resource contentions to ana-lyze high-level inter-query interactions. Specifically, wemake the following contributions: (a) Blame Attribu-tion: We use the Blocked Times [26] values for mul-tiple resources (CPU, Network, IO, Memory, Schedul-ing Queue) to develop a metric called Resource Ac-quire Time Penalty (RATP) that aids us in computingblame towards a concurrent task.(b) Explanations us-ing Proto-Graph: We present a multi-level DirectedAcyclic Graph (DAG), called Proto-Graph, that en-ables consolidation of RATP and blame values at differ-ent granularity. (c) Web-based UI for deep explo-ration: Our web-based front-end (to be demonstrated inSIGMOD’18 [8]) allows users to explore ProtoXploreand find answers for detecting: (i) hot resources, (ii) slownodes, (iii) high impact causing culprit queries, and (iv)high impact receiving victim queries..

Our experiments evaluate ProtoXplore usingTPCDS queries on Apache Spark. The results of our user-study [1] shows how ProtoXplore helps user of differ-ent expertise to reduce debugging effort significantly.

1.1 Related WorkPrevention of Interferences: Approaches like [3, 32]help an admin control resource allocations in a clusterexecuting mixed workloads. Techniques like resourcereservations[17] or dynamic reprovisioning[22] isolatejobs from the effects of performance variability caused bysharing of resources. Our on-going research focuses on

prescriptive mechanisms using online explanations gener-ated from ProtoXplore to avoid interferences.

Diagnosis and Cure: Figure 1 presents a summary ofhow our work compares to the following approaches:(1) Monitoring Tools: Cluster health monitoring toolslike Ganglia [7] and application monitoring tools likeSpark UI [11] and Ambari [2] provide an interface to in-spect performance of queries.Table 1: Comparison of ProtoXplore with other ap-proaches

(Category) Re-lated Work

OLAPWork-loads

DetectSlow-down

DetectBottle-necks

BlameAttri-bution

Data-flowAware

(1) Ganglia,Spark UI,Ambari

✓ ✓

(2) Starfish,Dr. Elephant,OtterTune

✓ ✓ ✓

(3) PerfXplain,Blocked Time,PerfAugur

✓ ✓ ✓

(3) OracleADDM, DI-ADS

✓ ✓ ✓ ✓

(4) CPI2 ✓ ✓CPU ✓(4) DBSherlock ✓ ✓ ✓

ProtoXplore✓ ✓ ✓ ✓ ✓

(2) Configuration recommendation: More recent toolslike Starfish[20], Dr.Elephant[6], and OtterTune[30] an-alyze performance of dataflows and suggest changes inconfiguration parameters. While these approaches helpsin identifying and fixing problems in configurations, it ishard to predict how these system-wide changes will affectthe resource interferences from inter-query interactions inan online workload.(3) Root Cause Diagnosis tools: Performance diagnosishas been studied in the database community [19, 15, 33],for cluster computing frameworks [24, 26], and in cloudbased services [27]. The design of ProtoXplore isbased on several concepts introduced in these works:(a) Database Community: In ADDM [19], DatabaseTime of a SQL query is used for performing an im-pact analysis of any resource or activity in the database.ProtoXplore furthers this approach to provide an end-to-end query contention analysis platform while also en-abling multi-step deep exploration of contention symp-toms. ProtoXplore’s multi-level explanations ap-proach is motivated from DIADS [15]. They use Anno-tated Plan Graphs that combines the details of databaseoperations and Storage Area Networks (SANs) to providean integrated performance diagnosis tool. They, however,do not consider any factors related to the impact of thesequeries in affecting the values of low-level metrics. We

2

believe that this is really important to do an accurate im-pact analysis. The problem addressed in DIADS is notrelated, but our multi-level explanation framework bearssimilarity to their multi-level analysis. Causality basedmonitoring tools like DBSherlock [33] use causal mod-els to perform a root cause analysis. DBSherlock ana-lyzes performance problems in OLTP workloads, whereasProtoXplore focuses on performance issues caused byconcurrency in data analytical workloads. (b) ClusterComputing: PerfXplain is a debugging toolkit that usesa decision-tree approach to provide explanations for theperformance of MapReduce jobs. ProtoXplore con-siders dataflow dependencies and workload interactionsto generate multi-granularity explanations for a query’swait-time on resources. Blocked Time metric [26] em-phasizes the need for using resource waiting times for per-formance analysis of data analytical workloads; we usethis pedestal to identify the role of concurrent query ex-ecutions in causing these blocked times for a task. (c)Cloud: PerfAugur [27] identifies systemic causes forquery slowdown. Finding the root cause for slowdownis not the focus of ProtoXplore. Our motivation is toenable an admin dignose whether and why this slowdownis a result of resource contentions caused by other queriesor some other external factor.(4) Blame Attribution:CPI2 [36] uses hardware counters for low-level profil-ing to capture resource usage by antagonist queries whilethe CPI (CPU Cycles-Per-Instruction) of the victim querytakes a hit. Since this approach does not capture multi-resource contentions at application-level, it suffers fromfinding poor correlations when queries impact on otherresources. Blame attribution has also been studied in thecontext of program actions [18]. The purpose and appli-cation addressed in ProtoXplore is totally different.(5) Other work on explanations in databases: The fieldof explanations has been studied in many different con-texts like data provenance [16], causality and responsi-bility [25], explaining outliers and unexpected query an-swers [32, 28], etc. The problem studied and the methodsapplied in this paper are unrelated to these approaches.

Roadmap. In Section 2 we describe the key challengesin analyzing dataflows, multi-resource contentions and at-tributing blame in a shared cluster. We present our blameattribution process in Section 4. We describe the construc-tion of Proto-Graph in Section 6. Our experimentaland user-study results are in Section 8, and finally con-clude with on-going and future work in Section 9.2 ChallengesIn this section, we review some challenges for blame at-tribution and discuss the motivation behind our work.

Example 2.1 Consider the dataflow DAG of a query Q0

in Figure 1 comprising stages s0, s1,⋯, s5. The stage s5is the final or root stage, whereas the leaves s0, s1, s2

Figure 1: Dataflow DAG of an example query Q0.scan input data. The stage s3 can start only when all ofs0, s1, s2 are completed. Suppose an admin notices slow-down of a recurring query Q0 shown in Figure 1 in an ex-ecution compared to a previous one, and wants to analyzethe contentions that caused this slowdown. The admin canuse tools like SparkUI [11] to detect that tasks of stage s1took much longer than the median task runtime on host(i.e., machine) X , and then can use logs from tools likeGanglia [7] to see that host X had a high memory con-tention. Similarly she notices that s5 was running on hostY that had a high IO contention. Further, the admin seesthat stage r3 of another query Q1 was executing concur-rently with only stage s5 of Q0, while stages u5, u7, u9 ofquery Q2 were concurrent with both s1 and s5.Challenge 1. Assessing Performance bottlenecks:Use of blocked time [26] alone to compare impact due toresource contentions can be misleading.Suppose both the target stages s1 and s5 spent the sametime waiting for a resource (say, CPU); the impact of con-tention faced by them could still differ if they processeddifferent sizes of input data. Identifying such dispropor-tionality in wait-times can help an admin diagnose victimstages in a dataflow .

Challenge 2. Capturing Resource Utilizations:Low-level resource usage trends captured using hardwarecounters cannot be used towards blame attribution forqueries.Approaches like capturing the OS level resource utiliza-tions [36] require computing the precise acquisition andrelease timeline of each resource using hardware counters,and associating them with the high-level abstract task en-tities. This is non-trivial especially for resources like net-work (as requests are issued in multiple threads in back-ground) and IO (requests are pipelined by the OS).

Figure 2: Example overlap of resource usage betweenTask1 and Task2. Notice the multiplexing between thetasks for compute time.Challenge 3. Capturing Resource Interferences:CPU Time stolen from concurrent queries is inadequateto capture multi-resource contentions.

3

As shown in Figure 2, tasks can use and be also blockedon multiple resources simultaneously [26] resulting inmulti-resource interferences between concurrent tasks.As a result, metrics like stolen time [9] cannot be usedas is to quantify blame as it is used only in the contextof identifying processes that steal CPU time. Capturingtime stolen for multiple resources simultaneously requiresa more involved metric for blame attribution.

Challenge 4. Avoiding False Attributions:Overlap Time between tasks of a query does not necessar-ily signal a contention for resources between them.Today, admins consider only the % overlap between con-current queries to assign blame, which may lead to faultyattributions. For example, even if stage s5 of Q0 had to-tal overlap with IO-intensive tasks of Q2, this contentionmay not have impacted s5 owing to its minimal IO activ-ity (later stages in data analytical queries typically processless data).

Challenge 5. Analyzing Contentions on Dataflows:Dataflow dependencies and parallel stage executionsmake it harder to diagnose the root cause of overall queryslowdown.In the above example, stages s1 and s5 of Q0 (in darkred) were a victim of contention, but there is no easy wayfor the admin to know which of these stage of Q0 wasresponsible in its overall slowdown. Additionally, diag-nosing whether Q1 or Q2 (and which of their stages) isprimarily responsible for creating this contention is hard.

Motivation: There are several intricate possibilities evenin this toy example, whereas there may be long chains ofstages running in parallel in a real production environmentmaking this process challenging. Today admins have lim-ited means to analyze the impact due to query-interactionsapart from looking at individual cluster utilization logs,specific query logs, and manually identifying correlationsin both. This forms the motivation of our work.

3 System OverviewProtoXplore is a system that helps us address theabove challenges using our methodology for blame at-tribution and our graph-based model used for consolida-tion of blame. It enabes users to detect contentions on-line while the queries are executing, or perform a deepexploration of contentious scenarios in the cluster offline.Figure 3 shows an overview of ProtoXplore’s archi-tecture.

In Step (1), an admin uses tools like Spark UI [11]to monitor the execution of queries and then identifies aset of queries to be analyzed. Target query and targetstages: Any query in the system can be a potential sourceor a target of contention. Each of the queries chosen fordeep exploration in ProtoXplore is termed as a tar-get query Qt, its stages as target stages, and its tasks as

Figure 3: ProtoXplore System Architecture

target tasks. ProtoXplore currently provides an in-terface to analyze contentions faced by (i) all stages in thelongest path of execution, (ii) any single stage, and (iii) allstages in the target query(our default option).Source query and source stages: We refer to other con-currently running queries Qs that can possibly cause con-tention to a target query Qt as source queries (if thestage(s) of Qs are waiting in the scheduler queue or run-ning concurrently with the stage(s) of Qt). Stages of Qsare source stage, and its tasks are source tasks. Finally,we refer to source queries that cause multi-resource con-tentions for several concurrent queries with high impactas aggressive queries.

Once the user submits target queries to ProtoXplorein step (3), the Proto-Graph Graph Constructorbuilds a multi-level graph as described in Section 6 forthese queries (Step (4)). In addition, users can configurethe resources or hosts for which they want to analyze thecontentions. For example, users can diagnose impact ofconcurrency on only scheduler queue contentions, or CPUcontention on a subset of hosts, or source queries submit-ted by a particular user etc. For each task of a target stage,ProtoXplore computes the RATP and blame metricsas described in Section 4. The Blame Attributor mod-ule then consolidates these values for stages, as shown inStep (5), which are then used to update the node and edgeweights of the graph subsequently. The ExplanationsGenerator updates a degree of responsibility (DOR) met-ric for each node, which is then used by the Blame Anal-ysis API to generate explanations at various granularities(step (6)). Specifically, we generate three levels of expla-nations for a query’s slowdown due to resource interfer-ences, namely Immediate Explanations (IE), Deep Expla-nations (DE), and Blame Explanations (BE) that lets usersidentify sources of contentions at resource, host and querylevels respectively. These explanations are then consumedin Step (7) by the Rule Generator to output heuristics forthe admin (like alternate query placement, dynamic pri-ority readjustment for stages and queries). The details of

4

rule generation is out of scope of this paper and is focus ofour ongoing research; a preliminary implementation withbasic rules can be found in the demo paper [8]. In step (8),these rules are stored for the admin to later review andapply to the cluster scheduler. The web-based interfacepresents a dashboard to the admin to visualize cluster andquery performance statistics, contention summary graphsfor a selected workload, hints in selection of target queriesfor analysis, top-k sources of impact to the selected targetqueries, and the corresponding Proto-Graph [8]. Next,we discuss the theory behind our Blame Attributor mod-ule.

4 Blame AttributionIn order to address the challenges 1-4 described in Sec-tion 2, we develop a metric called Resource Acquire TimePenalty (RATP) that helps us formulate the blame towardsconcurrent tasks.

4.1 Resource Acquire Time Penalty (RATP)In a δτ interval, let the time spent by a target task tt toacquire RAδτtt,r,h units of resource r on host h be:

T δτtt,r,h = WTδτtt,r,h + CTδτtt,r,h (1)

where, WTδτtt,r,h is the time spent blocked or waiting forthe resource and CTδτtt is the time spent consuming it inthis interval. For the remaining section, we assume thatthe resource r and host h are fixed, so we omit r, h in thesubscripts and other places for simplicity where it is clearfrom the context.

Definition 4.1 We define the Resource Acquire TimePenalty (RATP) for tt for resource r on host h in intervalδτ as:

RATP δτtt =(WTδτtt + CTδτtt )

RAδτtt(2)

4.2 Slowdown of a taskIn the δτ interval, let the capacity of a host h to serve aresource r be C units/sec (note: r, h superscripts omitted).Thus, the total used capacity by all consumers of resourcer is bounded by the system capacity C, expressed as:

C = Ctt + C1 + C2 + ⋅ ⋅ ⋅ + Cn +k

∑1

Cki +l

∑1

Cui (3)

where, Ctt is the capacity used by target task tt;C1,C2, . . . Cn are the individual used capacities by n otherconcurrent tasks; whereas Cki and Cui represents capaci-ties used by other k known and l unknown causes. Theminimum time required to acquire resource r on this hostcan then be expressed as RATPδτ = 1

Csec/unit. Thus,

slowdown of task tt due to unavailability of resources is:

Definition 4.2 Slowdown Stt of a task tt in interval δτbetween time τ and τ + δτ is defined as

Sδτtt =(RATPδτtt − RATPδτ)

RATPδτ(4)

where, RATPδτtt is computed as per Equation 2.

Intuitively, it is the deviation from the ideal resource ac-quire time on that host. Therefore, the slowdown corre-sponds to the blame that can be attributed to unavailabilityof resources which can be attributed to other tasks runningconcurrently with tt on h during its execution, or on otherquantities as shown in Equation 3, giving us:

Sδτtt = (n

∑1

βδτst→tt)

´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶p1

+k

∑1

βδτki→tt +l

∑1

βδτui→tt

´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶p2

(5)

Here βδτss→ts is the blame assigned to each of the n concur-rent tasks executing on the same host during task tt’s ex-ecution. Wait-times for resources are sometimes also dueto processes that are part of the framework but not a validsymptom of interferences between tasks. (e.g., garbagecollection for CPU, HDFS replication for network). Theseprocesses impact all concurrent tasks alike for the specificresource. βδτki→tt gives the blame value assigned to otherknown non-concurrency related causes that result in tasktt’s wait-time. Finally, slowdown could be due to a vari-ety of other causes which are either not known or cannotbe attributed to any concurrent tasks. βδτui→tt captures thisvalue of slowdown due to systemic issues.

4.3 Slowdown due to Concurrency

The slowdown defined in p2 of Equation 5 is the best caseslowdown for using resource r on host h in the absence ofany interferences due to concurrency. We now derive p1of Equation 5. First we discuss a simpler case, when thereis a full overlap of tt with concurrently running tasks topresent the main ideas. Then we discuss the general casewith arbitrary overlap between st and tt.

4.3.1 Full overlap of tt with concurrent tasks

Rewriting Equation (3) using RATP values to account forcapacities used by running tasks (omitting the known andunknown causes) in interval δτ :

1

RATPδτ= 1

RATPδτtt+ 1

RATPδτ1+ 1

RATPδτ2+ ⋅ ⋅ ⋅ + 1

RATPδτn

Multiplying by RATPδτtt and subtracting 1 on both sidesyields,RATPδτtt − RATPδτ

RATPδτ= RATPδτtt

RATPδτ1+ ⋅ ⋅ ⋅ + RATPδτtt

RATPδτnThe left hand side above is the increase in RATP of tar-get task and thus represents its slowdown Sτtt given by(Definition 4.2)

Sδτtt = [ ∑st∈ n

RATPδτtt

RATPδτst] (6)

Each term on the right hand side of the above equationis contributed by one of the source tasks concurrent to tasktt, and corresponds to blame attributable to a source taskst in this interval, that is, βτst→tt =

RATPτttRATPτst

.

5

4.3.2 Partial overlap with concurrent tasksThe above derivation works for a time interval in which allconcurrent tasks have a total overlap with tt. In practice,they overlap for different length of intervals as can be seenfrom Figure 4. As a result, we cannot use Equation (6)directly to attribute blame to a source task st. To deriveblame for this scenario, we divide the total duration T =ttend − ttstart of task tt’s execution time in δτ intervalssuch that in each δτ time-frame the above Equation (6)holds.

Let S1,S2, . . . ,Sm be the slowdown in each of them = T

δτintervals of the task’s execution. This gives us

the mean slowdown of tt as (Ttt is the execution time oftt):

Smean =δτ

Ttt∑k∈m

Sk

Substituing the value of slowdown Sk in the kth intervalusing Equation (6),

Smean =δτ

Ttt∑k∈m

∑st∈θk

RATPktt

RATPkst

where, θk is the set of source tasks that are concurrent inthe kth interval with task tt.In order to understand the contribution of each source tasktowards the mean slowdown, the double summation canbe rearranged as.

Smean = ∑st∈θ

∑k∈m′

RATPktt

RATPkst

δτ

Ttt

where, for each st, m′ is the number of intervals fromtime ttstart to ttend in which st overlaps with tt. Theouter sum is now on the set of all tasks, θ, that had anoverlap with tt.

The above equation in the limiting case is:

Smean = ∑st∈θ

[∫oendst,tt

ostartst,tt

RATPtt

RATPst

dt

Ttt] (8)

where, oendst,tt - ostartst,tt is the overlap time between taskstt and st. The integral inside the summation is the totalblame towards a source task st, giving us the definition ofblame:

Figure 4: An example overlap between concurrent tasks.

Proposition 4.3 The blame βδτss→tt for the contentioncaused by a task st of a source query to the task tt ofa target query on host h for resource r in interval τ toτ + δτ can be expressed as

βδτst→tt =1

Ttt[∫

oendst,tt

ostartst,tt

RATPδτtt

RATPδτstdδτ] (9)

where, oendst,tt, ostartst,tt denote the ending and starting points

of the overlap time between tasks tt and st (on host h),and Ttt is the total execution time of task tt.

5 ImplementationIn this section, we discuss how we derive blame in prac-tice, the supported resources for analyzing blame, and themetrics support we have instrumented in Spark.

5.1 Blame Formulation using Blocked TimeA task continues to execute even when its resource requestis not completely fulfilled (especially for IO and Net-work) [26]. This is because these resources are pipelinedto it by background application threads or by the OS.Hence a source task’s real impact (attributable blame) isonly to the extent that the target task is blocked for theresource. This adjustment is also important to predict theexpected speed up of target task more accurately. If an ad-min decides to kill a source task because of its contentionthen he can only expect an improvement proportional totarget task’s blocked time (not its entire resource acqui-sion time). The blocked time WTδτtt,r,h values (see Equa-tion 1) are also relatively more easier to capture as shownin [26]. We thus define an adjusted RATP for a task tt as:

RATP-blocked δτtt =

WTδτtt

RAδτtt(10)

For the source task though, we continue to use the en-tire interval δτ as time spent to acquire RAδτst units of re-source instead of its blocked time. This lowers the at-tributable blame further but allows us to account for thosesources whose blocked time may not be available (espe-cially terms of p2 in Equation 5) For clarity, we representthis as max RATP which for a source st is as:

RATP-max δτst =δτ

RAδτst(11)

The blame value we use is therefore:

β-blocked δτst→tt =

1

Ttt[∫

oendst,tt

ostartst,tt

RATP-blocked δτtt

RATP-max δτstdδτ] (12)

As blocked time in an interval is bound by the total in-terval length, β-blocked δτ

st→tt ≤ βδτst→tt.5.2 Handling Known Causes:A query may experience slowdown due to myriad rea-sons like systemic issues (executors getting killed, ex-ternal processes, etc), the query’s own code (used thewrong index, change in query plan, etc.) and many more.

6

The wait-time for every resource can be attributed to anyof these issues besides concurrency. On the other hand,concurrenct executions may have additional indirect con-sequences like slowdown of a task due to increase ingarbage collection time owing to a previously executedresource-intensive task, etc. Research in the area of di-agnosing such known and unknown systemic causes forperformance degradation is well established [24, 33, 21],which is out of the scope of discussion of this paper. Al-though ProtoXplore’s primary focus is to account forthe βδτst→tt term, it also comes pre-configured for attribut-ing blame to select known causes like JVMGCTime (cpu),HDFS repartitioning (IO) and RDD storage (Memory)etc. ProtoXplore also allows adding known causes viaits feedback module where users with domain knowledgeand prior experience with ProtoXplore can configuresupport for them. ProtoXplore also avoids false attri-bution by using catch-all term for unknown causes as weshow next.

5.3 Avoiding False Attributions:Our quantification of blame β-blocked st→tt enables usto distinguish between legitimate and false sources ofcontention owing to the following characteristics: (i)Since the blame is computed only for periods where ttis blocked for resource r during the overlap period, theblame on st is zero if tt’s wait-time is zero during theoverlap. (ii) If both st and tt are not waiting for the sameresource in the overlapping time (see Challenge 4), stdoes not qualify for blame. Blame attribution using mereoverlap time can miss this subtlety. (iii) Finally, slowdowncould be due to a variety of other causes which are eithernot known or cannot be attributed to any concurrent tasks.To handle these cases, ProtoXplore creates a syntheticunknown source query in the framework. We assume thatsuch a query overlaps with all tasks of every query on ev-ery host of execution. For every resource, we then keep atrack of the system-level resource used values (currentlyProtoXplore is configured for IO bytes read/writtenand network bytes read resources) during each executionwindow. We compute the total-unaccountable-resourceby subtracting the resource acquired values for all tasksrunning during that window from the system-level metric.The blame attributed to this unknown source is then com-puted by using the RAδτst = total-unaccountable-resourcein the RATP-max δτst computation of Equation 12.

5.4 Supported ResourcesThe values of resource acquired RAδτtt for most resourceslike input or output bytes, shuffle bytes read, etc., andsome wait-time metrics like fetch-wait-time, shuffle-write-time and scan-time for a task are available through Spark’sdefault metrics. For additional metrics (denoted with a su-perscript ∗ against them) and time-series data we addedinstrumentation to Spark and built per-host agents (for

system metrics). In each window of data collected duringtask tt’s execution, we consider contention for the follow-ing resources on host h:

• Network: We compute RATPnetwork using the NetworkWait Time (NWT) or fetch-wait-time metric from Spark.The value of resource acquired RAnetwork is the shufflebytes read metric in that window.

• Memory: To capture the contention for process-managed memory , we compute (i) RATPstorage−memusing storage memory wait time (SMWT∗) and storagememory acquired∗, (ii) RATPexec−mem using executionmemory wait time (EMWT∗) and execution memory ac-quired∗ metrics.

• IO: We compute the RATP for IO Read RATPio−read us-ing scan-time and bytes read in the scan operation. ForIO Write, its RATPio−write is given using shuffle-write-time and shuffle-bytes-written metrics in Spark.

• CPU: Apart from above resources, a task’s executioncan wait due to other reasons like os scheduling wait,acquiring locks/monitors, priority lowering etc. Wecapture (a) monitor and lock wait time metrics fromJMX (b) garbage collection wait time from spark ex-isting metrics and (c) attribute all other wait time to osscheduling delay. We deal with garbage collection spe-cially by creating a synthetic GC query in our modelas a possible source of contention. This query is mod-eled to have a single long running task on each execu-tor. This task is assumed to block all other tasks duringthe duration of its execution. We also attribute the re-maining unaccounted wait time in an interval to the OSscheduling wait time. As the blame due to a concurrenttask is based on RATP ratios, any error in this way ofaccounting for scheduling wait is minimized to a largeextent. Finally for each of the above CPU-impactingwait-times, we use the value of executor-cpu-time as theresource acquired.

5.5 Frequency of Metrics Collection:For each task tt, we added support to collect the time-series data for its wait-time and the corresponding dataprocessed metrics at the boundaries of task start and fin-ish for all other tasks st concurrent to tt. Figure 5 showsthe four cases of task start end boundaries concurrent totarget task tt. That is, instead of collecting the timeseriesdata at regular δτ intervals as shown in Figure 4, we cap-ture it only for concurrency related events that can havean impact on a task’s RATP values. Note that with thisapproach, the extent of each δτ depends on the frequencyof arrival and exit of concurrent tasks, thus enabling us tocapture the effects of concurrency on a task’s executionmore accurately.

However, if the arrival rate of concurrent tasks is low,this can affect the distributions of our metric values. To

7

Figure 5: We collect metrics data at concurrent task startand end boundaries.

address this, we also record the metrics at heartbeat inter-vals in addition to above task entry and exit boundaries.On the other hand, for workloads consisting of tasks withsub-second latency, our approach gives more fine-grainedwindows for analysis. Section 8 compares the impact ofboth the approaches on the quality of our explanations.

6 iQC-GraphIn the previous section, we discussed our methodologyto assign blame towards a concurrent task st for causingcontention to a target task tt. As seen previously, a querytypically consists of a dataflow of stages where each stageis composed of multiple parallel tasks. In order to con-solidate blame caused or faced by a query, we constructProto-Graph, a multi-layered directed acyclic graph.

6.1 Levels in Proto-GraphWe now describe the levels in Proto-Graph (the fullpseudo-code for constructing it can be found in [23] dueto space constraints). Level 0 and Level 1 contain the tar-get queries and target stages respectively – these are thequeries or stages that the admin wants to analyze. Onthe other end of Proto-Graph, Level 6 represents allsource queries, while Level 5 contains all source stages,which are the queries and stages concurrently runningwith the target queries. Note that it is important to sep-arate the queries and their stages in two different levels atboth ends of Proto-Graph- this is to diagnose impactsfrom and to multiple stages of a query.

The middle three levels – Levels 2, 3 and 4 (describedshortly) – keep track of causes of contentions of differentforms and granularity that enable us to connect these twoends of Proto-Graph with appropriate attribution ofresponsibility to all intermediate nodes and edges. Levels2, 3, and 4 answer the impact questions described aboverespectively.

6.2 Vertex ContributionsFor each node u in the graph, we assign weights, calledVertex Contributions denoted by VCu, that are used laterfor analyzing impact and generating explanations. Specif-ically, VCu measures the standalone impact of u towardthe contention faced by any target query vertex in Level-0.The VC values of different nodes are carefully computed

at different levels by taking into account the semantics ofrespective nodes. For Level 0 target query, Level 5 sourcestage, and Level 6 source query vertices, we currently setVCu = 1 for all nodes u in each level. Our admin inter-face allows to configure these based on the requirementsof the workload, e.g. assign higher weights to queries fromcertain users, or SLA-driven queries. For the Level 1 tar-get stage vertices, we set this to the cumulative CPU time(i.e., the work done) by stage s in order to assign higherimpacts through stages that get more work done. We nowdescribe Levels 2-4, and show how we compute these val-ues for them.

Level 2 - Track Resource Level Contentions: For eachtarget stage in Level 1, we add five nodes in Level 2 cor-responding to each of the resources, namely schedulingslots, CPU, network, IO and memory. From Equation 2,the RATP of a task tt for resource r during its executionfrom time tstart to tend (integrate over all δτ intervals) onhost h (superscripts r, h ommitted) is:

RATPtt = ∫τendtt

τstarttt

RATPδτtt dδτ (13)

For each node u in Level-2, we assign its V C`2u to bethe cumulative RATP for all tasks of this target stage vertexin Level-1 as:

VC`2u =∑tt∫

τendtt

τstarttt

RATPτ,r,htt dδτ (14)

Level 3 - Track Host Level Contentions: Level-3 un-folds the components of wait time distribution of a targetstage further to keep track of the RATP-blocked values ofeach stage for a specific resource on each host of its execu-tion. First, we find all the hosts that were used to executethe tasks of a particular target stage. Then, for everynodein Level 2 corresponding to resource r, we add Pr × Hnew nodes in Level 3, where Pr is the number of differ-ent requests that can lead to a wait time for resource r,and H is the number of hosts involved in the executionof this target stage. For instance, the IO bottleneck of atarget stage can be explained by the distribution of timespent waiting for IO READ and IO WRITE. Therefore, forr = IO, Pr = 2, and for each host we add two nodes forIO BYTES READ RATP and IO BYTES WRITE RATPin Level 3. The Level-3 nodes for other resources are gen-erated as discussed in Section 5.4.

The value of VC`3u is set similar to Equation 14 exceptthat the summation is done over all tasks of a target stage(denoted as h − tasks) executing only on host h in ques-tion.

VC`3u = ∑tt∈h−tasks

∫τendtt

τstarttt

RATPτ,r,htt dδτ (15)

Level 4 - Linking Cause to Effect: For each node v inLevel 3 corresponding to a target stage, host h, and typeof resource request r, if the tasks of this target stage were

8

executing concurrently with tasks belonging to P distinctsource stages on host h, we add P nodes u in Level 4and connect them to v. The VC`4u of each node is thencomputed from the blame attribution of all target stagesin Level 3 that it can potentially affect.

VC`4u = ∑tt∈h−tasks

β-blocked r,hst→tt (16)

6.3 iQC-Graph by exampleIn example 2.1, suppose the user selects Q0 as the targetquery, and wants to analyze the contention of the stageson the critical path. First, we add a node for Q0 in Level0, and nodes for s1, s3, s4, s5 in Level 1. Then in Level2, for each of these stages, the admin can see five nodescorresponding to different resources. Although both s1and s5 faced high contentions, using Proto-Graph theadmin can understand questions such as whether the net-work contention faced by stage s5 was higher than the IOcontention faced by stage s1, and so on.

Figure 6: Proto-Graph illustrating the details in Exam-ple 6.3

Suppose only the trailing tasks of stage s1 executingon host Y faced IO contention due to data skew. Us-ing Proto-Graph, a deep explanation tells the user thatIO BYTES READ RATP on host Y for these tasks of s1was much less compared to the average RATP for tasksof stage s1 executing on other hosts. This insight tells theuser that the slowdown on host Y for tasks of s1 was notan issue due to IO contention. If the user lists the top-knodes at Level-3 using our blame analysis API, she cansee that the network RATP for tasks of stage s5 on host Xwas the highest, and can further explore the Level-4 nodesto find the source of this disproportionality.

Since stage r3 of source queryQ1, and stages u5, u7, u9of another source query Q2 were executing concurrentlyon host X with stage s5 of Q0, the user lists the top-kLevel-4 vertices responsible for this network contentionfor s5. The blame analysis API outputs stage u7 of queryQ2 as the top source of contention. Figure 6 shows somerelevant vertices for this example to help illustrate the lev-els in Proto-Graph.Discussion: Note that we consider the cumulative RATPvalues for computing stage-level values (in Equation 14

and Equation 15). The other alternative approaches in-clude considering max or average values for the tasks in astage; however, sum captures the total cluster time (sim-ilar to the notion of Database Time in [19]) spent on wait-ing for resources per unit data by each stage and, thus,enables us to analyze the overall slowdown of a stage inthe cluster.

7 ExplanationsIn this section, we describe how the choice of a graph-based model enables us to consolidate the contention andblame values for generating explanations for a query’s pe-formance.

7.1 Generating ExplanationsThe Explanations Generator module in ProtoXploreuses the VC values to update two metrics, namely ImpactFactor (IF) and Degree of Responsibility (DOR). We setthe IF as edge weights and DOR as the node properties.

Impact Factor (IF): Once the Vertex Contributions VCuof every node u in Proto-Graph is computed to esti-mate the standalone impact of u on a target stage, we com-pute the Impact Factor IFuv on the edges (u, v). This en-ables us to distribute the overall impact received by eachchild node v among its parent nodes u-s. For instance,IFuv from a DE node u to an IE node v gives what frac-tion of total impact on an IE node can be attributed toeach of its parent DE nodes. Figure 7 shows an exampleof the impact received by node u1 from nodes v1, v2, v3.

Figure 7: Example computation of DOR from IF and VC

values.

The IF values of the edge weights are used to generateexplanations as follows:

An explanation in ProtoXplore takes the form:Expl( tq, ts, res, res′, host, ss, sq, P)where ,tq denotes the target stage being explained by φ; tsdenotes the target stage being explained by φ; res∈ CPU,Memory, IO,Network,Scheduling; res’ isthe type of resource impacted (see Section 5.4); host isthe host of impact; ss is the stage of the concurrent querythat has caused impact; sq is the impacting source query;P is the cumulative weight of the path originating fromsq and ending at tq.

9

7.2 Contention Analysis using iQC-GraphA top-k analysis lets user specify the number of top expla-nations she is interested in. Additionally, ProtoXploreconsolidates the explanations further to aggregate respon-sibility of each entity (queries, stages, resources andhosts) towards the contention faced by every target query.

Degree of Responsibility (DOR): Finally we compute theDegree of Responsibility DORu of each node u, whichstores the cumulative impact of u on a target query t. Thevalue of DORu is computed as the sum of the weights of allpaths from any node u to the target query node t, wherethe weight of a path is the product of all IFvw values ofall the edges (v,w) on this path. If we choose more thanone query at Level 0 for analysis, a mapping of the valuesof DOR toward each query is stored on nodes at Level 5and 6. The computation of DOR of node v3 is illustratedin Figure 7. It can be noted that since our frameworkincorporates impacts originating from the synthetic ‘Un-known’ source, it enables an admin to rule out slowdowndue to concurrency issues if the impact through this nodesis high. Using DOR, users can perform the followinguse-cases that enable deep exploration of contentions in ashared cluster:

• Finding Top-K Contentions for Target Queries: Forany query in the workload, users can find answers for:(a) What is the impact through each resource for itsslowdown? (b) What is the impact through each host fora specific resource? and, (c) What is the impact througheach source query or source stage on a specific host andresource combination?

• Detecting Aggressive Source Queries:ProtoXplore allows the admin to do a top-down analysis on a source query or source stage toexplore how it has caused high impact to all concurrentqueries. To detect such aggressive queries, we find the(top-k) Level 6 nodes having the highest value of totalDOR toward all affected queries (recall that for eachsource query, we keep track of the DOR value towardeach target query).

• Identifying Slow Nodes and Hot Resources: Perform-ing a top-k analysis on levels 2 (resources) and 3 (hosts)will yield the hot resource and its corresponding slownode with respect to a particular target query. In orderto get the overall impact of each resource or each hoston all target queries, ProtoXplore provides an APIto (i) detect slow nodes, i.e., group all nodes in Level3 by hosts, and then output the total outgoing impact(sum of all IF values) per host, and (ii) detect hot re-sources, i.e., - output the total outgoing impact per wait-time component nodes in Level 2.

8 EvaluationOur experiments were conducted on Apache Spark 2.2[35] deployed on a 20-node local cluster (master and 19slaves). Spark was setup to run using the standalonescheduler in FAIR scheduling mode [34] with defaultconfigurations. Each machine in the cluster has 8 cores,16GB RAM, and 1 TB storage. A 300 GB TPC-DS [14]dataset was stored in HDFS and accessed through ApacheHive in Parquet [4] format. The SQLs for the TPC-DSqueries were taken from the implementations in [29] with-out any modifications.

Workload: Our analysis uses a TPCDS benchmarkworkload that models multiple users running a mix of dataanalytical queries in parallel. We have 6 users (or tenants)submitting their queries to their dedicated queue. Eachuser runs queries sequentially from a series of randomlychosen 15 TPCDS queries. When submitting to a queue,the selection of queries is randomized for each user. Thequery schedules were serialized and re-used for each ofour experiments to enable us to compare and contrast re-sults across executions. During the experiments, our clus-ter slots were 97% utilized and Ganglia showed around36% average CPU utilization.

We now describe how ProtoXplore enables deeperdiagnosis of contentions compared to (a) Blocked-TimeAnalysis (BTA): we use the values of blocked times forIO and Network [26] and aggregate them at stage andquery levels, (b) Naive-Overlap: based on overlap timesof query runtimes (a technique popularly used by supportstaff trying to resolve who is affecting my query tickets),and (c) Deep-Overlap: we compute the cumulative over-lap time between all tasks of a pair of concurrent queries.

(a) (b)

Figure 8: (a) Resource blocked times from ProtoXplore

and BTA approach; (b) Blocked times for IO, CPU and Net-work.

8.1 Comparison with Baseline

For this experiment, we identify a target query Q43 fromour workload that was 178% slower compared to its un-constrained execution (when run in isolation). Figure 8acompares the bottlenecks observed (y-axis is on log scale)with the BTA method vs the blocked times captured byProtoXplore for its impacted stages (other stages hadminimal blocked times to compare). In addition to the

10

blocked times from network and IO, ProtoXplore ac-crues additional blocked times that help to identify poten-tial causes of slowdown for stages like 21 and 23. Fig-ure 8b compares the relative per resource blocked times.The network and CPU blocked time is significantly highfor stages 20 and 22. However, when we compare theRATP’s for these stages using ProtoXplore, we see inFigure 9a that the CPU RATP was much higher for mul-tiple stages compared to their IO RATP, whereas the net-work RATP was significantly low to even compare. If weanalyze the impact on each of these stages generated usingthe explanations module, stage 20 received the maximumimpact through CPU compared to others. This detailedanalysis cannot be obtained from looking at BTA alone.

(a) (b)

Figure 9: (a) RATP for IO and CPU for high-impactingstages, and (b) their DOR in impacting Q43.

Next, for assessing who is causing the above slowdownto Q43, we compare our results with Naive-Overlap andDeep-Overlap. For the overlap-based approaches, high-est blame is attributed to query with most overlap. Itcan be seen from Figure 10 that the top queries that over-lap differ significantly between Naive-Overlap and Deep-Overlap. The queries with minimal overlap shown inNaive-Overlap (Q4 and Q27) have relatively more tasksexecuting in parallel on the same host as that of targetquery, causing higher cumulative Deep-Overlap. The out-put from ProtoXplore is more comparable to Deep-Overlap, but has different contributions. Especially forQ11, where the tasks had a high overlap with tasks ofour target query Q43, the impact paths showed low pathweights between these end points. A further explorationrevealed that only 18% of the execution windows (cap-tured via the time-series metrics), showed increments inCPU and IO acquired values for Q11 for the matchingoverlapping windows. Q11 itself was blocked for these re-sources in 64% of the overlapping windows. Without theimpact analysis API of ProtoXplore, an admin willneed significant effort to unravel this.8.2 Test CasesThe purpose of our second set of experiments is three-fold to: (i) verify that ProtoXplore detects symptomsof contentions induced in our workload and outputs newlyidentified sources, (ii) demonstrate how ProtoXploreachieves this for a running workload, and (iii) how we

(a) (b) (c)

Figure 10: Compare top-5 impacting queries between (a)Naive-Overlap (b) Deep-Overlap and (c) ProtoXplore.

can perform a time-series analysis with ProtoXplore.Our test-cases for induced resource contention scenariosare summarized in Table 2. ProtoXplore detects con-tentions for Spark’s managed memory, hence the test-caseof Memory-external is out of scope. Note, we were notable to isolate contention only due to increased shuffledata as it was always resulting in increasd IO contentiontoo owing to shuffle write. So induction of network con-tention in our workload (sql queries) was not possible. Wenow share our induction methodology and observations:

8.2.1 CPU-InternalFor this experiment, we use the same workload as before(the schedules were serialized), and induce contentionquery Qcpu−int shown in Listing 1 along with our targetquery Q43. This query Qcpu−int exhibits the followingcharacteristics: (a) low-overhead of IO read owing to ascan over two small TPCDS customer tables each con-taining 5 million records stored in parquet format, (b) lownetwork overhead since we project only two columns, (c)minimal data skew between partitions as the join is onc customer sk column which is a unique sequential id forevery row and shuffle operation used hash-partitioning ,and (d) high CPU requirements owing to the sha2 functionon a long string (generated by using the repeat function ona string column). We now analyze the contentions of tar-get query Q43 further. Note that our induction can affectQ43 only to a certain extent owing to multiple aspects ofscheduling on a cluster, for example the concurrency oftasks of Qcpu−int with tasks of Q43 is dependent on thenumber of slots it can get under the configured schedulingpolicy. Despite this, ProtoXplore is able to captureQcpu−int as one of the top sources in the concerned win-dow as we show next.

Listing 1: Qcpu−int: CPU-Internal Induction querywi th temp( s e l e c t c1 . c f i r s t n a m e as f i r s t n a m e ,

sum ( sha2 (r e p e a t ( c1 . c l a s t n a m e ,4 5 0 0 0 ) , 5 1 2 ) ) as shasum

from c u s t o m e r c1 , c u s t o m e r c2where c1 . c c u s t o m e r s k

11

Table 2: Summary of Induced Contention ScenariosTest-Case DetailCPU-Internal User-7 submits a CPU-intensive custom SparkSQL query just after Q43 starts.CPU-External We inject a CPU-intensive custom query in a seperate Spark application instance.IO-Internal User-7 submits a IO-intensive custom SparkSQL query at a random time.IO-External A large 60GB file is read in 256MB block size on every host just after Q43 starts.Memory-Internal User-7 caches web sales table just before Q43 starts.

= c2 . c c u s t o m e r s kgroup by c1 . c f i r s t n a m e )

s e l e c t max ( shasum )from templ i m i t 100 ;

Observations: Figure 11 shows that the CPU utilizationreaches almost 80% during our induction, much higherthan the 36% average utilization for our workload.

Figure 11: Ganglia snapshot showing CPU Utilization whenQcpu−int was running.

Figure 12 shows the change in DOR values of thesource queries towards Q43. The live analysis windowscpu live 1, cpu live 2, cpu live 3 and cpu live total showthe periods in which we invoke ProtoXplore to ana-lyze contentions on Q43. The stacked bars show the rela-tive contribution from each of the top-5 running sources,and their heights denote the total contribution from thesefive sources. Since we are analyzing contentions on onlyQ43, the cpu live 1 window shows no contributions asQ43 had not started yet. Qcpu−int was induced at theend of window cpu live 2, hence no contribution fromQcpu−int in this window. However, as seen in cpu live 3,once we induceQcpu−int whenQ43 starts, its contributionof 27% is detected.

Figure 12: DOR ofQcpu−int is about 27% towards queryQ43

during CPU induction.

The end-of-window analysis in cpu live total shows the

overall impact Q43 from all sources during its execu-tion. Although Qcpu−int caused high contention to Q43

for a period, its overall impact was still limited (0.05%).Details like this are missed by alternative approacheslike Naive-Overlap) and Deep-Overlap as shown in Fig-ure 13. Moreover, an analysis with even Deep-Overlapcan mislead the user into attributing about 35% of re-ceived impact to Q11, whereas, the actual impact outputby ProtoXplore shows less than 1% overall impactfrom Q11.

(a) Naive Overlap (b) Deep Overlap

Figure 13: Impact from source queries.

8.2.2 IO-External

To induce IO outside of Spark, we read and dump a largefile on every host in the cluster after the workload stabi-lizes (at least one query is completed for each user). Letus call this query as IO − Other. The timing of induc-tion was chosen such that it overlaps with the scan stageof Q43. We created a 60GB file on each host and used thecommand in 2 to read in blocks of size 256MB using thebelow command:

Listing 2: IO −Other: IO-External Induction querydd i f = ˜ / f i l e 6 0 G B of = / dev / n u l l bs =256

Observations: Figure 14 shows over 1600% aggregatedisk utilization for all nodes (19 slaves) in the clusterduring this period of contention. We analyze impactsfrom sources for the following windows: (a) Q43 had notstarted in io live 1, (b) io live 2 analysis was done for tar-get query Q43 just before we induced IO − Other, (c)io live 3 to io live 5 windows while IO −Other is run-ning concurrently with Q43 (we omit showing io live 4due to plotting space constraints), (d) Q43 finishes execu-tion before io live 6 ,and (e) the io live total window toanalyze overall impact on Q43 from the beginning of the

12

Figure 14: Ganglia snapshot showing aggregate IO utiliza-tion during our external induction.

workload till the end. Figure 15 shows the relative con-tributions from each of the concurrent sources, showingthat ProtoXplore detects the induced contention firstin io live 3 (shown in green), and outputs an increasingimpact during io live 4 and io live 5 when it peaks.

Figure 15: DOR of source queries towards query Q43

changes as the workload progresses.

8.2.3 Mem-Internal

Since we monitor contention only for the managed mem-ory within Spark, our internal memory-contention test-case caches a large TPCDS table (web sales) in mem-ory just before Q43 is submitted. Let’s call this query asQmem−int.

Figure 16: Impact of induced internal memory contentionon Q43.

Observations: We now analyze the impact on Q43 forthe following four windows: (a) mem live 2 is the pe-riod where both Q43 and Qmem−int had begun execu-tion together (Q43 had not started in mem live 1), (b)mem live 3 shows a window where some other querieshad entered the workload, and both Q43 and Qmem−int

were still running, (c) mem live 4 and mem live 5 are

the windows when Q43 was still running and Qmem−int

had finished execution, and (d) mem live total analyzesthe overall impact on Q43 from top-5 sources during itscomplete execution window. WhileQ43 was running con-currently with multiple other queries, Qmem−int causesmore than 40% share of the total impact received once itbegins as shown in Figure 16. Note that Q43 scans storeand store sales, whereas we cached web sales in our in-duction query.

8.3 Frequency of Data CollectionIn this experiment, we analyze the impact of varying thefrequency of collecting time-series data on the quality ofour explanations (i.e.DOR values of the sources) towardsa target query’s performance. Figure 17 shows that as theintervals reduce from 10s-2s, the accuracy of the DOR val-ues for all queries (calculated using vector distance) im-proves and approaches to our ideal case of TE (task-eventand 2s interval collection). This value is highly sensitiveto the task median runtimes of the workload. For example,a 8s heartbeat time-series collection can miss the impactson 6s tasks. Our task-event metrics collection approach,thus, enables ProtoXplore to generate more accurateblames. We next discuss the overheads associated withthis instrumentation.

Figure 17: Impact of varying intervals of metrics collectionon explanation DORs. (TE denotes the metrics collection attask-event boundaries (Section 5.5) in addition to 2s logging.

8.4 InstrumentationFigure 18c compares the runtime of our workload in Sparkwith (grey) and without (blue) our instrumentation forpublishing time-series for the metrics listed in Section 5.4.The heartbeat was set to 2s and task-event metrics collec-tion was set to true - a scenario of maximum load that ourinstrumentation can put.

Since every node in Levels 2, 3 and 4 corresponds toa particular target stage, the subgraph formed by thesenodes for one target stage at Level 1 is disjoint from thesubgraph formed by the nodes for another target stage.They are connected back at Level 5 if multiple targetstages execute concurrently with the same source stage.This property enables us to construct the subgraphs from

13

(a) (b) (c)Figure 18: Time taken by the Graph Constructor (Part (a)) and the Explanations Generator (part (b)) modules in varioussteps. Part (c) shows the overhead of our instrumentation on the runtime of queries in the workload.

Level 0 to Level 4 in parallel. Figure 18a shows thetime taken in parsing time-series data and creating ourdata model, constructing graph, and times to update theedge weights (IF) and node weights (DOR). Figure 18bpresents the times spent for different types of analysis al-gorithms. From our experience, the runtime of this GraphConstructor module dominates the overall latency of out-putting explanations in ProtoXplore.

User-StudyNovice Skilled ExpertM P M P M P

Easy 28 80 0 100 0 75Medium 14 40 0 66 16 60Hard 16 70 0 66 33 66

Table 3: Percent correct answers - Manual (M) vsProtoXplore (P)8.5 User StudyWe performed a user study involving participation fromusers with varying levels of expertise in databases, Sparkand in using system monitoring tools like Ganglia.Briefly, we categorize our users based on their combinedexperience in all three of these categories as (a) Novice(e.g. undergrad or masters database student or users with≤ 1 yr experience), (b) Skilled (e.g. industry profession-als with 1-3 yrs experience) and (c) Expert (e.g. DB re-searcher or industry professionals with more than 3 yrsexperience). As such, we had 4 Experts, 2 Skilled and4 Novice users. This web-based tool enabled the usersto analyze a pre-executed micro-benchmark to answeran online questionnaire consisting of 10 multiple-choicequestions with mixed difficulty levels. We categorizedour questions as Easy, Medium and Hard based on thenumber of steps involved and the time it took to answerthem manually by one of our Experts. At the begin-ning of study, each user was randomly assigned 5 ques-tions to be answered with manual analysis using exist-ing monitoring tools [11, 7]. The users were then shownthe ProtoXplore user-interface (UI) to answer the restof the 5 questions. Each question was presented pro-gressively and the time taken to answer it was recordedin order to distinguish between genuine and casual par-ticipants. Note that in the ProtoXplore UI, we pur-

posefully refrained from showing the users the final an-swers generated by ProtoXplore; instead, we pre-sented them with plots based on (a) the measured wait-time and data processed values and (b) consolidated DORvalues at each level that aided the users in selecting theiranswers using ProtoXplore tool. This activity enabledus to match the answers generated from ProtoXplorewith those selected by our Expert users with manual anal-ysis.

In total, users attempted 54% of our questions requir-ing manual analysis since they gave up on the remain-ing questions, whereas they answered 98% of the ques-tions using ProtoXplore interface. They got 37% an-swers correct with manual analysis, and 69% right us-ing ProtoXplore. Table 3 summarizes the resultsbased on levels of expertise. It shows that using onlyvisual aids on impact values and resource-to-host RATPheatmaps, the accuracy of selecting correct answers in-creased for each type of user compared to manual anal-ysis. One interesting observation from our user-studyis that once the users were finally given an opportunityto compare and review their chosen answers with thosegenerated by ProtoXplore supplemented with a rea-soning, they agreed to accept the output generated byProtoXplore for 90% of the mis-matched answers. Inour results, the time taken by the users in answering thequestions manually was about 19% lesser than the timethey took to answer using ProtoXplore UI. When weexplored further to understand this counter-intuitive re-sult, we noticed that irrespective of user expertise, mostof them spent significant time in answering the first one ortwo questions but then gave up on the answers too easilyfor later questions. Whereas, they spent time in exploringthe ProtoXplore UI to attempt more answers.

9 ConclusionResource Interferences due to concurrent executions areone of the primary and yet highly mis-diagnosed causesof query slowdowns in shared clusters today. This paperdiscusses some challenges in detecting accurate causes ofcontentions, and illustrates with examples why blame at-tribution using existing methodologies can be inaccurate.

14

To this effect, we propose a theory for quantifying blamefor slowdown, and present techniques to filter genuineconcurrency related slowdowns from other known and un-known issues. We further showed how our graph-basedframework allows for consolidation of blame and generateexplanations allowing an admin to explore the contentionsand contributors of these contentions systematically.

References[1] A User Study for Evaluating ProtoXplore. http:

//152.3.145.69:9007/protoXploreFE/.

[2] Apache Ambari. http://ambari.apache.org.

[3] Apache Hadoop Capacity Scheduler. http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html.

[4] Apache Parquet. https://parquet.apache.org.

[5] Collection of small tips in further an-alyzing your hadoop cluster. https://www.slideshare.net/Hadoop_Summit/t-325p210-cnoguchi.

[6] Dr. Elephant. http://www.teradata.com.

[7] Ganglia Monitoring System. http://ganglia.info.

[8] iQCAR: A Demonstration of an Inter-QueryContention Analyzer for Cluster Computing Frame-works. https://users.cs.duke.edu/

˜sudeepa/SIGMOD2018-iQCAR-demo.pdf.

[9] Netflix and Stolen Time. https://www.sciencelogic.com/blog/netflix-steals-time-in-the-cloud-and-from-users.

[10] Oracle Autonomous Database. https://www.oracle.com/database/autonomous-database/index.html.

[11] Spark Monitoring and Instrumentation.http://spark.apache.org/docs/latest/monitoring.html.

[12] The Noisy Neighbor Problem. https://www.liquidweb.com/blog/why-aws-is-bad-for-small-organizations-and-users/.

[13] Understanding AWS stolen CPU andhow it affects your apps. https://www.datadoghq.com/blog/understanding-aws-stolen-cpu-and-how-it-affects-your-apps/.

[14] TPC Benchmark™DS . http://www.tpc.org/tpcds/, 2016. [Online; accessed 01-Nov-2016].

[15] N. Borisov, S. Babu, S. Uttamchandani, R. Routray,and A. Singh. Why did my query slow down? arXivpreprint arXiv:0907.3183, 2009.

15

[16] P. Buneman, S. Khanna, and W. C. Tan. Why andwhere: A characterization of data provenance. InProceedings of the 8th International Conference onDatabase Theory, pages 316–330, 2001.

[17] C. Curino, D. E. Difallah, C. Douglas, S. Krishnan,R. Ramakrishnan, and S. Rao. Reservation-basedscheduling: If you’re late don’t blame us! In Pro-ceedings of the ACM Symposium on Cloud Comput-ing, pages 1–14. ACM, 2014.

[18] A. Datta, D. Garg, D. Kaynar, D. Sharma, andA. Sinha. Program actions as actual causes: A build-ing block for accountability. In 2015 IEEE 28thComputer Security Foundations Symposium, pages261–275. IEEE, 2015.

[19] K. Dias, M. Ramacher, U. Shaft, V. Venkataramani,and G. Wood. Automatic performance diagnosis andtuning in oracle. In CIDR, pages 84–94, 2005.

[20] H. Herodotou, H. Lim, G. Luo, N. Borisov, L. Dong,F. B. Cetin, and S. Babu. Starfish: a self-tuning sys-tem for big data analytics. In Cidr, volume 11, pages261–272, 2011.

[21] M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fet-terly. Dryad: distributed data-parallel programsfrom sequential building blocks. In ACM SIGOPSOperating Systems Review, volume 41, pages 59–72.ACM, 2007.

[22] S. A. Jyothi, C. Curino, I. Menache, S. M. Narayana-murthy, A. Tumanov, J. Yaniv, I. Goiri, S. Krishnan,J. Kulkarni, and S. Rao. Morpheus: towards auto-mated slos for enterprise clusters. In Proceedingsof OSDI16: 12th USENIX Symposium on Operat-ing Systems Design and Implementation, page 117,2016.

[23] P. Kalmegh, S. Babu, and S. Roy. Analyzingquery performance and attributing blame for con-tentions in a cluster computing framework. CoRR,abs/1708.08435, 2017.

[24] N. Khoussainova, M. Balazinska, and D. Suciu.Perfxplain: debugging mapreduce job performance.PVLDB, 5(7):598–609, 2012.

[25] A. Meliou, W. Gatterbauer, K. F. Moore, and D. Su-ciu. The complexity of causality and responsibil-ity for query answers and non-answers. PVLDB,4(1):34–45, 2010.

[26] K. Ousterhout, R. Rasti, S. Ratnasamy, S. Shenker,and B.-G. Chun. Making sense of performance in

data analytics frameworks. In 12th USENIX Sympo-sium on Networked Systems Design and Implemen-tation (NSDI 15), pages 293–307. USENIX Associ-ation, 2015.

[27] S. Roy, A. C. Konig, I. Dvorkin, and M. Kumar. Per-faugur: Robust diagnostics for performance anoma-lies in cloud services. In 2015 IEEE 31st Inter-national Conference on Data Engineering, pages1167–1178. IEEE, 2015.

[28] S. Roy and D. Suciu. A formal approach to find-ing explanations for database queries. In SIGMOD,pages 1579–1590, 2014.

[29] spark-sql-perf team. Spark SQL Performance.https://github.com/databricks/spark-sql-perf, 2016. [Online; accessed01-Nov-2016].

[30] D. Van Aken, A. Pavlo, G. J. Gordon, and B. Zhang.Automatic database management system tuningthrough large-scale machine learning. In Proceed-ings of the 2017 ACM International Conferenceon Management of Data, pages 1009–1024. ACM,2017.

[31] V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agar-wal, M. Konar, R. Evans, T. Graves, J. Lowe,H. Shah, S. Seth, et al. Apache hadoop yarn: Yetanother resource negotiator. In Proceedings of the4th annual Symposium on Cloud Computing, page 5.ACM, 2013.

[32] E. Wu and S. Madden. Scorpion: Explaining awayoutliers in aggregate queries. PVLDB, 6(8), 2013.

[33] D. Y. Yoon, N. Niu, and B. Mozafari. Dbsher-lock: A performance diagnostic tool for transac-tional databases. In Proceedings of the 2016 Inter-national Conference on Management of Data, SIG-MOD ’16, pages 1599–1614, New York, NY, USA,2016. ACM.

[34] M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmele-egy, S. Shenker, and I. Stoica. Delay scheduling:a simple technique for achieving locality and fair-ness in cluster scheduling. In Proceedings of the 5thEuropean conference on Computer systems, pages265–278. ACM, 2010.

[35] M. Zaharia, M. Chowdhury, M. J. Franklin,S. Shenker, and I. Stoica. Spark: Cluster comput-ing with working sets. In Proceedings of the 2NdUSENIX Conference on Hot Topics in Cloud Com-puting, HotCloud’10, pages 10–10, Berkeley, CA,USA, 2010. USENIX Association.

16

[36] X. Zhang, E. Tune, R. Hagmann, R. Jnagal,V. Gokhale, and J. Wilkes. Cpi 2: Cpu performanceisolation for shared compute clusters. In Proceed-ings of the 8th ACM European Conference on Com-puter Systems, pages 379–391. ACM, 2013.

17