Impala Resource Management - OUTDATED

1 © Cloudera, Inc. All rights reserved.

Impala Resource Management: A Brief Overview MaAhew Jacobs | @maAjacobs November 2015 Relevant through Impala 2.2/CDH5.4


Impala Resource Management: Overview

• Problem: how to best uIlize cluster resources State of the world as of Impala 2.2/CDH5.4 • Within Impala • READY FOR USE: Built-‐in Admission Control (introduced in Impala 1.3/CDH 5.0)

• Between Impala and the rest of the world • READY FOR USE: “StaIc ParIIoning” from Cloudera Manager • NOT READY: IntegraIon with YARN

•  Experimental integraIon shipped in Impala 1.3/CDH 5.0 •  Some known issues exist, do not use it today! More on this later… •  We’re acIvely working on this, stay tuned!


Talk Overview

This is a very brief overview! Many details we can’t cover in 20min L • How to be successful today (including with Impala 2.3/CDH5.5) • Overview of Impala on YARN

• Architecture • Why you can’t use it yet • How it might look when you can


“Resource Management” Today

• Use one or both of: • StaIc ParIIoning with Cloudera Manager (also called “StaIc Resource Pools”) •  Impala’s built in Admission Control

•  StaIc ParIIoning: dedicate resources for Impala, HBase, YARN, etc. • Easy to use and works well. Set up by Cloudera Manager, uses cgroups • E.g. Impala gets 100GB/30% CPU, HBase gets 50GB/20% CPU, etc.

• Admission Control: throAle Impala queries • Set a limit on the max # queries or max memory used by those queries • E.g. queue queries once more than 20 queries are running concurrently, or queue once more than 100GB is used


When to Use AC? StaIc ParIIoning?

With Admission Control Without Admission Control

With Sta2c Par22oning

•  Using Impala with other systems (e.g. Hive, Spark) and need to guarantee each get resources

•  Heavy Impala workload, need to make sure queries aren’t stepping on each other

•  Using Impala with other systems and need to guarantee each get resources

•  Light to moderate Impala workload, not using all available resources yet

Without Sta2c Par22oning

•  Impala only cluster, or other systems have very light, non-‐compeIng workloads

•  Heavy Impala workload, need to make sure queries aren’t stepping on each other

•  Enough cluster resources are available for all workloads to consume as much as necessary


(Aside: A Plethora of Mem Limits) • Process (impalad) memory limit

•  Max memory the process can use across all queries. When a query consumes memory such that the process hits this limit the query is killed

•  Set with the “-‐-‐mem_limit” impalad command-‐line argument, or “Impala Daemon Memory Limit” in CM. The value is specified in terms of single-‐impalad memory.

• Pool (admission control) memory limit •  Max memory the queries in a pool/queue can use. The value is used only to admit queries, not enforced once queries are admiAed. The value is specified as the cluster-‐wide limit, i.e. aggregate limit across all impalads.

•  hAp://www.cloudera.com/content/cloudera/en/documentaIon/cloudera-‐impala/latest/topics/impala_admission.html

• Query (query opIon) memory limit •  Max memory a query can use; if a query uses more than it may have to be killed (if it can’t spill). •  Set via the “set mem_limit=Xg” query opIon. Can set a default query opIon via impalad command-‐line arguments (see the next slide).

•  The value is specified in terms of single-‐impalad memory, e.g. Xg per node •  hAp://www.cloudera.com/content/cloudera/en/documentaIon/cloudera-‐impala/latest/topics/impala_mem_limit.html


Important! AC with Mem Limits is Tricky

• Admission based on pool memory limits will use: •  the query memory limit if it is set (set MEM_LIMIT=Xg;) • Otherwise falls back to an esImate from planning, this is usually wrong!

• Do not use memory limits unless you set query memory limits • Consider serng a default value for the ‘mem_limit’ query opIon • Set via the ‘-‐-‐default_query_opIons’ impalad argument • E.g. -‐-‐default_query_options='mem_limit=5g' • Can sIll override the default with the ‘set mem_limit=X;’ query opIon.

• Picking a good memory limit is hard, use CM’s charts to help understand your workload


“Resource Management” Today, Summary

• Today: Use Admission Control and StaIc ParIIoning • We skipped over a lot of details, see the docs for more informaIon •  Impala Admission Control: hAp://www.cloudera.com/content/cloudera/en/documentaIon/cloudera-‐impala/latest/topics/impala_admission.html • “StaIc ParIIoning” in Cloudera Manager: (also called “StaIc Service Pools”) hAp://www.cloudera.com/content/cloudera/en/documentaIon/core/latest/topics/cm_mc_service_pools.html

• Ask us quesIons on impala-‐[email protected]


Impala on YARN

• YARN is a “resource negoIator” that helps share cluster resources within Hadoop • Works well for MapReduce and similar batch-‐oriented processing engines • Doesn’t work well for services/frameworks that need:

•  Long running processes • Gang scheduling • Very low-‐latency scheduling requirements

• Doesn’t work so well for Impala •  (And also HBase, MPI, Presto, custom apps, etc.)


Llama to the Rescue

•  Llama = Long Lived ApplicaIon MAster • On github: hAp://cloudera.github.io/llama/index.html • An interface between: • YARN’s ApplicaIonMaster (AM) model (batch jobs where tasks are each a process, coordinated by an AM) •  Impala’s low-‐latency, in-‐process query model

•  Llama provides: • Gang-‐scheduling • “Container” caching (to reduce resource acquisiIon cost)


How Llama fits in

11


How Llama fits in

12


How Llama fits in

13


How Llama fits in

14


How Llama fits in

15


Gang scheduling

•  YARN returns resources in a trickle, as they become available •  For MR this is perfect, as tasks are mostly independent (and checkpoint to disk) •  For low-‐latency queries, we require all resources to be available at once so that query tasks can stream results to one another •  Llama buffers resources between YARN and Impala to make resource requests appear atomic and indivisible

16


Resource caching

• Every container requires YARN to make an expensive resource allocaIon decision

• We ask Llama to cache resources between requests

• Containers stay in their queue in Llama, unIl YARN forcefully reclaims them

17


Impala on YARN: Current Status

• Experimental integraIon was shipped in Impala 1.4 / CDH 5.0 • Not ready for use yet!

• A number of known bugs, see umbrella JIRA IMPALA-‐2370 to track •  Some (but not all) important fixes in upcoming Impala 2.3 / CDH 5.5 release • Ongoing scale and performance tesIng work needed to provide guidance

•  In a future release (post-‐Impala 2.3), we will be able to recommend usage for some workloads, w/ guidance


Thank you @maAjacobs

Impala Resource Management - OUTDATED

Software

Transcript of Impala Resource Management - OUTDATED