MesosCon 2016: Frameworks in Harmony
-
Upload
brenden-matthews -
Category
Software
-
view
304 -
download
2
Transcript of MesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 2
BRENDEN MATTHEWSEngineer
Core Mesos committer, contributor to numerous Mesos frameworks, and various other projects in the Mesos
ecosystem. He spends his free time contemplating existentialism, cooking steak, and craigslist furniture hacking.
@brndnmtthws
VISHNU MOHANSolutions Engineer
Works closely with customers in the field on Performance and Automation. When he’s not analyzing sandbox logs or coding
against some API for fun, he may be found rock climbing, hiking, running or scuba diving.
@vm_mesos
© 2016 Mesosphere, Inc. All Rights Reserved. 4
● Problem Definition, Motivations and Applications● Intro to Roles, Weights, Quotas, Reservations, Revocation & Oversubscription● Theory
● Dominant Resource Fairness (DRF)● Hierarchical Dominant Resource Fairness (H-DRF)
● Mesos Quotas● Scenarios and putting it all together● Demo on
AGENDAMesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 5
MOTIVATIONS/APPLICATIONS
MesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 6
MOTIVATIONS
Running multiple environments on one cluster
Guaranteeing resource availabilityAPPLICATIONS
Multitenancy
Isolation
Introducing new frameworks
MOTIVATIONS/APPLICATIONSMesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 8
ROLES
A method of identifying frameworks for use with reservations, auth, ACLs, and other Mesos features.
WEIGHTS
Control the relative share of resources that is offered to different roles
INVERSE OFFERS
Mechanism to reclaim resources for maintenance, reallocation and pre-emption
DEFINITIONSMesosCon 2016: Frameworks in Harmony
RESERVATIONS
Mechanism to reserve resources on specific agents (Static and Dynamic)
QUOTAS
Lay away resources for consumption via cluster-wide dynamic reservation
OVERSUBSCRIPTION
Temporarily take advantage of unused resources
© 2016 Mesosphere, Inc. All Rights Reserved. 9
DOMINANT RESOURCE FAIRNESS(DRF)
MesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 10
DRF
Core resource allocation algorithm for Mesos
Generalization of Max-Min Fairness to multiple resources
WHY MAX-MIN FAIRNESS?
Generality
Ability to provide performance isolationhttps://www.cs.berkeley.edu/~alig/papers/drf.pdf
WEIGHTED MAX-MIN FAIRNESS
Priority
Reservation
Deadline-Based Allocation
User is guaranteed to receive her fair share irrespective of the demands of others
Applicable to a wide variety of resources
DOMINANT RESOURCE FAIRNESSMesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 11
DRF CORE CONCEPTS
In a multi-resource environment, the allocation of a user should be determined by the user’s “dominant share”, which is the maximum share that the user has been allocated of any resource
DRF then seeks to maximize the minimum dominant share across all users
DRF CORE PROPERTIES
Sharing Incentive
Strategy Proofness
Pareto Efficiency
Envy Freeness
NICE-TO-HAVE PROPERTIES
Single Resource Fairness
Bottleneck Fairness
Population Monotonicity
Resource Monotonicity
DOMINANT RESOURCE FAIRNESS contd.MesosCon 2016: Frameworks in Harmony
© 2015 Mesosphere, Inc. All Rights Reserved. 12
EXAMPLE
Total Capacity: 9 CPUs, 18 GB RAM
ROLE A
<1 CPU, 4 GB>, Dominant Resource: Mem
<x CPU, 4x GB>, Dominant Share: 2/9 (i.e., 4/18)
ROLE B
<3 CPUs, 1 GB>, Dominant Resource: CPU
<3y CPU, y GB>, Dominant Share: 1/3 (i.e., 3/9)
DRF IN ACTIONMesosCon 2016: Frameworks In Harmony
SOLUTION
x = 3, y = 2
© 2015 Mesosphere, Inc. All Rights Reserved. 13
DRF IN ACTION contd.MesosCon 2016: Frameworks In Harmony
© 2015 Mesosphere, Inc. All Rights Reserved. 14
DRF ALLOCATION SCHEDULE (Quantum 1)MesosCon 2016: Frameworks in Harmony
ScheduleRole A <1 CPU, 4 GB> Role B <3 CPU, 1 GB> CPU
Total Alloc.RAM
Total Alloc.res. shares dom. share res. shares dom. share
Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18
© 2015 Mesosphere, Inc. All Rights Reserved. 15
DRF ALLOCATION SCHEDULE (Quantum 2)MesosCon 2016: Frameworks in Harmony
ScheduleRole A <1 CPU, 4 GB> Role B <3 CPU, 1 GB> CPU
Total Alloc.RAM
Total Alloc.res. shares dom. share res. shares dom. share
Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18
Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18
© 2015 Mesosphere, Inc. All Rights Reserved. 16
DRF ALLOCATION SCHEDULE (Quantum 3)MesosCon 2016: Frameworks in Harmony
ScheduleRole A <1 CPU, 4 GB> Role B <3 CPU, 1 GB> CPU
Total Alloc.RAM
Total Alloc.res. shares dom. share res. shares dom. share
Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18
Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18
Role A <2/9, 8/18> 4/9 <3/9, 1/18> 1/3 5/9 9/18
© 2015 Mesosphere, Inc. All Rights Reserved. 17
DRF ALLOCATION SCHEDULE (Quantum 4)MesosCon 2016: Frameworks in Harmony
ScheduleRole A <1 CPU, 4 GB> Role B <3 CPU, 1 GB> CPU
Total Alloc.RAM
Total Alloc.res. shares dom. share res. shares dom. share
Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18
Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18
Role A <2/9, 8/18> 4/9 <3/9, 1/18> 1/3 5/9 9/18
Role B <2/9, 8/18> 4/9 <6/9, 2/18> 2/3 8/9 10/18
© 2015 Mesosphere, Inc. All Rights Reserved. 18
DRF ALLOCATION SCHEDULE (Quantum 5) - CPU Maxed OutMesosCon 2016: Frameworks in Harmony
ScheduleRole A <1 CPU, 4 GB> Role B <3 CPU, 1 GB> CPU
Total Alloc.RAM
Total Alloc.res. shares dom. share res. shares dom. share
Role B <0,0> 0 <3/9, 1/18> 1/3 3/9 1/18
Role A <1/9, 4/18> 2/9 <3/9, 1/18> 1/3 4/9 5/18
Role A <2/9, 8/18> 4/9 <3/9, 1/18> 1/3 5/9 9/18
Role B <2/9, 8/18> 4/9 <6/9, 2/18> 2/3 8/9 10/18
Role A <3/9, 12/18> 2/3 <6/9, 2/18> 2/3 1 14/18
© 2015 Mesosphere, Inc. All Rights Reserved. 19
DRF ALLOCATION AND RESOURCE CONSUMPTIONMesosCon 2016: Frameworks In Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 20
HIERARCHICALDOMINANT RESOURCE FAIRNESS(H-DRF)https://www.cs.berkeley.edu/~alig/papers/h-drf.pdf
MesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 22
USE CASES
● Ensuring that resources on a particular agent are only offered to a particular framework
● Partitioning a cluster between organizations: resources reserved for use by Organization A will only be offered to frameworks that have registered using Organization A’s role (also, Reservations)
● Expressing that one group of frameworks has “higher priority” (and offered more resources) than another group of frameworks (also, Weights)
● Setting a guaranteed resource allocation for one or more frameworks belonging to a role (also, Quotas)
● Ensuring that persistent volumes created by one framework are not offered to frameworks registered with a different role
MESOS ROLESMesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 24
QUOTA
Quotas are a mechanism for dynamically reserving resources.
WHY QUOTA?
DRF may offer resources to frameworks beyond their fair share. Operators can use quotas to ensure that frameworks always receive enough resources.
MESOS QUOTASMesosCon 2016: Frameworks in Harmony
QUOTA CORE CONCEPTS
Quotas provide a mechanism for guaranteeing resources for roles. Frameworks may register with that role, and will be allocated at least the specified amount of resources for that role.
In other words, quotas are an implementation of cluster-wide dynamic resource reservations by role.
© 2016 Mesosphere, Inc. All Rights Reserved.
LIMITATIONS
Can’t specify granularity or constraints (e.g., can’t distribute resources evenly across fault domains).
Cannot update an existing quota (must be removed and re-added).
The current implementation is both the minimum and the limit. No more offers will be sent once all reserved resources are allocated for that role.
25
QUOTA FEATURES
Minimum Guarantee
Dynamically Allocated
Configured at Runtime
Applies Per Role
Cluster-wide
Easy Reasoning
Complementary to Dynamic Reservations
MESOS QUOTAS contd.MesosCon 2016: Frameworks in Harmony
© 2015 Mesosphere, Inc. All Rights Reserved. 26
QUOTA USAGEMesosCon 2016: Frameworks in Harmony
core@ip-10-0-6-151 ~ $ curl -d '{"role":"myRole","guarantee":[{"name":"cpus","type":"SCALAR","scalar":{"value":1}},{"name":"mem","type":"SCALAR","scalar":{"value":100}}]}' -X POST http://master.mesos:5050/quota
core@ip-10-0-6-151 ~ $ curl http://master.mesos:5050/quota{"infos":[{"guarantee":[{"name":"cpus","role":"*","scalar":{"value":1.0},"type":"SCALAR"},{"name":"mem","role":"*","scalar":{"value":100.0},"type":"SCALAR"}],"role":"myRole"}]}
core@ip-10-0-6-151 ~ $ curl -X DELETE http://master.mesos:5050/quota/myRole
core@ip-10-0-6-151 ~ $ curl http://master.mesos:5050/quota{}
© 2016 Mesosphere, Inc. All Rights Reserved. 28
CONCLUSIONS & FUTURE WORK
MesosCon 2016: Frameworks in Harmony
© 2016 Mesosphere, Inc. All Rights Reserved. 29
Q&AMesosCon 2016: Frameworks in HarmonyThursday, 11:00am: Quota: Reliable
Guarantees - Jörg Schad & Joris Van Remoortere, Mesosphere - Grand Mesa A, http://sched.co/6jtn
Thursday, 12:00pm: Maintenance: Automating Collaboration Between Frameworks and Operators - Joseph Wu, Mesosphere - Chasm Creek (this room), http://sched.co/6jtt
Thursday, 2:00pm: Resource Allocation in Apache Mesos: Present and Future - Benjamin Mahler, Mesosphere - Grand Mesa A, http://sched.co/6jtm
© 2016 Mesosphere, Inc. All Rights Reserved. 30
VISIT OUR BOOTHLocated at D1, near the main entrance
Learn more by visiting dcos.io and mesosphere.com
Vishnu’s Office HoursThursday 10-12:50pm
© 2016 Mesosphere, Inc. All Rights Reserved. 34
ReferencesMesosCon 2016: Frameworks in Harmony
DRF: https://www.cs.berkeley.edu/~alig/papers/drf.pdf
H-DRF: http://www.cs.berkeley.edu/~alig/papers/h-drf.pdf
https://www.usenix.org/conference/nsdi11/dominant-resource-fairness-fair-allocation-multiple-resource-types
Mesos Docs:
https://www.cs.berkeley.edu/~alig/papers/mesos.pdf
https://mesos.apache.org/documentation/latest/roles/
https://mesos.apache.org/documentation/latest/weights/
https://mesos.apache.org/documentation/latest/attributes-resources/
https://mesos.apache.org/documentation/latest/reservation/
https://mesos.apache.org/documentation/latest/quota/
https://mesos.apache.org/documentation/latest/oversubscription/
n-level Hierarchies: https://issues.apache.org/jira/browse/MESOS-1763
© 2016 Mesosphere, Inc. All Rights Reserved.
THANK YOU!Slides: http://sched.co/6ju7
35