Re-Platforming All the Things
-
Upload
mesosphere-inc -
Category
Technology
-
view
650 -
download
0
Transcript of Re-Platforming All the Things
![Page 1: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/1.jpg)
Re-Platforming All the Things
![Page 2: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/2.jpg)
© 2015 Mesosphere, Inc.
Who in the world are you?
Thomas Rampelberg
Product Manager
@grampelberg
2
![Page 3: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/3.jpg)
© 2015 Mesosphere, Inc.
Who do you work for?
3
![Page 4: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/4.jpg)
© 2015 Mesosphere, Inc.
Interested in trying it out now?
• Mesosphere DCOS• Apache Mesos + Marathon• https://mesosphere.com/amazon/
4
![Page 5: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/5.jpg)
© 2015 Mesosphere, Inc.
• The datacenter of yesterday• The datacenter of today• The datacenter of tomorrow
Agenda
5
![Page 6: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/6.jpg)
© 2015 Mesosphere, Inc. 6
Yesterday
![Page 7: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/7.jpg)
© 2015 Mesosphere, Inc.
History of the World
7
The Mainframe
Linux + x86
Virtual Machines / Cloud
Containers
![Page 8: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/8.jpg)
© 2015 Mesosphere, Inc.
Mainframes
• Applications - 1 service• Hardware - 1 box• Redundancy - Systems Architecture• Reboot - Never• Services - 100s/”host”
8
![Page 9: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/9.jpg)
© 2015 Mesosphere, Inc.
• Applications - 2(ish) services• Hardware - 10s of boxes• Redundancy - Multiple boxes• Reboot - Occasionally• Services - 10s/”host”
Linux + x86 - Commoditization
9
![Page 10: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/10.jpg)
© 2015 Mesosphere, Inc.
Virtual Machines / Cloud
• Applications - 3+ service• Hardware - 100s of “hosts”• Redundancy - Hardware + Software• Reboot - Occasionally• Services - A couple/”host”
10
![Page 11: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/11.jpg)
© 2015 Mesosphere, Inc.
• Applications - Microservices• Hardware - 1000s of containers• Redundancy - Multiple boxes• Reboot - Often• Services - 1 per container
Containers
11
![Page 12: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/12.jpg)
© 2015 Mesosphere, Inc. 12
Today
![Page 13: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/13.jpg)
© 2015 Mesosphere, Inc.
Pets
• They’re Individuals• You have to clean up• They fail eventually
13
![Page 14: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/14.jpg)
© 2015 Mesosphere, Inc.
• Treated as a herd• Cow pies happen• They fail eventually
Cattle
14
![Page 15: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/15.jpg)
© 2015 Mesosphere, Inc.
How are these different?
• 144 Cores• 1.15 TB Memory
15
• 144 Cores• 1.15 TB Memory
![Page 16: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/16.jpg)
© 2015 Mesosphere, Inc.
“Sixty-four cores or 128 cores in a single computer looks a lot like 64 or 128
machines in a datcenter” - Benjamin Hindman
16
![Page 17: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/17.jpg)
© 2015 Mesosphere, Inc.
One Big Computer
• Manage Resources Globally• Manage Failures Globally• Scale Across Hosts
17
![Page 18: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/18.jpg)
© 2015 Mesosphere, Inc.
• Scalability• Multi-tennancy• Efficiency• Reliability & availability• On-demand resources
What do you get?
18
![Page 19: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/19.jpg)
© 2015 Mesosphere, Inc.
Who’s doing it?
Closed Source
• Google Borg• Google Omega
19
![Page 20: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/20.jpg)
© 2015 Mesosphere, Inc.
Who’s doing it?
20
Open Source
• Apache Mesos• Apache Hadoop (YARN)• Kubernetes
![Page 21: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/21.jpg)
© 2015 Mesosphere, Inc.
Apache Mesos
21
![Page 22: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/22.jpg)
© 2015 Mesosphere, Inc.
Overview
• Borg is awesome, let’s do something better
• Came out of UC Berkeley's AMP Lab• First paper in 2009• Running at Twitter and Apple
22
![Page 23: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/23.jpg)
© 2015 Mesosphere, Inc.
• Separate resources from placement• Focus on running batch jobs and
services• All about the scheduler
What’s Different?
23
![Page 24: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/24.jpg)
© 2015 Mesosphere, Inc.
Apache Hadoop (YARN)
24
![Page 25: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/25.jpg)
© 2015 Mesosphere, Inc.
Overview
• Hadoop needed a better scheduler• YARN became the foundation for
Hadoop 2.0• Paper came out in 2013• Running at Yahoo and LinkedIn
25
![Page 26: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/26.jpg)
© 2015 Mesosphere, Inc.
• Centralize the resource and placement decisions
• Optimize for somewhat long batch jobs (eg. MR)
• Bake the storage layer into the solution
What’s Different?
26
![Page 27: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/27.jpg)
© 2015 Mesosphere, Inc.
Kubernetes
27
![Page 28: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/28.jpg)
© 2015 Mesosphere, Inc.
Overview
• Google took everything they learned from Borg and Omega
• Kubernetes was open source in 2014, just went 1.0 last month
• Running in production ….. ?
28
![Page 29: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/29.jpg)
© 2015 Mesosphere, Inc.
• Containers first, only containers• Pods are a first class citizen• Labels, labels everywhere
What’s Different?
29
![Page 30: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/30.jpg)
© 2015 Mesosphere, Inc. 30
Tomorrow
![Page 31: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/31.jpg)
© 2015 Mesosphere, Inc.
Stateful Applications
• Persistent Volumes• Durable Pods• Distributed File Systems
31
![Page 32: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/32.jpg)
© 2015 Mesosphere, Inc.
• Networking virtualization• Flannel• Weave• Calico
• Dynamic load balancing• IPv6
Service Discovery
32
![Page 33: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/33.jpg)
© 2015 Mesosphere, Inc.
Security
• Not everyone is a small startup• Compliance is important• Long tail of process• Containers are still early• Schedule policy, automatically
33
![Page 34: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/34.jpg)
© 2015 Mesosphere, Inc.
• Sizing is hard• Not all processes need guarantees• Let the scheduler sort it out
Oversubscription
34
![Page 35: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/35.jpg)
© 2015 Mesosphere, Inc. 35
Next Steps
![Page 36: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/36.jpg)
© 2015 Mesosphere, Inc.
Getting Started
• Apache Mesos - http://mesosphere.com/product
• Apache Hadoop (YARN) - http://bit.ly/1K13E4S
• Kubernetes - http://kubernetes.io/gettingstarted/
36
![Page 37: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/37.jpg)
© 2015 Mesosphere, Inc.
• http://chat.mesosphere.com/• http://mesos.apache.org/community/• https://wiki.apache.
org/hadoop/HowToContribute• http://kubernetes.io/community/
Get Involved!
37
![Page 38: Re-Platforming All the Things](https://reader031.fdocuments.net/reader031/viewer/2022021422/586e8c931a28aba0038b8511/html5/thumbnails/38.jpg)
© 2015 Mesosphere, Inc.
Get Awesome
38