Zeta architecture - Hive London May15
-
Upload
mapr-technologies -
Category
Documents
-
view
131 -
download
2
Transcript of Zeta architecture - Hive London May15
![Page 1: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/1.jpg)
®© 2014 MapR Technologies 1
®
© 2014 MapR Technologies
Zeta Architecture Ted Dunning– Chief Application Architect Hive London – May 20, 2015
![Page 2: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/2.jpg)
®© 2014 MapR Technologies 2
Agenda • Current State
– History – Moving Forward
• The Next Enterprise Architecture • Business Implications • Concrete Implementations
![Page 3: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/3.jpg)
®© 2014 MapR Technologies 3 © 2014 MapR Technologies ®
Current State
![Page 4: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/4.jpg)
®© 2014 MapR Technologies 4
Study History to Prepare for the Future • A data center was built
• The servers were statically partitioned
• If we want to break the cycle we have to break the partitions and become dynamic
![Page 5: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/5.jpg)
®© 2014 MapR Technologies 5
Understanding the Why’s • Isolation of resources
– Assists in troubleshooting – Prevents the analytics team from impacting production
• Maximum throughput of an application
– Guaranteed volume (maximum): compute, memory and storage • Business Continuity
– We know exactly what is backed up, when, and where – Difficult to perfect and to test
![Page 6: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/6.jpg)
®© 2014 MapR Technologies 6
Issues with Isolated Workloads • Segregated servers lead to under utilized hardware
– Wasted capacity and energy
• Complicated processes to move data to the required processing servers – Operational impact, including extra monitoring – Time delays moving data (not real-time) – Troubleshooting time when there are issues
• Difficult to thoroughly test DEV vs. QA vs. Production – Environments have different shapes and sizes – They will not have identical configurations
![Page 7: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/7.jpg)
®© 2014 MapR Technologies 7
Goals Moving Forward • Leverage all existing hardware
• Create isolation in a different way
• Improve production operational processes
• Fix process of moving from DEV to QA to Production
• Support real-time business continuity
![Page 8: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/8.jpg)
®© 2014 MapR Technologies 8 © 2014 MapR Technologies ®
The Next “Last” Enterprise Architecture
![Page 9: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/9.jpg)
®© 2014 MapR Technologies 9
The Next Generation Enterprise Architecture • Dynamic compute resources • Common storage platform • Real-time application support • Flexible programming models • Deployment management • Solution based approach • Applications to operate a
business
* This is a pluggable architecture Distributed File System
Enterprise Applications
Global Resource Management
![Page 10: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/10.jpg)
®© 2014 MapR Technologies 10
Technologies That Work
Global Resource Management
Distributed File System
Enterprise Applications
Mesos + Myriad YARN
MapR-FS HDFS S3
Web Servers
Business Applications
![Page 11: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/11.jpg)
®© 2014 MapR Technologies 11
We Will Call This Architecture…
![Page 12: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/12.jpg)
®© 2014 MapR Technologies 12
What’s in a Name • The letter Z is the last letter in the English
alphabet, but Zeta is not the last letter of the Greek alphabet
– But this is the last generalized architecture you will need.
• Sixth letter of the Greek alphabet – Hexagon represents the 6 surrounding pieces
• Zeta represents the number 7 – 7 total components in this architecture – Components work with a global resource
manager
![Page 13: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/13.jpg)
®© 2014 MapR Technologies 13
Origin Story of the Zeta Architecture • Cultivated by Jim Scott
– Created the pretty diagrams – Put a nice name on it – Documented the concepts
• Not really a new concept
– Google pretty much pioneered these technology concepts
– They have never really discussed it cohesively in this way
![Page 14: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/14.jpg)
®© 2014 MapR Technologies 14
Zeta Architecture at Google
Global Resource Management
Distributed File System
Enterprise Applications
Borg & Omega
GoogleFS
HTTP Servers
GMail
![Page 15: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/15.jpg)
®© 2014 MapR Technologies 15 © 2014 MapR Technologies ®
Concrete Implementations
![Page 16: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/16.jpg)
®© 2014 MapR Technologies 16
Web Server Logs • Web server generates logs • Land on local disk
– Logs periodically rotated
• Shipped to other servers
• Run jobs on logs
![Page 17: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/17.jpg)
®© 2014 MapR Technologies 17
Web Server Logs • Web server generates logs
• Land on DFS – Logs still rotate – Logs now tolerant of a server
failure prior to rotation – Logs are instantaneously
available for computation
• Run jobs on logs – Data locality
![Page 18: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/18.jpg)
®© 2014 MapR Technologies 18
Advertising Platform
![Page 19: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/19.jpg)
®© 2014 MapR Technologies 19
Advertising Platform - Simplified
![Page 20: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/20.jpg)
®© 2014 MapR Technologies 20
Advertising Platform on Zeta
![Page 21: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/21.jpg)
®© 2014 MapR Technologies 21 © 2014 MapR Technologies ®
Business Implications
![Page 22: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/22.jpg)
®© 2014 MapR Technologies 22
Integration of Existing Systems • Use standards like NFS to connect existing
systems
• Pluggable security models fit into your companies current standards
• Many conventional tools work well
• Not everything works well in this model – Oracle, DB2, SQL Server, PostgreSQL, MySQL
• They tend to not support being resource managed, containers or other DFS
• Applications in this architecture can still use them • If they start supporting these technologies then things
change
®
![Page 23: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/23.jpg)
®© 2014 MapR Technologies 23
Rethink the Data Center • All Servers
– Run Mesos – Participate in the Distributed File System
• Dynamic Allocation of Resources – Spin up more web servers – Custom Business Applications – Big Data Analytics
• Data Locality – No more shipping data – Store and process the data where it was created
![Page 24: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/24.jpg)
®© 2014 MapR Technologies 24
Simplified Architecture • Less moving parts
– Less things to go wrong
• Better resource utilization – Scale any application up or down on demand
• Common deployment model (new isolation model) – Repeatability between environments (dev, qa, production)
• Shared file system – Get at the data anywhere in the cluster – Simplifies business continuity
![Page 25: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/25.jpg)
®© 2014 MapR Technologies 25
Business Continuity • Resilience
– Redundancy – High Availability – Spare Capacity
• Recovery – Snapshots – Disaster Recovery
• Contingency – Protect against the unforeseen – Multisite Capability
Production
WAN
Production Research
Datacenter 1 Datacenter 2
WAN EC2
![Page 26: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/26.jpg)
®© 2014 MapR Technologies 26
Platform-wide Security and Compliance • Authentication, Authorization, Auditing
– Users and jobs – All tiers
• Data protection – Wire-level encryption between servers – Masking
• Regulatory Compliance – Automatic expiration of “old” data – Data locality supported by distributed file system
![Page 27: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/27.jpg)
®© 2014 MapR Technologies 27
Net Benefit • Reduced operating expenses (OPEX)
– Better utilization of available capacity and data center space
• Reduced capital expenses (CAPEX) – Less total hardware needed
• Improves time to market – Streamlined deployments – Environments become consistent and predictable
• Delivers a competitive advantage – Via platform scaling – Performance improvements
![Page 28: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/28.jpg)
®© 2014 MapR Technologies 28
Recap • Saves valuable time and money
• Enables stronger business continuity capabilities • Google has been doing this for years
– Real-time is the crux of everything Google does • Time for the rest of us to operate at Google scale
– The technologies are there and they play together nicely – Process changes must occur internally to achieve this architecture
• This approach will become the “traditional” way of thinking – Don’t get beat to it by your competitors
![Page 29: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/29.jpg)
®© 2014 MapR Technologies 29
Go Forth and Implement the Zeta Architecture
![Page 30: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/30.jpg)
®© 2014 MapR Technologies 30
$50M $50M in Free Training
www.mapr.com/odt
![Page 31: Zeta architecture - Hive London May15](https://reader038.fdocuments.net/reader038/viewer/2022102816/55ada9901a28aba4748b476a/html5/thumbnails/31.jpg)
®© 2014 MapR Technologies 31
Q & A
@ted_dunning maprtech
Engage with us!
MapR
maprtech
mapr-technologies