Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar...
Transcript of Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar...
![Page 1: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/1.jpg)
Val Bercovici Cloud & Big Data Czar NetApp
Top 10 Things We Learned About
Hadoop
![Page 2: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/2.jpg)
Val Bercovici @valb00
![Page 3: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/3.jpg)
What I hope to accomplish today...
&
![Page 4: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/4.jpg)
...and avoid this.
![Page 5: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/5.jpg)
10. Both Kinds of Hadoop Customers
Data is their business Data supports their business
![Page 6: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/6.jpg)
9. Technology Marches On
MRPerf: A Simulator for MapReduce
![Page 7: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/7.jpg)
8. SuperNodes to Dominate
http://www.slideshare.net/Hadoop_Summit/dynamic-namespace-partitioning-with-giraffa-file-system
![Page 8: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/8.jpg)
7. C-Level Fear & Envy
¡ 2010 – “What is Hadoop?” – Google, Wikipedia, Slideshare
¡ 2011 – “How can Hadoop help us?” – “Let’s bring in a consultant”
¡ 2012 – “Our competition is using
Hadoop” – “I can’t afford to be behind”
![Page 9: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/9.jpg)
6. Cluster – Get Healthy, Stay Healthy
9
Source: Garrett, Brian and Lockner, Julie, “NetApp Open Solution for Hadoop”, ESG Report, May 2012, http://bit.ly/LyYG0t
Network Overhead Useful Work
Availability and Resiliency
Burst Handling and Queuing
Oversubscription Ratio
Data Node Network Speed
Network Latency
Source: Cisco: http://bit.ly/yL54Ts
![Page 10: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/10.jpg)
5. It Takes a Village
10
Sensors
Applications
Logs
Location/GPS
Mobile Devices
Storage (All other storage, i.e. internal DAS)
Content Repositories
Shared Storage Infrastructure
Storage File Systems
Data Management
Analytics
Applications
Reporting/Dashboard/Visualization
ETL
OLAP
OLTP
Other Data Sources
OLAP ETL
Storage Data Management
![Page 11: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/11.jpg)
4. Hadoop has crossed Verticals
![Page 12: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/12.jpg)
3. HFDS I/O Profile is Diverse
MRPerf
![Page 13: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/13.jpg)
2. Hadoop often over-provisioned
MRPerf
![Page 14: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/14.jpg)
1. Big Brother = Big Compliance
Full Motion Video Capture
Digital Video Surveillance
![Page 15: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/15.jpg)
0. Fast Data is the new Big Data
Apache Drill
![Page 16: Top 10 Things We Learned About Hadoop - SNIA | Advancing ... · Val Bercovici Cloud & Big Data Czar NetApp Top 10 Things We Learned About Hadoop](https://reader030.fdocuments.net/reader030/viewer/2022011806/5bf0ce2c09d3f2ec148c9654/html5/thumbnails/16.jpg)