Lessons learned from the proverbial battlefield - Hortonworks roadshow

35
from the proverbial battlefield Suhail Shergill, Scotiabank

Transcript of Lessons learned from the proverbial battlefield - Hortonworks roadshow

Page 1: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Lessons learned from the proverbial

battlefield

Suhail Shergill, Scotiabank

Anonymous
The Wisconsin Heights Battlefield is an area in Dane County, Wisconsin where the penultimate battle of the 1832 Black Hawk War occurred. The conflict was fought between the Illinois and Michigan Territory militias and Sauk chief Black Hawk and his band of warriors, who were fleeing their homeland following the Fox Wars. The Wisconsin Heights Battlefield is the only intact battle site from the Indian Wars in the U.S. Midwest. Today, the battlefield is managed and preserved by the state of Wisconsin as part of the Lower Wisconsin State Riverway. In 2002, it was listed on the U.S. National Register of Historic Places.
Page 2: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Who Am ISuhail Shergill (@suhailshergill)

• Computer Science background (Programming Languages and Machine Learning)

• create and run skunkworks teams focused on data science and technology

• technical advisor to startups

• organizer of a few technical meetups

• leading the Data Science & Model Innovation group in GRM at Scotiabank.

Page 3: Lessons learned from the proverbial battlefield - Hortonworks roadshow

ObjectiveWhat’s in scope• What is “Big Data”

• What are the challenges of “Big Data”

• How can some of these challenges be addressed – lessons learned

• What are we doing in Scotia

Page 4: Lessons learned from the proverbial battlefield - Hortonworks roadshow

“Big Data” and Hadoop

Page 5: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Hadoop

Page 6: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Challenges of Big Data

Page 7: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Feedback loops • Very “big”

• Getting “bigger” at a faster rate

• Long-term solutions need to have exponential/logarithmic characteristics

Page 8: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Feedback loops• Very “big”

• Getting “bigger” at a faster rate

• Long-term solutions need to have exponential/logarithmic characteristics

Page 9: Lessons learned from the proverbial battlefield - Hortonworks roadshow

From data to insights

Page 10: Lessons learned from the proverbial battlefield - Hortonworks roadshow

From data to insights

Page 11: Lessons learned from the proverbial battlefield - Hortonworks roadshow

No free lunches / silver bullets

Page 12: Lessons learned from the proverbial battlefield - Hortonworks roadshow

No free lunches / silver bullets

Page 13: Lessons learned from the proverbial battlefield - Hortonworks roadshow

The challenges of “Big Data”We have a very “big” problem. How do we solve it?

Page 14: Lessons learned from the proverbial battlefield - Hortonworks roadshow

How to solve it

Page 15: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Lessons learned

Page 16: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Data quality is paramount

Page 17: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Build tools

Page 18: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Teach enough to question

Page 19: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Rotations and harmonics

Page 20: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Open doors

Page 21: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Faster and shorter feedback loops

Page 22: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Summary

Page 23: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Page 24: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Page 25: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Build tools

Page 26: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Build tools

Teach enough to question

Page 27: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Build tools

Teach enough to question

Rotations and harmonics

Page 28: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Build tools

Teach enough to question

Rotations and harmonics

Open doors

Page 29: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Build tools

Teach enough to question

Rotations and harmonics

Open doors

Faster & shorter feedback loops

Page 30: Lessons learned from the proverbial battlefield - Hortonworks roadshow

SummaryNo silver bullet

Data quality is paramount

Build tools

Teach enough to question

Rotations and harmonics

Open doors

Faster & shorter feedback loops

Page 31: Lessons learned from the proverbial battlefield - Hortonworks roadshow

What we’re doing in Scotia

Page 32: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Scotiabank’s Enterprise Data Lake InitiativeScotiabank’s 2015 business strategy focuses on these priorities:

• Improving the customer experience;

• Enhancing leadership capabilities throughout the organization; and

• Improving operational efficiency and effectiveness.

• A key component of the digital strategy supporting these priorities is to leverage big data analytics in order to better understand and address customer needs and preferences.

• To this end, Scotiabank is making material investments in the Hadoop technology used to support big data analytics across a wide spectrum of companies and industries.

Page 33: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Scotiabank’s Enterprise Data Lake – Next Steps 1. EDL 1.0 :

• Initial cluster 1PB (Jan-2016) rapidly growing to accommodate more tenants

• A very good start with consistent and commoditized stack• A review of areas we can further optimize and identify gaps• A review of areas where we require higher level flexibility &

portability• A review of what made sense to be directed where to achieve

scale , yet preserve consistency• A review of where are the limiting factors : agile and repeatable

periodically every 2-3 months2. EDL 2.0:

• Need to drive velocity: refactor engineered infrastructure environment

• Need flexibility on workload: decouple compute & data• Need workload portability: next gen hybrid architecture & cloud

Page 34: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Scotiabank’s Enterprise Data Lake – Highlights 1. What we got out of EDL 1.0 :

• Regulatory & Risk Reporting (RDARR)• Consolidation of divisional data repositories• Capability for Anti Money Laundering• Capability for Asset Liability Management• Consolidation of International Banking Datawarehouses• M&A and Credit Card data acquisition and analysis

Page 35: Lessons learned from the proverbial battlefield - Hortonworks roadshow

Thank you