Big Data, Big Disappointment

Big Data, Big Disappointment

A diagnosis and prescription (sort of) for (somewhat) successful analytics efforts in medium to large firms in

Mexico

(c) 2015 Jesus Ramos 1

“Big Data has arrived, but big insights have not” - “Big data: are we making a big mistake? Tim Harford. Financial Times.


And with all the money Gartner says we’re to fork over, the question is…

Why?


In mature businesses, mostly because…

•  False positives are ignored

•  Correlation implies causation

•  We don’t care about sampling

•  Machine Learning for all


From “8 Reasons why Big Data projects fail”. Matt Asay. InformationWeek. 8/714

And in the rest of us, because…

We don’t understand what Big Data is!

So…we need definitions:


BD is a 2-part deal


Big Data Technology for storing and processing large

amounts of data

Analytics The insights gained from such large data

“Without ‘analytics’, Big Data is a sleeping giant!”

- me

Don’t talk about ‘BD’ w/o the ‘A’

From this slide on, and for the rest of your professional lives, I urge you to please add the ‘Analytics’ suffix to the buzzword ‘Big Data’.


Why this distinction matters?


Big Data Quality Attributes to

watch out for:

Analytics Quality attributes to

watch out for:

-  Performance -  Fault-tolerance -  Replication -  High Availability -  Integration with

current ecosystem

-  Read Performance -  Insert Performance -  Integration with

Analytical Tools -  In-DB Analytics

Why this distinction matters?


We might end up buying/building the wrong technology.

The purpose of BDA

1. Development of new products

2. Gain operational efficiencies

3. Support decision-making


If our BDA initiative doesn’t touch these goals, we’re doing it wrong!

CEO/COO

CFO CTO CDO

The right place for BDA within the firm…


In a startup:

BDA

BDA BDA BDA

Analytics is part of the org’s DNA

The right place for BDA within the firm…


In an mature org:

CEO

CTO CFO COO

CDO BDA

CEO sponsorhip needed to break cultural resistance! BD

The WORST place for BDA within the firm…


CEO

CTO/CIO COO CFO BDA

Why?

Reasons why BDA should not be born in IT (unless core biz is tech)

1.  Asking the wrong questions

2.  Lacking the right skills

3.  Culture change happens elsewhere


Asking the right questions

Even though IT enables the value chain through technology, burning operational

questions may be out of our reach, grasp, or jurisdiction.


Lack of the right skills Forget Drew Conway’s Venn Diagram. The problem is deeper: 1.  IT is a labor of engineering. 2.  The fundamental question of engineering is

‘How’. 3.  To answer questions we need statistics. 4.  The fundamental question of Stats is

‘Why’. 5.  When we answer ‘Why’ we gain insight.


Lack of the right skills (2)

•  Of course, our engineers could go through training to become statisticians, and when they do, they are sometimes better at it than classically-trained statisticians.

•  Only this training is long, and often requires

a change of mindset to become true Data Scientists.


Culture change happens elsewhere

If tech is not the core business nor is central to strategy, IT will not have enough ‘gravitas’ to

pull the entire org from a hunch-based decision management, to a data-driven one.


A case for for giving birth to Analytics in IT


Survey of +200 data professionals. Those closer to SW dev had a negative correlation to those closer to the business. When the pale red dot turns into a tight, upward-facing, dark blue oval, not only will be have software built with a purpose, but also SW devs turned excellent data analysts.

Source: Entry survey for @TheDataPub meetup

If you have no choice but give birth to BDA in IT…

1.  Set up a DWH (if not present). 2.  Federate data. 3.  Establish data ingestion frequency (must match my

decision-making frequency) & pipeline. 4.  Hire the right people with the right skill (and keep

the BI people at bay lest they spread an illness called Reportitis

Operativitis). 5.  Seize IT’s presence all across the value chain and

acquire political capital. 6.  Address the low-hanging fruit of analytics.


1.  Set up a DWH (if not present). 2.  Federate data. 3.  Establish data ingestion frequency (must match my

decision-making frequency) & pipeline. 4.  Hire the right people with the right skill (and keep

the BI people at bay lest they spread an illness called Reportitis

Operativitis). 5.  Seize IT’s presence all across the value chain and

acquire political capital. 6.  Address the low-hanging fruit of analytics.

If you have no choice but give birth to BDA in IT…


Big Data

Analytics

Where do I get the right people (in Mexico) ?

1.  MSc Data Science – ITAM. 2.  MSc Analytic Intelligence – U. Anahuac. 3.  BS Applied Maths + MSc Economics/

Econometrics. 4.  BS Industrial Engineering + MSc Computer

Science. 5.  BS Actuarial Sciences + MSc Computer

Science


Where do I get the right people (in Mexico) ?

•  Note that they’re all master degrees, so don’t expect to pay average developer salaries.

•  Industrial Engineering and Economics appear a lot because those guys know how to measure processes.

•  Note that when we mention Computing, it’s Computer Science, not Engineering.


Take aways: •  BigData does nothing without Analytics. •  BDA must deliver 1) new products, 2)

operational efficiency, 3) decision support. •  The right place for BDA is a position of influence. •  BDA living in IT has many drawbacks related to

skill + political capital. •  But IT is in a priviledged position to deliver value

through BDA if it blends with the business.


Pending discussions:

•  Big Data Ethics •  Beware Data Charlatanry! •  Analytics team-building •  Data Science + Software Engineering •  What mexican education system

needs to produce data professionals.



Thanks! tw: @xuxoramos

linkedin: xuxoramos [email protected]

Big Data, Big Disappointment

Data & Analytics

Transcript of Big Data, Big Disappointment