EUROPEAN DATA FORUMFrom Near to Maturity – Making Big Data relevant to Business

© 2013 Castlebridge Associates

TrainingConsultingCoaching/

MentoringProject

Management

Quality Assured

InformationQuality

DataProtection

DataGovernance

InformationQuality

DataProtection

DataGovernance

Certified Trainers

External QA

Audits

Irish State Approved Training Provider

Quality Assured Syllabu

s

Qualified & Experienced

IQCP & DP

Certified

Certified PMs

Many Industries

Govt Edu UtilitiesFin. Svcs

Non-Profit

Telco

About Castlebridge Associates (www.castlebridge.ie)

What are your expectations?

HISTORYOr: How we came to have all this data anyway…

Ancient Sumeria

• Written in Accadian• Used pictographic representations of information and concepts baked/carved

into tablets made of clay (high sand content)

Portable Information Management

Filing: The Birth of Big Data

Image by Nic McPhee @ commons.wikimedia.com

6 thousand years

Tablets Tablets

Physical Data (5925 years approx.)

Electronic Data(c.75 years)

• More Information processed• Information processed faster• More ‘self service’ data

processing • Changed expectations of data and processing.

But the BIG QUESTION is:

SO WHAT??

Particularly as we may be too late!• Barry Devlin,

• “Big Data is Dead. It’s all just Data!!” • (B-EyeNetwork, December 2012)

• Samuel Arbesman (Wired.com)• “Stop Hyping Big Data and Start Paying

Attention to ‘Long Data’”• (Wired.com – January 2013)

• Ted Friedman (Gartner) on Twitter:

Image © Barry Devlin/B-EYENetwork

Is Big Data just a matter of perspective?

MATURITY

General Overview

Weak

Strong

Maturity models are (almost always) 5 step models that associate common characteristics of organisations that are doing things well or are on the way to doing them better.

Initial

Repeatable

Defined

Managed

Optimising

Where is Big Data?

Uncertainty

Awakening

Enlightenment

Wisdom

Certainty

(Overlaying Crosby CMM model with DMBOK Maturity model)

Initial

Repeatable

Defined

Managed

Optimising

Where is Big Data?

Uncertainty

Awakening

Enlightenment

Wisdom

Certainty

Maturity: Answering So What Questions

So What…

…is it?

…problems will it solve?

…will we be able to differently?

… legal / regulatory risks does all this pose?

… do we need to do to tap this gold mine?

… are we not doing today that this will enable?

… are we not doing today that this make worse?

THE CHALLENGES

Organisations don’t manage data well

Information Governance / Data Governance only now emerging as formal disciplines

Information Quality / Data Quality also only beginning to be coherently tackled in many organisations

Phone companies still get bills wrong

Data Protection breaches still occur• Note – this is more than just SECURITY

breaches

Data Migrations, CRM, ERP still fail

Metadata largely under-managed

Estimated % of TURNOVER wasted by companies due to poor information quality

Time lost to organisations from staff rechecking information

Bottom Line Impact% of Risk Managers who see Information as “Significant” in their Risk Management plans

88%

% of Chief Financial Officers who see Information Management as a barrier to achieving Business goals 75%

30%

35%

Deloitte

Forrester

Gartner

IBM

% Data Migrations that FAIL (don’t deliver, over run time/budget, deliver reduced functionality)

84%Bloor

This is when dealing with “traditional” structured/semi-structured data..

“So far, for 50 years, the information revolution has centered on data—their collection, storage, transmission, analysis, and presentation. It has centered on the "T" in IT.

The next information revolution asks, what is the MEANING of information, and what is its PURPOSE?”

Peter Drucker, Forbes ASAP, August 1998

After the Hype Comes the Hangover

Data Is the New Oil

Picture from NASA

Oil Slick

Water

Pic: US Coast Guard

A REAL EXAMPLE

Names have been changed to protect the innocent

(and the guilty)

The Pending Order Crisis of 2006

If order not completed, cannot be billed

The Pending Order Crisis of 2006

OMG There’s MILLIONS of unbilled revenue out

there.This is a CRISIS!!!

The Pending Order Crisis of 2006

The Sky is FALLING

The Pending Orders Solution 2006

Elite Specialist Information Quality Agent

Licensed to “Fix the Data by all means necessary”

(firearms not actually used…)

The Pending Orders Solution 2006

Orders for infrastructure had engineering statuses

Orders for could have multiple dependent

products – double counted

Revenue Assurance did not look at all relevant data

sources

Dependencies between process steps not

understood

The Pending Order Solution 2006

There wasn’t a Crisis situation • External Factors affected order completion times

• Intra-order product dependencies lead to double counting

• Context of the process was important

Revenue Assurance Hypothesis was flawed

ASKING THE RIGHT QUESTIONS

One way of thinking about data

Question 1: So What Data Do We Need?

No doubt that more data helps, but don’t for a minute think that you need all data to make an informed business decision.

Phil Simon To Big To Ignore: The Business Case for Big Data

Organizations that are effectively leveraging the power of Big Data realize that they will never capture all relevant information.

Question 1: So What Data Do We Need?

Chicken Little © 2005 Disney Corporation

Question 1: So What Data Do We Need?

What is the problem we are trying to solve?

What is the “Information Environment” for this problem?

What is the Process Context for this problem?

The Pending Orders Crisis

What is the problem we are trying to solve?

• Customers are not being billed for services they have• Revenue from services is not being realised• We have orders that are not being completed

What is the Process Context for this problem?

What is the “Information Environment” for this problem?

Question 1: So What Data Do We Need?

To properly answer this question you need to have:

A PLAN

Question 2: So What is Stopping us doing it?

Regulation: • Data Protection Rules• Industry Regulations re: Data Governance

Technology: • Legacy architecture• Technology Management (Silos)

Human Factors: • Skills (technical/problem solving/analytical• Political (Change Management)

Question 2: So What is Stopping us doing it?

Data: • Quality of internal data• Completeness, consistency, “transactability”

• Ability to link external data to internal data• Governance of data

• Decision rights• Supplier relationship management• Roles & Responsibilities

Example of Regulation

Use of Location Data in Telecommunications is affected by EU Data Protection rules

Consent is required for it to be used for “Value Adding” services

Location Data

Data Quality

I am incredibly sceptical about claims that “Big Data” is immune to Data Quality problems.

Statistically, Data Quality errors will skew your mean, and create outliers that affect your analysis.

While “Big Data” might not be as prone to ‘fat finger’ errors, you still have to consider whether the mechanisms gathering the data are correctly calibrated and the algorithms for analysis are running correctly or whether you have measurement errors you don’t know about.

Dr Thomas C Redman, thought leader in Data Quality

Data Quality & Lineage are Key

Databases are like lakes

System B

SystemA

System C

Bias within the Data?

The greatest number of tweets about Sandy came from Manhattan. This makes sense given the city's high level of smartphone ownership and Twitter use, but it creates the illusion that Manhattan was the hub of the disaster. Very few messages originated from more severely affected locations, such as Breezy Point, Coney Island and Rockaway. As extended power blackouts drained batteries and limited cellular access, even fewer tweets came from the worst hit areas.

Kate Crawford Hidden Biases in Big Data, HBR 1st April 2013

Human Factors

• Bias• Politics• Skills• “Attachment Disorder”• Change & Transition Management