ClearDQ Intro Pack v0.5

Data Quality for Financial Performance

Got Bad Data ? How much is bad data really costing your business ?

Why do DQ teams perpetually struggle to solve this problem ?

ClearDQ

© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902

Data Issues Aren’t going away.The commercial costs are real, and significant.Global technology industry paradigm problem.20

% The damage bill starts est. 20% of FTE.

ClearDQ

Poor Data Quality The Unmanaged business overhead

Costs organizations in excess of 11-13% FTE (some as high as 24% !)

Exacerbated by the Business vs IT “data disconnect” fuelled by IT vendors selling DQ technology tools…

Most Data Warehouses and Business Intelligence efforts fail to address Data Quality and in fact, in many proven cases make Data Quality worse..

Poor DQ wreaks havoc in Regulatory Reporting and Regulatory Markets ( Banking, Energy, Health, etc… ), fines, customer churn, etc…

Vicious Cycle - Poor DQ leads to data duplication, which in turn exacerbates data quality problems.

11

IT

DW

$$

2x


ClearDQ

Telco Industry Systemic DQ Issues

Regulator TIO refers major systems issues to to regulators - the Australian Communications and Media Authority and the Australian Competition and Consumer Commission (*) Privacy and protection of customers' personal information (Data Classification,

Data Privacy, Data Access, Data Security) Telemarketing practices Misleading sales information Incorrect billing (Billing Data, Data Movement, Data Aggregation, Manual

Adjustments) unfair contract terms, and compliance with consumer guarantees under Australian Consumer Law.Other data quality problematic areas leading to business function problems are: Poor Access to Data, Poor Reporting Solutions, Multiple sources of duplicate data

(merge, filter damage), Poor Data Meaning (Ambiguity), Duplicated Data Costs (Duplicated platforms, teams, processes), Manual Data Efforts, Large Datasets (difficult to manage), Data Retention and Recovery (Data loss), etc..

(*) source: Telecommunication Industry Ombudsman (TIO) - Annual Report 2015)

ClearDQ

ClearDQ Data Quality for Business

Data Quality issues expressed in commercial impact terms - $ and time

A system that places Data Quality under the control of business leaders

Helps IT by instructing IT when, why and what to do, as dictated by the priorities set by the business DQ sponsor.

Hosted app, gets your business team solving their most urgent DQ issues immediately

Strong business relevance for Energy, Banking, Finance, Health, Airline, FMCG and other industry verticals


ClearDQ

Foundational Data Capabilities

Technology(Design, Build,Operate)

Data EnabledBusinessFunctions & Capabilities(example)

Business Strategy(example)

ProductInnovation

Better Supplier

Management

Simpler Customer

Offer

Ops & Supply Chain

Transformation

Transform Sales

OnlineInnovation

Your Data Mission: Capture, store, secure, move, supply, share and analyse Your Business Data with ease, at speeds, and in consumable formats, that meet our operational and

strategic needs to support fact-based decision making and optimal business execution, within an understood data cost/risk profile.

SocialPlatfor

mDigital Shops Finance

HR

Assets Merchandise Sales Supply

Chain

Services Capabilities

Processes Delivery People

Agile Data Delivery,

Innovation & Data Value Realisation

Corporate Memory - Reusable Foundations, Simplified Technology, Training Uplift, Speed Delivery, Maturity Uplift, Funding Optimisation, Data Innovation

Accountability

Meaning

Single Source

Assurance

Data Quality

Data Moveme

ntAccess

Platforms & Tools

Data Value Maturity & Innovation

Ownership,Governance, Stewardship,

Produce Data,

Consume Data

Business vs IT

Glossary,

MetadataData ModelsRules

Master,Referen

ce & Txn

Data, PK’s FK’s

ProfilingMonitorin

gCleansingEnrichme

ntTriage

Migration, Integratio

n, Extraction

, ETL, ESB, DV, Process Mapping

Apps, Reporting, Analytics, DW / BI

Big Data, Cloud,

Platforms, DB

Engines, Formats

Reporting

Business Case,

Funding, Frameworks, Training,Innovation

MarketingLoyalty

ProjectDelivery

Solution Arch

Artefacts,Proactive Solution

Governance

Funding

Network PlanningProperty Workforc

eCorporat

eSystems

FinancialProducts

Security

IS&O

Customer Value Investment

Effective Teams

Improve Product Margins

Better Partner Network

Loyalty Expansion

StoreUplift

Risk, Controls,

Maint.

Classify,Security, Regulator

y,Archiving, Retention, Recovery

Search and Docs

Documents

ContractsPDF,

WORDEXCELSearch,

Taxonomy


Any function can be seriously impacted by bad day. However, Finance, Marketing and Sales functions are (typically) more negatively impacted by bad

data that other functions.

Globally, Data Capability Matrurity is generally LOW or NON-EXISTENT, or UNFUNDED.

ClearDQ

Data Economics & MonetizationLinking Data Capabilities to Data Risks, Costs and Data Value Unlock


Accountability

Meaning

Single Source

Assurance

Data Quality

Data Moveme

ntAccess

Platforms & Tools



Produce Data,

Consume Data

Business vs IT

Glossary,


Master,Referen

ce & Txn

Data, PK’s FK’s

ProfilingMonitorin

gCleansingEnrichme

ntTriage


n, Extraction



Big Data, Cloud,

Platforms, DB

Engines, Formats

Reporting

Business Case,


ProjectDelivery

Solution Arch


Governance

Risk, Controls,

Maint.



Search and Docs

Search, Taxonomy

Waste (bad data)

Buy Data Sets

Revenue Leakage

Staff Attrition(due t bad data)-$

+$Sell Data Sets

(Customer Analytics) Cross Sell

(Business Analytics) Automate Data Activity

(Marketing Analytics) Campaign Optimisation

Etc.

Etc.

Business is usually unaware of key Data Capabilities for value to be unlocked

Data Damage exists due to poor (or missing) Data Capabilities

Data Quality is a function of the existence and performance of key Data Capabilities

Data Governance is a fruitless activity if it is not linked to the inspection, evaluation and

performance of Data CapabilitiesData Initiatives built on poor foundations, bad data

and poor governance COST MORE to maintain.

Data must be trusted, accessible and fit-for-purpose, to deliver value.

Data is an “Engineered Product” that acts precisely as it is instructed.

The state and rigor of a company’s Data Capabilities is an accurate reflection of it’s

maturity and values.


Start Here, Then Build.

IT DQ Teams struggle and repeatedly fail here (global trend for 15+ years). Not

holistic.

ClearDQ

Data Quality DamageThe base-cost of poor quality data in non-IT (*)

(*) 2012 and 2013, 2015 Research Results – Detailed Customer Assessment Metrics.

GlobalBank

23%FTE

EnergyRetail

13%FTE

HealthRetail

24%FTE

Yours ??

Shadow IT Re Reconcile

Manual Correction

Re Extract De Duplicate Report Preparation

$24m

$3.4m

$60m

$39m


ClearDQ

Annual DQ Damage Estimator (Telco Examples) Running ClearDQ discovers the accurate detail and $ precision in 2-8 weeks depending on org size.

FTE20% FTE

WasteAvg Salary

$40,000 $45,000 $50,000 $55,000 $60,000 $65,000 $70,000 $75,000 $80,000Due to Bad

DataDamage in

$m $m $m $m $m $m $m $m $m200 40 $2 $2 $2 $2 $2 $3 $3 $3 $3 300 60 $2 $3 $3 $3 $4 $4 $4 $5 $5 500 100 $4 $5 $5 $6 $6 $7 $7 $8 $8 800 160 $6 $7 $8 $9 $10 $10 $11 $12 $13

1000 200 $8 $9 $10 $11 $12 $13 $14 $15 $16 2000 400 $16 $18 $20 $22 $24 $26 $28 $30 $32 5000 1000 $40 $45 $50 $55 $60 $65 $70 $75 $80

10000 2000 $80 $90 $100 $110 $120 $130 $140 $150 $160 15000 3000 $120 $135 $150 $165 $180 $195 $210 $225 $240 20000 4000 $160 $180 $200 $220 $240 $260 $280 $300 $320 25000 5000 $200 $225 $250 $275 $300 $325 $350 $375 $400 30000 6000 $240 $270 $300 $330 $360 $390 $420 $450 $480 35000 7000 $280 $315 $350 $385 $420 $455 $490 $525 $560 40000 8000 $320 $360 $400 $440 $480 $520 $560 $600 $640 50000 10000 $400 $450 $500 $550 $600 $650 $700 $750 $800 55000 11000 $440 $495 $550 $605 $660 $715 $770 $825 $880 60000 12000 $480 $540 $600 $660 $720 $780 $840 $900 $960 80000 16000 $640 $720 $800 $880 $960 $1,040 $1,120 $1,200 $1,280

100000 20000 $800 $900 $1,000 $1,100 $1,200 $1,300 $1,400 $1,500 $1,600 200000 40000 $1,600 $1,800 $2,000 $2,200 $2,400 $2,600 $2,800 $3,000 $3,200 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800


Telstra

Optus

AT&T

Verizon

ClearDQ

Annual DQ Damage Estimator (Banking Examples) Running ClearDQ actually discovers the accurate detail and $ precision in 2-8 weeks depending on org size.

FTE20% FTE

WasteAvg Salary

$40,000 $45,000 $50,000 $55,000 $60,000 $65,000 $70,000 $75,000 $80,000Due to Bad

DataDamage in

$m $m $m $m $m $m $m $m $m200 40 $2 $2 $2 $2 $2 $3 $3 $3 $3 300 60 $2 $3 $3 $3 $4 $4 $4 $5 $5 500 100 $4 $5 $5 $6 $6 $7 $7 $8 $8 800 160 $6 $7 $8 $9 $10 $10 $11 $12 $13

1000 200 $8 $9 $10 $11 $12 $13 $14 $15 $16 2000 400 $16 $18 $20 $22 $24 $26 $28 $30 $32 5000 1000 $40 $45 $50 $55 $60 $65 $70 $75 $80

10000 2000 $80 $90 $100 $110 $120 $130 $140 $150 $160 15000 3000 $120 $135 $150 $165 $180 $195 $210 $225 $240 20000 4000 $160 $180 $200 $220 $240 $260 $280 $300 $320 25000 5000 $200 $225 $250 $275 $300 $325 $350 $375 $400 30000 6000 $240 $270 $300 $330 $360 $390 $420 $450 $480 35000 7000 $280 $315 $350 $385 $420 $455 $490 $525 $560 40000 8000 $320 $360 $400 $440 $480 $520 $560 $600 $640 50000 10000 $400 $450 $500 $550 $600 $650 $700 $750 $800 55000 11000 $440 $495 $550 $605 $660 $715 $770 $825 $880 60000 12000 $480 $540 $600 $660 $720 $780 $840 $900 $960 80000 16000 $640 $720 $800 $880 $960 $1,040 $1,120 $1,200 $1,280

100000 20000 $800 $900 $1,000 $1,100 $1,200 $1,300 $1,400 $1,500 $1,600 200000 40000 $1,600 $1,800 $2,000 $2,200 $2,400 $2,600 $2,800 $3,000 $3,200 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800


ANZ Bank

NAB

Wells FargoBoA, Barclays

ClearDQ

Annual DQ Damage Estimator Running ClearDQ actually discovers the accurate detail and $ precision in 2-8 weeks depending on org size.

FTE20% FTE

WasteAvg Salary

$40,000 $45,000 $50,000 $55,000 $60,000 $65,000 $70,000 $75,000 $80,000Due to Bad

DataDamage in

$m $m $m $m $m $m $m $m $m200 40 $2 $2 $2 $2 $2 $3 $3 $3 $3 300 60 $2 $3 $3 $3 $4 $4 $4 $5 $5 500 100 $4 $5 $5 $6 $6 $7 $7 $8 $8 800 160 $6 $7 $8 $9 $10 $10 $11 $12 $13

1000 200 $8 $9 $10 $11 $12 $13 $14 $15 $16 2000 400 $16 $18 $20 $22 $24 $26 $28 $30 $32 5000 1000 $40 $45 $50 $55 $60 $65 $70 $75 $80

10000 2000 $80 $90 $100 $110 $120 $130 $140 $150 $160 15000 3000 $120 $135 $150 $165 $180 $195 $210 $225 $240 20000 4000 $160 $180 $200 $220 $240 $260 $280 $300 $320 25000 5000 $200 $225 $250 $275 $300 $325 $350 $375 $400 30000 6000 $240 $270 $300 $330 $360 $390 $420 $450 $480 35000 7000 $280 $315 $350 $385 $420 $455 $490 $525 $560 40000 8000 $320 $360 $400 $440 $480 $520 $560 $600 $640 50000 10000 $400 $450 $500 $550 $600 $650 $700 $750 $800 55000 11000 $440 $495 $550 $605 $660 $715 $770 $825 $880 60000 12000 $480 $540 $600 $660 $720 $780 $840 $900 $960 80000 16000 $640 $720 $800 $880 $960 $1,040 $1,120 $1,200 $1,280

100000 20000 $800 $900 $1,000 $1,100 $1,200 $1,300 $1,400 $1,500 $1,600 200000 40000 $1,600 $1,800 $2,000 $2,200 $2,400 $2,600 $2,800 $3,000 $3,200 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800 300000 60000 $2,400 $2,700 $3,000 $3,300 $3,600 $3,900 $4,200 $4,500 $4,800

(*) In November 2013, the average full-time wage is $74,724 before tax. Source Australian Bureau of Statistics© Copyright 2013. All rights reserved. Clear Strategic IT Partners Pty Ltd ABN 69 162 232 902

2. What is your FTE Average Salary ?

1. How Many FTE

do you have ?

ClearDQ

??

Challenge: Convert the Est. into Precision Numbers (1-8 weeks depending on the workforce size)

(*) In November 2013, the average full-time wage is $74,724 before tax. Source Australian Bureau of Statistics



Accountability

Meaning

Single Source

Assurance

Data Quality

Data Moveme

ntAccess

Platforms & Tools



Produce Data,

Consume Data

Business vs IT

Glossary,


Master,Referen

ce & Txn

Data, PK’s FK’s

ProfilingMonitorin

gCleansingEnrichme

ntTriage


n, Extraction



Big Data, Cloud,

Platforms, DB

Engines, Formats

Reporting

Business Case,


ProjectDelivery

Solution Arch


Governance

Risk, Controls,

Maint.



Search and Docs

Search, Taxonomy

Accurate $$ Business DamageGranular Business SentimentMapped to Entire IT DATA LandscapeFramework, Method and App

?? ??

Globally, no tools exist to autamate this linkage, at scale, in real-time until

now.

ClearDQ

ClearDQ is made up of 4 key parts…

Data Framework

A proven framework which can be applied as comprehensively or as lightly as needed – to manage your Information challenges.

Data Quality Method

A consistent, repeatable method to track, manage, cost and control DQ issues that impact your business.

Business Web App

A web-based tracking, measurement and collaboration tool to productively manage DQ issues with minimal effort.

Benefits Dashboard

Keeps business managers, GM’s team leaders and your data champions up to date with DQ success metrics.

Data Leadership…

Proven Process…

Smart App… Financial Results…

ProvenFramework

RepeatableMethod

CollaborativeApp

Exec & Team Dashboard


ClearDQ

DQ User Sentiment Screen Shot (Web App)Granular Data Capability Failures in the Business Population

ClearDQ

DQ Issues Screen Shot (Web App)Granular Data Capability Metrics

ClearDQ

Screen Shot (Web Dashboard)Annual DQ Damage Bill (Minimum Damage **)

Annua l DQ Damage to

PEOPLE.

(**) Does not include Fraud, Revenue Leakage, Lost Sales, Regulatory Fines, etc…)

ClearDQ

Screen Shot (Architectural Damage)Total $ Damage Breakdown, by WORST THEME,

and DEPT.

ClearDQ

Screen Shot (User DQ Sentiment)1.0

Catastrophic Data breakage

ClearDQ

ClearDQ – Benefits

Tracks impact of poor DQ, measured across the entire organization, by team

All DQ Issues are compared, priced and prioritized the SAME way

DQ issues are commercialized into $ and dates

Put the business in control of Data Quality processes, priority and spend


Precision $ commercial metrics to satisfy the CFO.

Mapped to 4 Different Data Capability Maturity Models, to satisfy any Enterprise Data Architect or IT leader.

At scale to support any size organization or workforce.

A Business “friendly” enterprise-wide approach, that drives the detailed technology data improvement investments.

Linked to APRA (CPG 235) and any other industry regulator demands.

ClearDQ

Case Study: Major Energy Utility

Pain Points

Suffered major data quality issues and disruption for 3 years after a major system migration. IT made many attempts to solve the DQ issues without success.

Benefits

80+ Data Quality issues were identified, to a total value of $24m in immediate cash OPEX savings. The recovery of the $24m would cost $2m in systems fixes and repairs. Staff waste due to data re-work and data repairs including data duplication, re-extraction, re-reconciling, etc… totaled 13% across a workforce of 500 FTE (not IT !) Data re-work workforce to (65FTE) not focused on their primary business function. Regulatory fines detected and prevented in excess of $5m. Customer Care, Customer churn and and customer satisfaction metrics all positively influenced.

80 24 2 13 65

3 IT


ClearDQ

Engagement ProcessDepending on the size of the organization

1-2 hours 2-10 days 10-30 days

Train & Implement

DQ Forensic Discovery

DQ Impact Assessment

Consultation & Demo

2-5 days

Is Data Quality a challenge for your organization ? Can our proven approach help you ?

What are the major commercial impacts, risks and team impacts due to poor data quality ? What DQ improvements have been made ? What hasn’t worked ? What needs improving ?

Where are your data issues captured and managed ? Where are your business data rules ? Who manages, applies and measures (and automates) your business data rules ?Who are your data champions ? Where are the biggest DQ improvements yet to be made ?

Who needs to be trained ? What DQ metrics need to be captured & presented ?


ClearDQ

Get the Business DQ help you need…

Martin SprattEnterprise Data Architect

Author & Trainer

Mob: +61-466-839944

[email protected]

Data Quality

Trusted Data Across

the Enterprise

ISBN: 978-1-906355-21-0


ClearDQ

About - Clear Strategic IT Partners

Outcomes & Solutions

Training &Advisory

DomainArchitecture

Enterprise Architecture

Other Specialized Architected Solutions, Delivery, Training, Advisory (& Rescue):

Clear DQ Data Quality for Business (v2.0, V3.0)ClearCDO Chief Data Office – Data Monetization PlatformClear RDM Reference Data ManagementClear META Active-Active Metadata ManagementClear BACK Enterprise scale archiving & recoveryClear 360 360 degree view of customer, supplier, part, employee, etc…Clear MIG Data Migrations & TransformationsClear CLOUD Cloud Data Transformation & IntegrationClear PM Project Management Training & Certification (Prince2)Clear MOBI Business reports on the goClear ARC IT Architecture, Training & Strategic Services


ClearDQ

Our growing partner networkTrusted, Credible, CapableIT Project & Porfolio Management

MetaPM

Data Centre

UberGlobal

IT Asset Management

The Mastermind Group

IBM Software

Certus Solutions

Data Architecture

CMMI DMM

EDMCouncil.org

Cloud & Dev Ops

Amazon

Broadcoast

Base2Services

Anatas

Red Hat OpenShift

Big Data & Analytics

Actian

Yellowfin

Hadoop Eco-systems

(Cloudera, MapR, Hortonworks)

Data Virtualization

Cisco

Integration

Anatas

Six Tree

Open Source

Six Tree

Apache.org

Bitnami.com

ClearStrategicIT.comClearDQ.com

Clear.


ClearDQ Intro Pack v0.5

Documents

Transcript of ClearDQ Intro Pack v0.5