Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage

Subscribing to Your Critical Data Supply ChainGetting Value from True Data Lineage

Stewart BondResearch Director, Data Integration Software

1

Data is core to digital transformation: intelligence is critical to data integrityThe data environment and supply chain today is more complex than it has ever been. Data needs to be available when and

where it is needed.

Data needs to be secured at all levels of access and consumption.

Data needs to be compliant with increasing regulatory environments.

Data needs to be trusted.

Data intelligence gives organizations the ability to answer the 5 W’s of data: Who, What, Where, When, Why and How.

© IDC Visit us at IDC.com and follow us on Twitter: @IDC 2

A Lack of Enterprise Data Integrity is Already Impacting Digital Initiatives


3.07

3.08

3.19

3.22

3.31

3.39

3.41

2.90 3.00 3.10 3.20 3.30 3.40 3.50

Lack of an Enterprise Data Architecture

Poorly defined requirements

Human resource constraints

Technology (network bandwidth, storage, processing capacity)constraints

Data constraints (Availability, Quality, Metadata, Lineage)

Budget constraints

Security and/or Compliance Policy constraints

(Mean Response)

(Scale 1=No Impact, 5=High Impact)

N=650Source: IDC, 2015

Q. To what extent did the following project issues impact your ability to deliver solutions?

The 3rd Platform is Re-distributing Data into Cloud and On-premises Silos


The number of hybrid and cloud-only data environments is now greater than the number of environments on-premises.Data Deployment EnvironmentsQ. Thinking of your IT environment, select the platforms that data are being sent from or to in your data integration solutions.

Poll Question:

Where is the data your organization is integrating?

(Pick one)

On-Premise Only

Hybrid

Cloud Only

Cloudy Data is Harder to Trust


N=650Source: IDC, 2015

As data moves into the cloud, the number of organizations with positive measurements decreases.

The number of organizations with negative measurements increases.

Q. Thinking of your IT environment, select the platforms that data are being sent from or to in your data integration solutions.

Q. Has your organization experienced a positive, negative, or neutral change as a result of tracking each metric (duplicates, conflicts, stale, incomplete)?

Where and How Lineage


Source One

Vendor

Vendor_ID

Name

Street_Address

City_Address

State_Address

Country_Address

Tax_ID

Invoices

Invoice_Number

Vendor_ID

Invoice_Total

Number_Lines

Target

Source Two

Periods

2015Q1

2015Q2

Spend Report

Vendor Name

Vendor Tax_ID

Sum(Invoice_Total)

Period

Where

How

Data Transformation

Process

Schema

InstanceVendor Tax ID Spend Period

Acme 8524X921 $100000 2015Q1

Acme 8524X921 $150000 2015Q2

SumGroup

By

Data intelligence requires more than Where and How lineage


Internal to external relationships How do my customers relate to each other? How are my products and services related to where my customer is and what my customer is doing? How do my customers relate to, and engage with me?

External relationships Who is my customer? Where is my customer? How does my customer relate to people, places

and things?

Internal relationships Where is my customer’s data inside my organization? What relationships exist between master data about my

customer? Are transactions for my customer related to each other?

To other customers?

5 W’s aren’t enough; Relationship now needs to be part of data intelligence

8

Data lineage delivers a range of impact and value across use cases Data Governance

Compliance

Change Management

Solution Development

Storage Optimization

Data Quality

Problem Resolution

Business Impact


Measuring the impacts of lineage on data trust


n = 352Base = respondents filtered to those that have implemented data integration softwareSource: IDC's Data Integration End User Survey, 2015

Q. Has your organization experienced a positive, negative, or neutral change in each metric?

Measuring the impacts of data lineage on availability and security


Q. Has your organization experienced a positive, negative, or neutral change in each metric?

n = 352Base = respondents filtered to those that have implemented data integration softwareSource: IDC's Data Integration End User Survey, 2015

Poll Question:

Is your organization tracing lineage and is if yes, is it automated?

(Pick one)

Yes/Automated

Yes/Manual

No

Data lineage as an input to solution design: a case study


• Metadata management software implemented and provided to development teams.

• Information catalog software with business glossary.

Solution

• More than 80% of information about data elements available at beginning of sprints.

• Removed at least one, 2-week sprint per iteration.

• Higher quality deliverables.

Outcome

• Multiple distributed agile development teams didn’t have a consistent view of data and data lineage.

• Agile sprint length impacted by data lineage research and impact analysis.

Problem

Data lineage in data stewardship and protection


• Metadata management software inclusive of lineage and a business glossary.

• A data lineage and element glossary dashboard was developed and rolled out to business users.

Solution

• Time data stewards spend on data forensics has become negligible.

• Backward and forward lineage, discovered compliance and security gaps, then closed for policy adherence.

Outcome

• Data stewards spent 30-50% of their time in data forensics in responding to business users’ requests.

• Stale system architecture documents and information.

Problem

Data lineage in compliance reporting and app modernization: a case study


• Metadata management software inclusive of lineage and glossary.

• Automated zero-gap data lineage discovery software able to interrogate application code and data schema.

Solution

• Regulatory audit cycle time and compliance outcomes have improved.

• Improved change management and reduced operational risk in modernization solutions.

Outcome

• TARP and Basel II regulation audit requirements drove a need for data lineage.

• Application modernization projects also required data forensics and impact analysis.

Problem

Zero-Gap Lineage

Data Intelligence

Data Governance

Instance Lineage


Copyright © 2016 ASG.All rights reserved.

Sue HabasVice President of Product Management

Tuesday, June 28, 2016 – Dataversity Webinar

SUBSCRIBING TO YOUR CRITICAL DATA SUPPLY CHAIN – GETTING VALUE FROM TRUE DATA LINEAGE

App-File-Field Transform Rule DB-Tab-Col Calculation

RuleUniverse-Rep-

Field

The business asset foundationCRITICAL LINEAGE SUPPLY CHAIN

Data Supply Chain

Business Traceability

E2E Data Driven Lineage

Verti

cal B

usin

ess

Con

text

Driv

en

Customer/Patient/Event

Business Terms

Policies

Critical Data Elements

About 2 million hospital stayswere for patients without insurance.

Patients covered by Medicare experienced the longest averagelength of stay (5.2 days), and privately insured patients had the shortest average length of stay(3.8 days).

FAST ACCURATE ALIGNED

ReferenceData

DataInventory

Bus GlossaryPackage

DataGovernance

DataLineage

The Complete Workflow

VALIDATE DATA INSIGHTS (HEALTHCARE)

How are claims processing impacted by a

change to LOS?

Rep DB

BDL

MDM Hub

Patient

Claims

EDW

Data Insights

Patient Claim Reports

Where LineageHow Lineage

CHANGE DETECTION AND SUBSCRIPTIONLevel 1: Application Supply Chain

Are the upstream Application owners aware of the EDW change?

The Critical Supply Chain for Average Length of Stay (ALOS)

SQL Override: In April of 2016 the LOS Supply Chain snapshot showed Private Claims set to “Y” use “procedure start/end” versus Hospital “check in/out date time” for “Region 14 (UK) “ for “out patient stays”.

ClaimsIn_pat_stayOut_pat_stayClaim_numPrivate Y/N

PatientGender_cd:Region:State:Age:

DischargeDoc_releaseHosp_check_outProcedure_compClaim_num

PatientReportsIn_pat_stayOut_pat_stayClaim_num

BDLPatient InsightsMedicare vs Insured

LOSClaims

LOS Claims

May

April

MDM Hub

CHANGE DETECTION AND SUBSCRIPTIONLevel 2: Detail

AdmissionPatient_arv_dtPatient_arv_tmCheck_inStart_procClaim_numHos_Check_in

INDUSTRY USE CASES FOR SUPPLY CHAIN CHANGE DETECTION

FINANCE

• BCBS –CCAR Compliance

• FR Y-14 Attestation from CFO

RETAIL

• Salesforce • Facebook• Big Data• Information

Security

PHARMA

• Clinical Outcomes

• Treatment Patterns

• Safety

Poll question:

Do you have resources managing BI lineage supply chains in your organizations today?

(Pick one)YesNo

If yes, what industry are you in: Finance, Retail, Healthcare, Pharma, Manufacture, Entertainment

HEALTHCARE

• Regional Policies (ALOS)

• Claims Processing

• Analytics

MANUFACTURING

• Product Demographics

• Solution Development and Design

ENTERTAINMENT

• Client Demographics

• PII

IMPLEMENTING DATA LINEAGE

We build out a reverse tracing methodology and base line for comprehensive and accurate end to end data lineage.

Lineage

Change DetectionBusiness TraceabilityBusiness Glossary

ID Critical Data Scan Lineage Automate

EASILY DEPLOY DATA FACTS TOANYONE, ANYWHERE

ZERO GAP DATA LINEAGE

LINEAGE ANYWHERE

DATA GOVERNANCE

THE LOB LINEAGE

WORKFLOW

+

LINEAGE WITH THE DATA INTELLIGENCE SOLUTION

COBOL IMS/DBD

COBOL BMS MapCopybook

Business Objects Universe

Informatica ETL

EDW to BI

Dynamic views of cross platform tracing

JAVA

Oracle EDW{PLSQL}

ZERO GAP LINEAGE - FEATURES

• Level

• Trace

• Unpack

SNAPSHOTS STITCH LINEAGE LIFECYCLE

COMPLIANCE REPORTS

ISSUE MANAGEMENT

220+ Scanners and Parsers, Tool Agnostic

MAINFRAME

LINEAGE APPLIANCEIncludes code based ETL

MF/Dist/Hadoop

• Scan • Analyze & Parse• Link• Store (version history/snapshots)• Port to External System

LineageAnywhere

Portable E2E Lineage Package

THANK YOU!

Stewart BondResearch Director, Data Integration SoftwareIDC

@StewartLBond

ca.linkedin.com/in/stewartlbond

Sue HabasVice President, Product ManagementASG

[email protected]

Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage

Business

Transcript of Subscribing to Your Critical Data Supply Chain - Getting Value from True Data Lineage