Intelligent Data Catalog for IICS, EDP and Axon · 2020. 8. 7. · Intelligent Data Catalog for...

`

Intelligent Data Catalog for IICS, EDP and Axon

Srinivasa Gopal

Mar 06, 2020

Principal Technologist – Customer Success

2 © Informatica. Proprietary and Confidential.

Housekeeping Tips

➢ Today’s Webinar is scheduled for 1 hour

➢ The session will include a webcast and then your questions will be answered live at the end of the presentation

➢ All dial-in participants will be muted to enable the speakers to present without interruption

➢ Questions can be submitted to “All Panelists" via the Q&A option and we will respond at the end of the presentation

➢ The webinar is being recorded and will be available to view on our INFASupport YouTube channel and Success Portal.

The link will be emailed as well.

➢ Please take time to complete the post-webinar survey and provide your feedback and suggestions for upcoming topics.

Success Portal https://success.informatica.com

Learn. Adopt. Succeed.

© Informatica. Proprietary and Confidential.

FREE Product Learning Paths

and weekly Expert sessions

Bootstrap product trial experience

InformaticaConcierge with

Chatbot integrations

Enriched Onboarding experience

Tailored training and content

recommendations

https://success.informatica.com/


Agenda

• Cloud Data Integration UseCases

• Enterprise Data Preparation Integration UseCases

• Data Governance Integration UseCases

• Demo

• Q&A

Data Catalog for Cloud Journey

© Informatica. Proprietary and Confidential.66

Cloud Data Warehouse

Cloud Storage Cloud Data

Warehouse

Line of Business Analytics

Self-Service Analytics

Line of Business

DataAnalyst

LineageDiscovery

Databases

Data Marts

ERP CRM

Files

Devices

Source Systems

Da

ta I

ng

es

tio

n

2

Da

ta I

nte

gra

tio

n &

Qu

alit

y

3

Da

ta P

rovi

sio

nin

g

4

Data Catalog LineageDiscovery1

Clo

ud

Ma

ss In

ge

stio

n

Inte

gra

tio

n &

Qu

alit

y C

lou

d

Inte

gra

tio

n C

lou

d

Enterprise Data

Catalog 1

Azure SQLDW

Azure Data Lake Store

(ADLS Gen 2)

Amazon Redshift

AWS S3

Am

azo

n Q

uick

S

ight


1. Find data to ingest

• Enterprise Data Catalog – industry’s #1 data catalog to find all your enterprise data, detailed data lineage, certified data, social collaboration

• Automate data provisioning from the data catalog


2. Ingest data to landing zone

• Cloud Mass Ingestion – one tool to quickly do all types of ingestion at scale

• Cloud Data Integration – connect to all types of data, easy to use ETL, integrated to the catalog for a shopping cart experience

• Files• Databases• CDC / Schema drift• Real-time streaming• API


3. Integrate and cleanse data into cloud data warehouse

• Cloud Data Integration (CDI) – simply and easy-to-use ETL for any source, pushdown optimization to leverage target engine

• Cloud Data Integration Elastic (CDIe) – high performance cloud-native large volume processing

• Cloud Data Quality – the industry’s only cloud-native data profiling and cleansing tool


4. Self Service BI with embeddedCloud DI and Cataloging

• Embedded BI tool capabilities (i.e. Tableau, PowerBI):

• Cloud Data Integration –automated data provisioning

• Enterprise Data Catalog – catalog at your fingertips and detailed lineage to show where data came from



Cloud

Firewall

Enterprise Analytics

Line of Business Analytics

On-premises

DatabasesApplication Servers

Documents

Mainframe

On-Premises

FTP Servers

Staging Database

Enterprise Data Warehouse

Enterprise Analytics

SaaS

Data Integration3

Da

ta C

ata

log

&

Da

ta I

nte

gra

tio

n

2

Data Catalog & Data Migration

(DI & DQ)1

Data Catalog & Data Migration (DI

& DQ)1

Enterprise Data Catalog

Informatica Cloud Data Integration

1

En

terp

rise

Da

ta C

ata

log

Po

we

rCen

ter

2

Data Integration3 IICS - Integration Cloud

Streaming

Log files Azure Event Hub

Amazon Kinesis / Firehose

Cloud Data Warehouse Migration


1. Discover and prioritize data that needs to migrate

• Enterprise Data Catalog – industry’s #1 data catalog that provides detailed lineage to determine what data and data pipelines needs to be migrated

• Automate data provisioning from the catalog across the source system landscape

• View relevance and quality of data before migrating


2. Migrate data from on-premDW to Cloud DW


• Key connectors • AWS Redshift• Azure SQL DW• Snowflake• Google Big Query

• Operational Insights


3. Ingest data from new sources

• Cloud Mass Ingestion – one tool to quickly do all types of ingestion at scale



Self-Service Data Provisioning

• Empower analysts to access the datathey need

• Simple, click-through data provisioning to easily deliver your data to desired target

• Broad choice of sources & targets, including AWS Redshift, Azure SQL DW, Google BigQuery, Snowflake and BI tools like Tableau

Data Catalog for Data Preparation


Typical Data Preparation Challenges

• Difficulty finding trusted data

• Limited access to the data

• Frustrated by slow response from IT

• Constrained by disparate tools, manual steps

• No way to collaborate, share, and update curated datasets, reuse knowledge

• Can’t cope with growing demand for data from the business

• No visibility into what the business is doing with the data

• Struggling to deliver value to the business

• Losing the ability to govern and manage data as an asset

Business/Data Analysts IT/Data Engineers


Hours / Minutes

Enterprise Data Preparation (EDP) Flips the 80/20 Rule

Data Analysts

I need data for my analysis…

Here’s what I use. Not sure if it’s right…

Colleague

IT

Manager

No idea…

That’s your jobto figure it out.

Ask these people…What data should I use?

Does this lookright to you?

? Hand-CodingTrial & ErrorLots of FilesGuesswork

?

Manual – Errors – Time Consuming – Incomplete

STUGGLE TO FIND DATA STUGGLE TO USE DATA

Months / Weeks

Data Analysts

EDCFind the

Right Data

EDPPrep the

Data Easily

VS

80% of time spent on data prep

20% on analytics

20% of time spent on data prep

80% on analyticsEND-TO-END SELF-SERVICE

FASTER TIME TO VALUE



Cloud Data Lakes(Refined zone)

Enterprise Data Preparation Reference Architecture

Files and Databases

DatabasesFiles

Streaming

IoT Machine Data

Logs

Messaging

Kafka Amazon Kinesis

Azure EventHub

Cloud Storage

Cloud Data Lakes(raw zone)

ADLS Gen2 Amazon S3

ENTERPRISE DATA PREPARATION

AI/ML Data Science

Self-Service Analytics

Visualization

Business Intelligence

MA

SS

ING

ES

TIO

N

ENTERPRISE DATA CATALOG

DATA QUALITY & GOVERNANCE

DATA PROTECTION


Enterprise Data Preparation Primary Use cases

Data Preparation for Self-service Analytics on Data Lakes

Find, explore and prepare data quickly in the data lake to support ad-hoc analytics followed by operationalization in a governed environment

Data Preparation for Advanced Analytics (AI/ML Projects)

Find, explore and prepare data that quickly so that it is ready for Data Science projects followed by operationalization of Data Science process

22 © Informatica. Proprietary and Confidential.22

Data EngineerHow can I quickly prepare data for advanced analytics and operationalize data pipelines for reliable data delivery?

Data ScientistHow can I extract completely new patterns and models that detect or predict behavior?

Data AnalystHow can I quickly and

easily find, explore and prepare data for my

analysis and create a repeatable process without

depending on IT?

Data StewardHow can I enable self-

service and ensure right data is available to right

people in trusted and secure manner?

Enterprise Data Preparation Empowers DataOps TeamsDemocratize data pipelines for analytics and AI/ML workloads at scale

Data Preparation ProcessAgile, Iterative, Collaborative

23 © Informatica. Proprietary and Confidential.23

Data Preparation on Modern Data Lakes with EDP 10.4Enable citizen integrators to be self-served on ADLS/S3 data lakes

Informatica Enterprise Data Preparation

Compute

Data formats

Self-Serve Data Prep

AZURE AWS

Data Lake Storage

Files Datawarehouse Database

On-Premises

Data Catalog for Data Governance Journey

Third Party

Applications

LDA

P A

uth

en

tica

tion

(Ac

tive

Dire

cto

ry)

Applicationslike SAP

Business Intelligence

Data IntegrationTools

Data Warehouses

Hadoop Clusters

Databases

Source

SystemsData Quality

Repository

Data Quality

Rules

Data Profiling

Results

Oracle/DB2/SQL Server

EDC/DPM Metadata

Repository

Cloudera/Hortonworks/HD

Insight/IBM BI

YARN APPS

HBase

Data Quality

Enterprise Data Catalog

• Taxonomies

and Hierarchy

• Glossaries

• Reference

Data

• Workflow

• Physical

evidence

(systems,

data)

Axon

Postgre

SQL

Informatica Platform

*Not all services are shown here

Users

Web

Browser

Informatica Developer

Client

Data Governance Architecture

Data Privacy Management

• Data Quality Rules Creation

• Data Profiling• Other DQ application

development

• Risk rank data stores• Subject Registry• Data protection

workflows• User Activities• UBA

• Technical Metadata Extraction

• Technical Metadata Lineage

• Google for Data

Data Profiling

Metadata Extraction

Masking

• Build global rules

• Execute data protection

Data

domains

Data

domain

s

Scan

Results

Data Protection

Archive/Retire

• Legal Holds• Retention

Policies• Lifecycle Mgt

Retire Data

*Not all sources are

shown


Solution Approach for Actionable Understanding

Glossary Relate Control

Collaborate


Automate Data Discovery, Cataloging, and Context

Leverage ML and AI to find critical data across structured and unstructured sources

Onboard discovered data automatically with oversight and control

Automatically tag data with business context to help users assess relevance


AXON - EDC - Business & Technical LineageConceptual

Physical

Informatica

Axon

Enterprise

Data

Catalog

Do

cu

me

nt/E

nfo

rce

Va

lida

te/R

ecom

mendBusiness

SME

Enterprise

Architect

Maps

Lineage

Business Glossary

Policy ProcessCDE

Classification

CDE Mapping

MetadataTechnical Lineage

Domain

Curation

CDE Tagging

BG Linkage

Link Only CDE Attributes to EDC Columns


Automating Data Quality for Data Governance


Axon integration with EDCConfiguration

Axon

RE

ST

AP

I

EDC

RE

ST

AP

I

AxonScanner

ExternalReference

EDC Bundle(Search)

Axon Glossaries

Resources, Columns etcREST API Request

REST API Response

Questions?

?

Thank You!


References

• IICS – EDC Integration Properties : https://docs.informatica.com/integration-cloud/cloud-platform/current-version/administrator/organizations/organization-properties/enterprise-data-catalog-integration-properties.html

• EDC Managing Cloud Org : https://docs.informatica.com/data-catalog/enterprise-data-catalog/10-4-0/catalog-administrator-guide/managing-cloud-organization.html

• EDC Data Preview & Provisioning : https://docs.informatica.com/data-catalog/enterprise-data-catalog/10-4-0/enterprise-data-catalog-user-guide/data-preview-and-provisioning.html

• Using Axon with Enterprise Data Catalog https://kb.informatica.com/h2l/HowTo%20Library/1/1150-UsingAxonDataGovernancewithEnterpriseDataCatalog-H2L.pdf

• Axon Checklist to Automate Data Quality Rules https://docs.informatica.com/data-quality-and-governance/axon-data-governance/h2l/checklist-to-automate-data-quality-rules-in-axon-data-governance/abstract.html

• 7 Best Practices to Drive Data Catalog Adoption https://www.informatica.com/lp/7-best-practices-to-drive-data-catalog-adoption_3712.html#fbid=PYYnGF22B9T

• Managing the Data Lake using EDP : https://docs.informatica.com/data-catalog/enterprise-data-preparation/10-4-0/enterprise-data-preparation-administrator-guide/managing-the-data-lake/data-lake-management-overview.html

• New Features in EDP v10.4 : https://docs.informatica.com/data-catalog/enterprise-data-preparation/10-4-0/new-features-guide/new-features--10-4-0-/enterprise-data-preparation.html

https://docs.informatica.com/integration-cloud/cloud-platform/current-version/administrator/organizations/organization-properties/enterprise-data-catalog-integration-properties.html

https://docs.informatica.com/data-catalog/enterprise-data-catalog/10-4-0/catalog-administrator-guide/managing-cloud-organization.html

https://docs.informatica.com/data-catalog/enterprise-data-catalog/10-4-0/enterprise-data-catalog-user-guide/data-preview-and-provisioning.html

https://kb.informatica.com/h2l/HowTo%20Library/1/1150-UsingAxonDataGovernancewithEnterpriseDataCatalog-H2L.pdf

https://docs.informatica.com/data-quality-and-governance/axon-data-governance/h2l/checklist-to-automate-data-quality-rules-in-axon-data-governance/abstract.html

https://www.informatica.com/lp/7-best-practices-to-drive-data-catalog-adoption_3712.html#fbid=PYYnGF22B9T

https://docs.informatica.com/data-catalog/enterprise-data-preparation/10-4-0/enterprise-data-preparation-administrator-guide/managing-the-data-lake/data-lake-management-overview.html

https://docs.informatica.com/data-catalog/enterprise-data-preparation/10-4-0/new-features-guide/new-features--10-4-0-/enterprise-data-preparation.html

Intelligent Data Catalog for IICS, EDP and Axon · 2020. 8. 7. · Intelligent Data Catalog for...

Documents

Transcript of Intelligent Data Catalog for IICS, EDP and Axon · 2020. 8. 7. · Intelligent Data Catalog for...