Information Server on System z - IBM · invoices (VSAM) – 3rd system that contains status of...

IBM Information Server for System z

Toon LatinneTechnical Sales, [email protected]

1

Agenda

� Information Integration– The Challenges– IBM Information Server

� IBM Information Server– Part I:

• Analyze & metadata

• Metadata

• Cleanse

– Part II: • Transform,

• Synchronize / replication

• virtualize

� Cocktail

2

Example of a business challenge

� Supplier of Energy, Information Systems :– SAP system for ordering (Oracle on AIX)– In house development system for billing &

invoices (VSAM)– 3rd system that contains status of delivered

service/installation (DB2 on z/OS)

� Business problem :– Orders without invoice – Electricity delivered but not yet paid – Invoice without a service delivered – Same customer with different names

� Technical problem :– How to compare three different sources,– Common key ?– Connectivity:

• Read SAP-data , • How to read VSAM-file, • Swift information

– Huge volumes

3

Business Challenge 2 :Newly Loaded Data – Incorrect Data Loaded Correctly

4

Information Issues and Business Challenges

� Too much information, not knowing what’s important– Not using demand signals to drive supply chain

– Not using customer analysis to tailor marketing and sales

– Not leveraging valuable unstructured information

� Multiple versions of the truth– Problems managing customer, product and partner interactions

– Regulatory compliance inhibited by poor transparency

� Lack of trusted information– Incomplete, out-of-date, inaccurate, misinterpreted data

– Difficult to understand or control how information is used

� Lack of agility– Inability to take advantage of opportunities for innovation

– Escalating costs due to inflexible systems and changing needs

5

Customer Examples

Where Business and IT Initiatives Meet

IT ProjectStrategicInitiative

Customer ValueCustomer Success

Business Intelligence

Business intelligenceRicoh Europe is able to gain a clear view of

performance across its 14 European sales

operations, based on common KPI’s.

InfrastructureRationalization

Migration to SAP single client

Data migration from 15 different systems to

one single SAP instance.

Risk andCompliance

Global financial reporting

CorporatePortals

Master DataIntegration

Customer master data integration

Support better data quality for EttArla Program

resulting in reduction of both suppliers and

bought in goods of ~40%

BusinessTransformation

Provided the transactional information

integration capability for Nestle’s GLOBE

project.

Provided the transactional information

integration capability for Nestle’s GLOBE

project.

Basel II Data Warehouse for credit checks

and fraud detection.

Synchronize supply management

Channel synchronizationDelivered accurate product & inventory detail

on web channel, reducing burden on call

center employees by automating order entry.

6

IBM Information Server for System z Delivering information you can trust

Understand Cleanse Transform Deliver

Parallel Processing

Rich Connectivity to Applications, Data, and Content

Discover, model, and govern information

structure and content

Standardize, merge,and correct information

Combine and restructure information

for new uses

Synchronize, virtualize and move information

for in-line delivery

Unified SOA Deployment

Unified Metadata Management


7



Parallel Processing






for new uses






Part I:

Define&

Analyze&

Cleanse

Part II:

Transform&

Synchronize&

Virtualize

8

IBM Information Server for System zDelivering information you can trust

Transform Deliver

Parallel Processing



for new uses






Understand

Information Analyzerfor Linux on System z

Business Glossaryfor Linux on System z

Cleanse


Understand your data and the terms that describe it to ensure consistency,

trust and effective communication between the business user and the

technology user

Part I:

Define&

Analyze&

Cleanse

9

Understand your data � Metadata

What is metadata anyway?

Data Metadata

10

Understand your data � Metadata

� Business Metadata– Business rules, Definitions, Terminology, Glossaries, Algorithms

and Lineage using business language. – Audience: Business users.

� Technical Metadata– Defines Source and Target systems, their Table and Fields

structures and attributes, Derivations and Dependencies.– Audience: Specific Tool Users – BI, ETL, Profiling, Modeling.

� Operational Metadata– Information about application runs: their frequency, record counts,

component by component analysis and other statistics. – Audience: Operations, Management and Business Users.

� Project Metadata– Documents and audits development efforts, assigns Stewards, and

handles change management. – Audience: Operations, Stewards, Tool Users and Management.

B

T

O

P

Literally, “data about data” that describes your company’s information from both a business and a technical perspective.

Business Glossary

Information Analyzer

11

– Managing & sharing of business metadata, Technical Metadata

– Aligns the efforts of IT with the goals of the business– Provides business context to information technology

assets– Establishes responsibility and accountability– Terminology, vocabulary, synonyms

Business & Project MetadataB

P

Business Glossary

12

GL Account Number

The ten digit account number. Sometimes referred to as the account ID. This value is of the form L-FIIIIVVVV.

Column = ACCT_NO

Database = DB2

Schema = NAACCT

Table = DLYTRANS

data type = char(11)

Creates a common vocabulary

between business and technical users

IBM Confidential

DBABusiness Analyst/Data Steward

MetadataServer

Business & Project MetadataB

P

Business Glossary

13

Technical & Operational Metadata

Critical Problems:

� You don’t know what data is really in your legacy

systems

� Sources have changed or are new and unknown

Why?

� Data values and relationships are inconsistent and

divergent from documented rules

� Incomplete and missing documentation

� Data sources are never static and frequently change

without warning

Alternative Approach

� Labor intensive, resource devouring process

� Never review 100% of data elements

� No standardized approach across projects

� 1st generation tools document but don’t address the

problem resolution

Mainframe manufacturing system

Demographic

Contact

Billing / Accounts

External Lists

Distribution

ERP from acquisition

Parts BOM

Data SourcesData Sources

T

O


14

What does Information Analyzer ?

1. Source System Analysis - provides the key understanding of the source data

a) Column analysis

b) Primary Key analysis

c) Foreign Key analysis

d) Cross Domain analysis

2. Monitoring and Integration- leverages the analysis to expedite the development

a) Baseline Analysis to track changes

b) Reference tables for Transformation, Completeness and Validity enforcement

c) Integrated Metadata – results visible in rest of Suite

Foreign Key & Cross Domain Analysis

PrimaryKey Analysis

Co

lum

nA

na

lys

is Source 1 Source 2


15

Column Analysis – Example

Status Code is ‘I’, ‘A’ or ‘ACCT_STATUS’Why ?


16

Column Analysis – Example

ZipCode has different formats

Why ?


17


Transform Deliver

Parallel Processing



for new uses






Understand



Cleanse

QualityStage forLinux on System z

Part I:

Define&

Analyze&

Cleanse

Match, merge and cleanse information

from your enterprise data sources to ensure a consistent representation of

all of your critical business data representing customers, products,

accounts, etc.

18

Should I Care about Cleansing Information?

● Lack of information standards

�Different formats and structures across different systems

● Data surprises in individual fields

�Data misplaced in the database

● Information buried in free-form fields

● Data myopia

�Lack of consistent identifiers inhibit a single view

● The redundancy nightmare

�Duplicate records with a lack of standards

Kate A. Roberts 416 Columbus Ave #2, Boston, Mass 02116

Catherine Roberts Four sixteen Columbus APT2, Boston, MA 02116

Mrs. K. Roberts 416 Columbus Suite #2, Suffolk County 02116

Name Tax ID Telephone

J Smith DBA Lime Cons. 228-02-1975 6173380300

Williams & Co. C/O Bill 025-37-1888 415-392-2000

1st Natl Provident 34-2671434 3380321

HP 15 State St. 508-466-1200 Orlando

WING ASSY DRILL 4 HOLE USE 5J868A HEXBOLT 1/4 INCH

WING ASSEMBY, USE 5J868-A HEX BOLT .25” - DRILL FOUR HOLES

USE 4 5J868A BOLTS (HEX .25) - DRILL HOLES FOR EA ON WING ASSEM

RUDER, TAP 6 WHOLES, SECURE W/KL2301 RIVETS (10 CM)

19-84-103 RS232 Cable 6' M-F CandS

CS-89641 6 ft. Cable Male-F, RS232 #87951

C&SUCH6 Male/Female 25 PIN 6 Foot Cable

90328574 IBM 187 N.Pk. Str. Salem NH 01456

90328575 I.B.M. Inc. 187 N.Pk. St. Salem NH 01456

90238495 Int. Bus. Machines 187 No. Park St Salem NH 04156

90233479 International Bus. M. 187 Park Ave Salem NH 04156

90233489 Inter-Nation Consults 15 Main Street Andover MA 02341

90345672 I.B. Manufacturing Park Blvd. Bostno MA 04106

QualityStage™

19

QualityStage, what does it do?

CUSNUM NAME ADDRESS SALES $

8,494.00

3,432.00

2,243.00

5,900.00

6,800.00

10,243.00

15,999.00

903-284-5714

903-284-5715

902-938-1495

902-233-1479

902-334-1189

902-348-3389

903-456-7211

Int’l Business Machines

I.B.M. Inc

I B M

IBM Corp

IBM Consulting

IBN Info Services

IBN Integration

187 N.Pk. St. Salem NH 01456

187 N.Pk. St. Sarem NH 01456

187 N. Park StSalem NH 04156

Park Ave 187, Salem NH 04156

15 Main St. Andover MA 02341

PO Box 9 Boston MA 02210

Park Blvd. Boston MA 04106

No unique key AnomaliesNo standards

Spelling

WebSphereQualityStage

Process

1. Free Form Investigation

2. Data Standardization

3. Data Matching

4. Data Survivorship

QualityStage™

20

� Standardization - Example

Input File:

Address Line 1 Address Line 2

639 N MILLS AVENUE ORLANDO, FLA 32803306 W MAIN STR, CUMMING, GA 301303142 WEST CENTRAL AV TOLEDO OH 43606843 HEARD AVE AUGUSTA-GA-309041139 GREENE ST ACCT #1234 AUGUSTA GEORGIA 309014275 OWENS ROAD SUITE 536 EVANS GA 30809

Result File:

House # Dir Str. Name Type Unit No. NYSIIS City SOUNDEX State Zip ACCT#

639 N MILLS AVE MAL ORLANDO O645 FL 32803 306 W MAIN ST MAN CUMMING C552 GA 301303142 W CENTRAL AVE CANTRAL TOLEDO T430 OH 43606843 HEARD AVE HAD AUGUSTA A223 GA 309041139 GREENE ST GRAN AUGUSTA A223 GA 30901 1234 4275 OWENS RD STE 536 ON EVANS E152 GA 30809

QualityStage™

21

WILLIAM J KAZANGIAN 128 MAIN ST 02111 12/8/62

WILLAIM JOHN KAZANGIAN 128 MAINE AVE 02110 12/8/62

Are these two records a match?

Deterministic Decisions Tables:• Fields are compared• Letter grade assigned• Combined letter grades are compared to a vendor delivered file• Result: Match; Fail; Suspect

B B A A B D B A = BBAABDBA

+5 +2 +20 +3 +4 -1 +7 +9 = +49

Probabilistic Record Linkage:• Fields are evaluated for degree-of-match• Weight assigned: represents the “information content” by value• Weights are summed to derived a total score• Result: Statistical probability of a match

� Two Methods to Decide a Match

QualityStage™

22

Input File Record ID

WNG ASSY DRL 3 HOLE USE HEXBOLT ¼ INCH 1

WING ASSEMBY, HEX BOLT .25” - DRILL FOUR, USE 5J868-A 2

USE 4 5J868A BOLTS (HEX .25) - DRILL HOLES FOR EACH ON WING ASSEM 3

RUDER, TAP 6 WHOLES, SECURE W/KL2301 RIVETS (10 CM) 4

Assembly Instruction QTY Type Part Size Unit Measure SKU

WING DRILL 3 HOLES HEXBOLT .25 INCH

WING DRILL 4 HEXBOLT .25 INCH 5J868A

WING DRILL 4 HOLES HEXBOLT .25 5J868A

RUDDER TAP 6 HOLES RIVET 10 CM KL2301

Standardization


WING DRILL 3 HOLES HEXBOLT .25 INCH

WING DRILL 4 HEXBOLT .25 INCH 5J868A

WING DRILL 4 HOLES HEXBOLT .25 5J868A


Matching

MATCH


WING DRILL 4 HOLES HEXBOLT .25 INCH 5J868A


Survivorship

Cross-ReferencesTable

Record Group

1 12 13 14 2

1

1

1

2

Group

Most Frequent, Most complete, Most recent

Survivorship – Product ExampleQualityStage™

23



Parallel Processing






for new uses






Part I:

Define&

Analyze&

Cleanse

Part II:

Transform&

Synchronize&

Virtualize

24

The Information Delivery Models

Synchronize

EAI & Replicate

(Near Real Time)

ReplicationPublication

Changes

Proactively deliver data to:– Drive integration– Data changes trigger action– Change only updating to:

• Eliminate batch window• Reduce latency

Create & synchronize copies – Distributed operations– High availability– Local copies for performance– DBMS migrations

Transform

ETL

(Scheduled)

WarehouseODS

Data Mart

Consolidation

Repackage & repurpose– Data Warehousing– Operational Data Stores– MDM– Interfacing– Conversion

Virtualize

Federate

Real Time

Federation

SQL

Seamlessly integrate with:– e-commerce– Self service– All data is an equal participant

25

Where everything fits

Synchronize

EAITranslation

Messaging

Transaction

Event driven

Low transformation

Near Real-time

Application to application

Synchronize

Replication

DB to DB copy

Low transformation

High availability

Structured

Near real-time

Transform

ETL

Transformation

Derived data

DB to DB

High volume

Multiple heterogeneous sources

Historical and current

Right-time

Structured, semi-structured

Virtualize

Federation

DB Query

Low transformation

Medium volume

Multiple Sources

Historical and Current

Real-time

Structured and Unstructured

Transactional Operational Analytical

Messaging AccessMovement/Enrichment

26


Deliver

Parallel Processing







Transform

DataStage for z/OS

DataStage MVS

DataStage for Linuxon System z

Understand



Cleanse


Part II:

Transform&

Synchronize&

VirtualizeRepackage and repurpose information

throughout the enterprise to ensure the information is provided where it is

needed and in the format that is appropriate for the user or application

27

Transform (move)

� Codeless visual design of data flows with hundreds of built-in transforms

� Optimized reuse of data integration objects

� Leverages parallel processing without requiring design changes

� Capable of supporting batch and real-time operations

� broadest range of sources

DataStage®

Extract, Transform and Load

28

Application Development, Deployment & Management With & Without DataStage (ETL)

Without DataStage With DataStage

• Intensive scripting

• Embedded SQL

• File management by hand

• Record management embedded in application code

• Application tied to specific hardware

• Multithreading

• Parallel debugging

• Migration from development to production

• Integration of best-of-breed commercial tools

• No code reusability

• Database interfaces

• I/O buffering

• Application management (checkpointing, performance monitoring, error and event handling)

Orchestrate Application Framework

(Code Execution Engine)

Application Execution

Components

Application AssemblyApplication Assembly

Components Provide 80% of

Your Application Logic

Application Framework provides

100 % of Your Application Infrastructure

DataStage®

29

Broadest range of Connectivity

General Access

Sequential File

Complex Flat File

File / Data Sets

Named Pipe

FTP

Compressed / Encoded Data

External Command Call

Parallel/wrapped 3rd party apps

EMC InfoMover

Web logs

Unstructured: e-mail, docs, etc.

Content Management Systems

Life Sciences

Standards & Real Time

WebSphere MQ

Java Messaging Services (JMS)

Java

XML & XSL-T

EBXML

Web Services (SOAP)

Enterprise Java Beans (EJB)

EDI

FIX

SWIFT

HIPAA

Enterprise Applications

JDE/PeopleSoft EnterpriseOne

Oracle Applications

PeopleSoft Enterprise

SAS

SAP R/3 & BI

SAP XI

Siebel

JDA

Ariba

Manugistics

I2

And more…

Legacy

Allbase/SQL

C-ISAM

D-ISAM

Datacom/DB

DS Mumps

Enscribe

Essbase

FOCUS

IDMS/SQL

ImageSQL

Infoman

KSAM

M204

MS Analysis

Nomad

Nucleus

RMS S2000

Supra

TOTAL

TurboImage

Unify

And many more….

RDBMS

DB2 (on Z, I, P or X series)

Oracle

Informix (IDS and XPS)

Ingres

MySQL

Netezza

Progress

RDB

RedBrick

SQL/DS

SQL Server

Sybase (ASE & IQ)

Teradata

Universe

UniData

NonStopSQL

And more…..

CDC / Replication

DB2 (on Z, I, P, X series)

Oracle

SQL Server

Sybase

Informix

IMS

VSAM

ADABAS

IDMS

NonStopSQL

Enscribe

DataStage®

30


Parallel Processing





Transform

DataStage for z/OS

DataStage MVS


Understand



Cleanse


Access and/or deliver information when and where needed, ensuring all

information is an equal participant … whether structured or unstructured,

distributed or mainframe-based, from within the enterprise or from without

Deliver

Federation Server & Classic Federation

Data Event Publisher& Classic Data EP

Replication Server & Classic Replication

Transformation Server

Part II:

Transform&

Synchronize&

Virtualize

31

Replication principles

� Replication is a technology solution that enables the continuous movement of data with minimal impact to mission critical systems.

� Only changes (delta’s)

� Copy of source

� How– Log-Based Change Data Capture

Use of native database logs to capture all insert, update and delete operations committed to a relational or non relational database.

� Alternatives to Database Logs• SQL Query (Date- and timestamp to subset) • Application Generated Tables• Database triggers• File Comparison (Diff Compare)




32

DB2

Journal LogRedo/Archive Logs

Transaction log

Publisher EngineAnd Metadata

Subscriber EngineAnd Metadata

BI Appliance

Audit Database

Message Queue

Web Services

Business Process

z/OSsource

Target

Scrape ApplyPush Confirm

ODS

Real Time/Continuous Architecture

IBM Information Server WebSphereDataStage

VSAM

IMS

Adabas

IDMS

Oracle

SQLServer

Sybase

DB2

Other

sources




33


Parallel Processing





Transform

DataStage for z/OS

DataStage MVS


Understand



Cleanse


Data Architect

Access and/or deliver information when and where needed, ensuring all

information is an equal participant … whether structured or unstructured,

distributed or mainframe-based, from within the enterprise or from without

Deliver





Part II:

Transform&

Synchronize&

Virtualize

34

What if you could…

Access data anywhere in your enterprise

� No matter where it resides

� Regardless of what format it is in

� Regardless of vendor

� Without creating new databases and without disruptive changes to existing ones

� Using standard SQL and any tool that supports JDBC or ODBC

� While the data appears to be in a single database


35

� CA Datacom

� DB2 Database

� IMS

� VSAM file

Federated Queries Make Integration as Easy as SQL

SELECT parameters_return_billto_key as BILL_TO_KEY,

billto_company_name,

parameters_return_shipto_key as SHIP_TO_KEY,

CASES_SHIPPED,

GROSS_SALES,

URL

FROM GETKEYSSOAP_GETKEYSREALTIME_NN,

GLOBAL_SALES_TRAN_NN,

BILLTO_DIMENSION,

URL_INVOICES

WHERE getkeysrealtime_ship_to_number = '13546'

and getkeysrealtime_ship_to_number = URL_INVOICES.shipno

and ltrim(rtrim(translate(ship_to_number, ' ', x'0a')))

= getkeysrealtime_ship_to_number

and parameters_return_billto_key = billto_key

and ltrim(rtrim(translate(sales_order_number, ' ', x'0a')))

= URL_INVOICES.orderno;

Single SQL Query Joins:


36

IBM WebSphere Classic Federation Server

Federation

�OLE DB�Excel�Flat files�WebSphere MQ�Custom-built

�WebSphere Adapter �SAP�PeopleSoft�Siebel

XMLWeb Services

PackagedApplications

SQL

�DB2 UDB �Informix�Oracle�Sybase �Teradata �SQL Server �ODBC

�VSAM,�Sequential

�DB2�IMS, �Adabas�CA-Datacom�CA-IDMS

MainframeFiles

MainframeDatabases

SQL

RelationalDatabases

WebOther

�XML�Web Services

Federation ServerClassic Federation Server for z/OS

� Access diverse & distributed information as

if it were in one system

� Industry leading query optimization with

single sign-on, unified views, and function

compensation

� Transactional write capabilities

across heterogeneous sources

� Visual tools for federated data

discovery & data modeling

Information virtualization


37

Linux for System z

Reports

Websphere

Information Service Framework

Logging

Authorization

Connector

Access

Services

Scheduling

Metadata

Reporting AccessAnalysis

WAS

DS,QS Client


Web Console

DS,QS

IA Client

IA WISD

WISD Client

ADMIN, BG,

Reports

Meta Brokers

Windows

Metadata

Workbench

� Common reusable servicesframework leverages the power

of a SOA environment

� Meta data repository promotes:

� reuse

� compliance to standards

� visual lineage

� impact analysis

DB2

MetadataIA/QS/WISD

Store

38

Linux for System z

Reports

Websphere


Logging

Authorization

Connector

Access

Services

Scheduling

Metadata


WASDB2

Metadata

PX Engine

DataStage Server

DSEngine

DS,QS Client


Web Console

DS,QS

IA Client

IA WISD

WISD Client

ADMIN, BG,

Reports

Meta Brokers

Windows

Metadata

Workbench

Linux for z Operational Environment� Full Information Server suite:

WebSphere QualityStage

WebSphere Information Analyzer

etcetera

� Robust, parallel processing:

Add IFLs to scale

� Minimal impact on z/OS costs:

Leverages IFLs and zIIPs

IA/QS/WISD Store

39

Linux for System z

Reports

Websphere


Logging

Authorization

Connector

Access

Services

Scheduling

Metadata


WASDB2

Metadata

PX Engine

DataStage Server

DSEngine

Data Event Publisher

Classic FederationServer

SEQ. VSAM IDMS Adabas Datacom

WebSphereMQ

DB2

SQL

SFTP, DRDA or Batch Pipes

z/OS

IMS

z/OSConnectors

Non-z/OSConnectors

Oracle, SQL Server, …

DS,QS Client


Web Console

DS,QS

IA Client

IA WISD

WISD Client

ADMIN, BG,

Reports

Meta Brokers

Windows

Metadata

Workbench

Rich connectivity with:

� Application & distributed data

� Hipersocket connectivity to z data� WS Classic Federation for non-DB2

� WS Data Event Publishers for CDC

IA/QS/WISD Store

40

Linux for System z

Reports

Websphere


Logging

Authorization

Connector

Access

Services

Scheduling

Metadata


WASDB2

Metadata

PX Engine

DataStage Server

DSEngine

Data Event Publisher

Classic FederationServer

SEQ. VSAM

WebSphereMQ

DB2

SQL

SFTP, DRDA or Batch Pipes

z/OS

IMS

z/OSConnectors

Non-z/OSConnectors

Oracle, SQL Server, …

DS,QS Client


Web Console

DS,QS

IA Client

IA WISD

WISD Client

ADMIN, BG,

Reports

Meta Brokers

Windows

Metadata

Workbench

IA/QS/WISD Store

Metadata

DB2IA/QS/WISD

Store

41

Are you still with me ?

How is everything linked ?

Change ERD-model in Rational Data Architect

What do I like to know ?

� Which table, tablespace, database is affected ?

� Which DataStage (ETL) program should I change ?

� Which datamart is based on in ?

� Which BI-report is related ?

� Who is responsible (key-user) for the BI-report ?

� Who has developed the DataStage program ?

Metadata

42

Where does a field of data in this report come from?

Source Tables

IBM Information Server

� Import & Browse Full BI Report Metadata

� Navigate through report attributes

� Visually navigate through data lineage across tools

� Increases trust and understanding of business information

Metadata

Information Server on System z - IBM · invoices (VSAM) – 3rd system that contains status of...

Documents

Transcript of Information Server on System z - IBM · invoices (VSAM) – 3rd system that contains status of...