Think Analytics with your IMS Applications & Data.ppt Analytics with your IMS... · Think Analytics...
Transcript of Think Analytics with your IMS Applications & Data.ppt Analytics with your IMS... · Think Analytics...
© 2015 IBM Corporation
Think Analytics with your IMS Applications & Data
Hélène Lyon
Distinguished Engineer & CTO, z Analytics & IMS for Europe
IBM Systems, Software Sales, Europe *
© 2015 IBM Corporation
Agenda
� Introduction
� A New Need: Analytics in Business Applications
� Focus On Analytics & Decision Management
� Focus on IDAA with your IMS Data
� Focus on Hadoop & Spark
2
© 2015 IBM Corporation
Analytics have become Business Critical
These applications may support a large concurrent user population with a high volume of requests
Failure of these applications for any length of time can result in lost business, customer turnover, reputational risk, etc.
Today, analytics are integrated with transaction systems running on the mainframe and are critical to the business
These applications need to deliver insight in real-time or near real-time and integrate with business processes
Prevent Fraud Reduce Customer Churn
Business Critical Analytic applications require superior qualities of service, including a high degree of reliability, continuous availability, scalability,
security and low data latency
Cross-sell/up-sell to customers Operational Reporting
3
© 2015 IBM Corporation
More users across the organization depend upon business critical analytics
Bu
sin
es
s C
riti
ca
lA
na
lyti
cs
Tra
dit
ion
al
Bu
sin
es
s A
na
lyti
cs
Customer Service & Support(e.g., call centers, sales personnel)
C-level Mgt.
Company Management
Customers
(e.g., external, web)
User Community
Analysts
(e.g., marketing, research)
CriticalSimpleVery Large
Analytic requirements have expanded
Many
Numberof Users
TransactionVolume
TransactionType
Qualities of Service
LessImportantFew Small Complex
4
© 2015 IBM Corporation
… but IT remains aligned to the old way of doing business analytics.
� Some reluctances from the past–Core business is primary, analytics is secondary!
• On core business side: High volume transactions and batch processing running
concurrently, shared everything DB
• On analytics side: Low volume complex queries and batch reporting, shared nothing
DB
–Cost of running analytics on z … without looking at all hidden costs concerning data movement – latency, data governance, IT complexity
–Impact on operational performance & security
� Key drivers to change IT perception–Recognized business value of advanced analytics, including in-
transaction analytics and “big data” analytics–Awareness of z position as primary Systems of Records–Technology availability to build a fully-integrated, end-to-end system
that executes “intelligent” business processes
5
© 2015 IBM Corporation
Traditional z Systems Customer Analytics Landscape
� Today challenges– Complexity
• both in systems management and in applications
– Difficulties in supporting real time analytics
– Inability to match ever more demanding SLA requirements
– Data protection & governance issues– High total cost of ownership
� Historical reasons– Different data access patterns
• impact on performance
– EDW as the data integration hub– Different life-cycle characteristics– Different Service Level Agreements
(SLA)• Lack of broadly available workload
management capabilities• Choice of lower cost-of-acquisition
offerings
zData
zAppsz/OS based Transaction &
Batch Workloads
“Transaction & Apps” Data
Core Business Applications and Data
ODS
EDW &Data Mart
ETL or ELTExternal Data & Master Data
Existing BA Infrastructure
ETL or ELT
“IT” Data IT Data
6
© 2015 IBM Corporation
z Systems Analytics Areas – Evolution & Extension
zData
zAppsz/OS based Transaction &
Batch Workloads
“Transaction & Apps” Data
Core Business Applications and Data
New Business Critical AnalyticsAdd, Extend, Modernize Analytics Services
New ODS
EDW &Data Mart
ETL or ELTExternal Data & Master Data
Optimized Analytics
Infrastructure with z in mind
Analytics in Business Applicationsor
In-Transaction Real Time Analyticsor
Embedded analytics
Data Serving analytics including “Big Data”
for both structured and unstructured data
Analytics on “IT” Data
ETL or ELT
“IT” Data
Unstructured data
Enhanced data
Analytics Platform
ODS, DW, DM
Data lakes
…
IT Data
A
A
A
A
A Analytics Components
7
© 2015 IBM Corporation
� Analytics in Business Applications– Just ReThink business applications with Hybrid Transaction & Analytical influence
– Bring analytics in z/OS based applications and don’t limit your thinking about what are
analytics! You can really do things you couldn’t do before!
– Discuss about zApps improvement even if analytics is today on distributed.
� Analytics on “IT” Data– Improve IT Operation & Run with IT Analytics – Predict / Search / Optimize
� Analytics Data Serving for both structured and unstructured data– Bring analytics to the data – Don’t extend what has been done in the past.
– Make zData easily available for reporting tools, distributed applications or ERPs
– Don’t miss the Spark / Hadoop trend – Make it just relevant with zData without moving
data off z
– Provide a way to improve data governance & data security
z Systems Analytics Solutions Areas
8
© 2015 IBM Corporation
IMS Data supports a range of analytics solutions today
Use Case Solution
IBM DB2 Query Management Facility• Real-time BI and dash
boarding of IMS Data
• Analyze IMS data with
unstructured data
InfoSphere Big
Insights
• Deep and fast analysis
of IMS data
• Visualize all data assets
including IMS data
Watson Explorer
72%say greatest value will come
from analyzing transactional data
IBM DB2 Analytics
Acceleraor
9
© 2015 IBM Corporation
… and we continue to integrate analytics with core IMS transactions
� Right-time insight at point of impact
� Increased business agility
55%of enterprise applications need
mainframe to complete transactions
Use Case Solution
• In-transaction analytics
Operational decision
management with
externalized business
rules
IBM Operational
Decision Management
on z/OS
• Execution of predictive
analytics in real time
IBM DB2 Analytics
Acceleraor
• Imbedded queries on
historical data
SPSS Real-Time
scoring & Zementis
solution
10
• Callout to non z/OS
based analytic serviceSpark Services
Watson Services, …
© 2015 IBM Corporation
Agenda
� Introduction
� A New Need: Analytics in Business Applications
� Focus On Analytics & Decision Management
� Focus on IDAA with your IMS Data
� Focus on Hadoop & Spark
11
© 2015 IBM Corporation
Focus on “Analytics in Business Applications” What is “real-time analytics?”
�This is the time during which a transactional
event is still occurring
–Someone is shopping at a store
–Someone is on the phone with a customer
service representative
–An electronic payment is being processed
–…
�By acting on threats or opportunities as they
arise…
–Revenues can be increased (up-sell, cross-
sell)
–Customer churn can be reduced
It’s about leveraging the power of analytics “in the transactional moment” to achieve a more favorable outcome for a transactional event, while the event is in progress
The person visiting a
store buys more than he
or she otherwise would
have
What would have been an
over-payment is stopped
before it gets out the door
The person on the phone,
who was about to cancel
a service, instead re-ups
A commission of fraud is
stopped before it is
effected
12
© 2015 IBM Corporation
Simple Performant AdaptiveIterative
Focus on “Analytics in Business Applications”Technology Pillars - A 100% z/OS based solution for Real-Time Analytics
BusinessRules
� Policy
� Regulation
� Best Practices
� Know-how
Business Critical Queries
Tra
nsa
ctio
n &
B
atc
h W
ork
loa
d
� API Programmable Orchestration to coordinate all activity in the same unit of work and ensure best performance
� IBM DB2 Analytics Accelerator augment analytics capabilities on historical data.
� Predictive modeling, business rules and orchestration together enable the most effective decisions
1
2
3
Add-on
Integration with other Service Provider, BPM
or ESBs
Orc
he
str
ati
on
w
ith
Sim
ple
A
PI
� Risk� Clustering� Segmentation� Propensity
Predictive Analytics
Model Execution
Predictive Analytics
Model Creation
13
© 2015 IBM Corporation
Focus on “Analytics in Business Applications”A key point about the real-time analytics technology pillars
�Three technology pillars enable real-time analytics in a z/OS runtime environment:1. Business Rules with Operational Decision Management – ODM on z/OS
2. Predictive analytics with SPSS Real-Time scoring adaptor or with Zementis solution
3. Business critical queries with IBM DB2 Analytics Accelerator for z/OS based data as
well as distributed data when needed by zApps
–Optionally: integration with Hadoop or Spark solutions, CPLEX mathematical algorithm for
Optimization and Watson-based cognitive services
�Each of the three pillars just mentioned delivers significant value on its own.
�Combined, they can work together to–Uncover patterns of events and behaviors
–Use those patterns to develop models to predict future occurrences
–Use those models to identify threats and opportunities as they arise
–Respond immediately to identified threats and opportunities in an autonomic fashion
An organization can go right to “ultimate” real-time analytics capability, or take a phased approach – and realize benefits at each intermediate stage (and the order of enabler implementation is up to the client)
14
© 2015 IBM Corporation
Implementation - Today Architecture for z/AppsA simple View of a Unit of Work
� An input layer processes the data coming from a client.
– At the end of this process we have an input area filled with EBCDIC characters.
� An orchestration layer analyses the input area and implements a sequence of calls to business logic.
– Usually subroutines ☺
� The business logic layers implement “services”, directly calling resource managers with simple API
– Access and update data managed by DB2, VSAM, or IMS DB
– Eventually send a message to MQ
� An output layer builds the data to be sent back to the client and finishes the unit of work.
– All updates are committed using the two phase commit protocol.
� Business Logic
� Data Layer
z/OS Resource Manager
CICS or IMS transaction
GettingRequest
Sending Answer
B
z/OS Apps & Data
Orchestration
Business Logic n°1
Business Logic n°p
D
D
Business Logic n°2 D
…
Mainframe = Backend
B
D
15
© 2015 IBM Corporation
Implementation - Adding Analytics Services in z/AppsPositioning z as Service Requester
� Co-Location for optimum performance & scalability
� Real-Time Analytics Simple API – SQL API for IDAA
– SQL API for SPSS RTS
– ODM API for ODM
� Other Integration Options– Java Integration for
• Zementis score execution• Spark supported API
– Web service calls also available• WOLA API to call local web services• Many integration options for
CICS or IMS Apps
– Using an external Integration Layer• IBM Integration Bus
z/OS Resource Manager
CICS or IMS transactionGettingRequest
Sending Answer
B
z/OS Apps & Data – The Benefit of Co-Location
InitialOrchestration
Business Logic n°1
Business Logic n°p
D
D
DB2IDAA
RT
A
Orc
hestr
ati
on Historical
Queries
Mainframe = Frontend
Score
Rules
16
© 2015 IBM Corporation
Agenda
� Introduction
� A New Need: Analytics in Business Applications
� Focus On Analytics & Decision Management
� Focus on IDAA with your IMS Data
� Focus on Hadoop & Spark
17
© 2015 IBM Corporation
Operational & Analytical Decision Management
Operational Decision Management Analytical Decision Management
Business Processes, Applications & Solutions
DecisionServices
Business
Rules
Internal & External Data
� Policy� Regulation� Best Practices� Know-how
Scenario Analysis& Simulation
Learn from the factsBuild automatically a predictive
model by self learning from data
Learn from the expertsAuthor a rule-based model capturing expert knowledge
Business rules and predictive scoring together enable most effective decisions
Externalized decisions are easy to change
Centralized decisions enable reuse and consistency..
Predictive
Analytics
� Risk� Clustering� Segmentation� Propensity
18
© 2015 IBM Corporation
Express the decision logic with business rules
Business language
1 rule for the business = 1 IBM ODM artifact
Transparency, visibility
ifthe product type is car insurance and the client has a car insurancethen
do not recommend the product;
Callable operational decision services
Retention next
best action
service
Shared and platform-agnostic services
Stateless: The calling application passes the context
Synchronous
Externalization, consistency and traceability
Controlled access
Decoupled from the application logic
Externalization and agility
Externalize & centralize the business logic Rule Repository
Bring the IT and the lines of business together Rule Designer Decision Center
One single view on rules
Test, simulation, versioning
Collaboration and governance
WebSite
CallCenter
The 4 IBM ODM Business Rules Essentials
19
© 2015 IBM Corporation
Business Problems & Benefits of ODM
Challenges for most z Systems clients
1. Consolidation, isolation, extension of COBOL & PL/I application portfolios
2. Ability to react to increasing pace, variety and volume of change requests
3. Sharing business rules across platforms & channels
4. Ensuring seamless business experience in migration/ application evolution
Benefits of the ODM Approach
� Cost savings
– Shorter change cycle, without increased business risk
– Rule engine processing is zIIP eligible
� Improved agility
– Improved Time to Market
– Manage business decisions in natural language
– Decouple development and business decision change
lifecycles
� Single version of the Truth
– Consolidated and shared expression of business policy
– Maintainable with a Center of Competency model
� Incremental Adoption
– Deploy decision methodology one decision at a time
– Focus on decisions that need to change often & quickly
– Expand adoption of “market validated” decisions
20
© 2015 IBM Corporation
IBM Operational Decision Manager
Rule DesignerEvent Designer
Rule Solutions for Office
Decision Center Versioned Assets
Rule Execution Event Execution Decision Monitoring Connectors
Manage
Decision ServerConsole
Design Monitor
Decision Server
Deploy Measure
VisibilityCollaborationGovernance
Define Update
Web Services – API - GUI
DevicesEnterpriseApplication
POS BPM CRM
Social
Event Widgets Space Business Console
Enterprise Console
Access and ControlDecision Artifacts
21
© 2015 IBM Corporation
Decision Management on z/OS Comprehensive Flexibility System z
z/OSDistributed or z
System
COBOL & PL/I
Applications
Dep
loy
zRES
Workstation
Rule Designer+ COBOL & PL/I Management
Decision
Service
Bu
IMS
COBOL & PL/I
Applications
Decision
Service
Business Rules
zRES
Decision
Service
Decision
Service
Business Rules
RES on WAS for z/OS
Decision
Service
z/OS Batch
COBOL & PL/I
Applications
Decision Center+ COBOL & PL/I Management
Architect,
Application
Developer
Business Analyst,
Business Manager
Decision Center Repository
CICS
Decision
Service
Business Rules
22
© 2015 IBM Corporation
Simplified Integration with zRES API
� Connect to Execution Region– call ‘HBRCONN’ using HBRA-CONN-AREA
� Populate Header with parameter data
� Connect to Execution Server– call ‘HBRRULE’ using HBRA-CONN-AREA– IF HBRA-CONN-COMPLETION-CODE =
HBR-CC-OK THEN
. . .
� Disconnect from Execution Region– call ‘HBRDISC’ using HBRA-CONN-AREA
01 HBRA-CONN-AREA.
10 HBRA-CONN-EYE PIC X(4) VALUE 'HBRC'.
10 HBRA-CONN-LENTH PIC S9(8) COMP.
10 HBRA-CONN-VERSION PIC S9(8) COMP VALUE +2.
10 HBRA-CONN-RETURN-CODES.
15 HBRA-CONN-COMPLETION-CODE PIC S9(8) COMP.
15 HBRA-CONN-REASON-CODE PIC S9(8) COMP.
10 HBRA-CONN-FLAGS PIC S9(8) COMP VALUE +1.
10 HBRA-CONN-INSTANCE PIC X(24).
10 HBRA-CONN-RULE-COUNT PIC S9(8) COMP.
10 HBRA-CONN-RULE-MAJOR-VERSION PIC S9(8) COMP.
10 HBRA-CONN-RULE-MINOR-VERSION PIC S9(8) COMP.
10 HBRA-CONN-RULEAPP-NAME PIC X(256).
10 HBRA-RESPONSE-AREA.
15 HBRA-RESPONSE-MESSAGE PIC X(512).
10 HBRA-RA-PARMETERS.
15 HBRA-RA-PARMS OCCURS 32.
20 HBRA-RA-PARAMETER-NAME PIC X(48).
20 HBRA-RA-DATA-ADDRESS USAGE POINTER.
20 HBRA-RA-DATA-LENGTH PIC 9(8) BINARY.
10 HBRA-RESERVED.
15 HBRA-RESERVED02 PIC X(12).
15 HBRA-RESERVED03 PIC X(64).
15 HBRA-RESERVED04 PIC X(64).
15 HBRA-RESERVED05 PIC X(128).
15 HBRA-RESERVED06 PIC X(128).
23
© 2015 IBM Corporation
Analytical Decision Management – Predictive ScoringA key-element of the real-time decisioning strategy
� Like the real world, predictive models are not binary– Understanding how closely a pattern of behavior matches a
known pattern of bad (or good) behavior can help uncover
crimes or non-obvious opportunities
� Predictive models detect patterns– Deviation from expected behavior can isolate bad (or good)
behavior, trigger additional actions or new targeted marketing and up-sell / cross-sell offers
� Predictive models can have many variations– Can be built to assess only specific transactions or more
generically for all transactions
– Multiple layers of models can be invoked for increasing
sophistication of analysis, triage leading to further inspection
of contributing factors and weights
24
© 2015 IBM Corporation
Predictive Scoring - A 2 steps approach
AnalysesSegments
Profiles
Scoring models
...
Scoring CustomerService
Center
Data:Demographics
Account activity
Transactions
Channel usage
Service queries
Renewals
…
Identify predictive models/patterns found in
historical data
Use those predictive models
with variables to score
transactions & identify the best possible future
outcomes
Practical scoring approaches�Off-line: Batch Scoring
�On-line: External scoring function�On-line: Within a transaction, in-DB, real
time
Step 1 – Build the predictive model Step 2 – Execute the predictive model
Model Creation
enhanced with
IDAA V5SPSS Model
Model Execution
enhanced with
SPSS RTS
25
© 2015 IBM Corporation
Real time scoring - What has changed?
CICS or IMStransaction
Select customer scoring data from database using customer id
Customer scoring data
Get score viaWeb Service using customer scoring data
Scoring Model
Customer score
CICS or IMStransaction
End of transaction
Option 1 : Real time analytics process with external scoring function
CICS or IMS transaction
Get score via database using customer id
Customer score
CICS or IMS transaction
End of transaction
Option 2 : Real time analytics process with in-database local scoring
Assumption : customer data needed to obtain scoring from model are located in operational database. If historical data are needed, process will vary.
Start of transaction Start of transaction
Scoring Engine
Scoring Engine
26
© 2015 IBM Corporation
Real time scoring of the transactional data in DB2 for z/OSSPSS & DB2 for z/OS
� IBM SPSS Real-Time Scoring Adapter for DB2 on z/OS
– Enables customers to score predictive
models built by IBM SPSS Modeler
directly within a specific online
transaction processing transaction that
is running with DB2 for z/OS.
� Business Value– Delivers better, more profitable
decisions, using the latest data, at the
point of customer impact
– Enables more informed customer
interaction • Improves customer service • Increases revenue per customer ratio• Heightens customer retention
– Improves fraud identification and
prevention• Reduces risk and exposure
DB2 for z/OS
Application w/latest data
Real-Time Score/Decision Out
ETLData In
R-T, min, hr, wk, mth
Reduced Networking
End to end solution
Support for both in-transaction and in-database scoring on the same platform
Consolidated Resources
DB2 for z/OS Data
Historical Store
Copy
SPSS Modeler
For Linux on System z
Scoring Algorithm
Business System / OLTP
Scoring Engine
27
© 2015 IBM Corporation
Core software for Real-Time Decision solutions on z Systems
� Predictive Analytics – Product: IBM SPSS Modeler with
Scoring Adapter for z or Zementissolution
– Delivers better, more profitable decisions, at the point of customer impact
– Improves accuracy by scoring directly within the transactional application against the latest committed data
– Delivers the performance needed to meet operations SLAs
– Avoid data governance and security issues, save network bandwidth, data copying latency, disk storage
– Same high qualities of service as operational systems
– Easier to incorporate scoring into applications
� Business Rules– Product: IBM Operational Decision
Manager for z/OS– Automate and manage frequently
occurring, repeatable business decisions
– Codifies business policies, practices and regulations
– Enables changes to be easily made by business people
– Automates decision making with the fidelity of an expert
– Centralized, externalized decisions enable consistency and reuse
– Manage business decisions in a natural language
– Decouple development and decision change lifecycle
28
© 2015 IBM Corporation
More Information
� Paper from Decision Management Solutions – James Taylor– Transforming Operational Systems: From Transactional Systems to Decision
Management Systems
� Videos on the value of real-time scoring for in-transaction analytics– https://www.youtube.com/watch?v=_vII97Ylq0Y
– https://www.youtube.com/watch?v=CWQNJ2ystic&feature=youtu.be
� Link to Zementis partnership & solutions– http://zementis.com/products/uppi/z4ibmzsystems_goz4z/
– http://www.prweb.com/releases/2015/07/prweb12867889.htm#!
– http://enterprisesystemsmedia.com/article/zementis-announces-predictive-analytics-
integrated-with-ibm-z-systems#sr=g&m=o&cp=or&ct=-
tmc&st=%28opu%20qspwjefe%29&ts=1442215130
29
© 2015 IBM Corporation
First National Bank of South Africa
� Business challenges – Increased business pressure due to international
expansion, regulation and consumer demand– Risk scoring rules replicated in each instance of engine with
poor reusability and increased testing needs– Inflexibility of the old rules engine to accommodate changes
� Solution– Shift from Business Rules to Decision Management– Start small with a single LOB – Personal Banking, grow
later based on success
� Benefits–Agility in order to rapidly cope with new regulations and
business conditions–Seamless integration into FNB Mainframe IMS Cobol
environment–Address change and complexity by revamping smoothly a
strategic system–Design of rules in a componentized format which forces
reusability.
Gaining business agility to better and faster adapt to changes with simplified and standardized decision management, shared between all Enterprise IT, including existing mainframe environment.
Solution components:� IBM IBM Operational Decision Management
� On z/OS
� On distributed
“ ODM provides us with a platform to deliver business rules through an agile framework making design decisions a lot simpler.” — Jay Prag - CIO Channel Integration | Core Technology Solutions | FNB Premium
30
© 2015 IBM Corporation
European Insurance Company
� Business Challenges– Rethink the underwriting process to become more flexible
and to respond faster to market demand with new products and services.
– Adapt quickly tariffs on competition behavior to reduce severe losses in the corresponding segment.
– Reduce IT cost for tariff rule maintenance
� Solution– Implement a Decision Management System – Easy integration with z/OS based applications– Rules sharing between all Enterprise
� Business Benefits– Increase flexibility in pricing & underwriting rules
management to allow quick changes in response to evolving market conditions
• From 5 months to minutes
– Reduce Time-To-Market for new products– Integrate the end-client together with the broker and back-
office as part of the Digital Strategy– Reinforce cross & up-selling capabilities as a key element to
increase average premium– Providing all users with a coherent user applicative
experience
Business Benefits
Project Approach
1. Designed a decision management solution using ODM on IBM PureApplication Service on SoftLayer
2. Enabled that solution for ODM zRES called from IMS applications.
3. Sharing rules between mainframe and the cloud.
The project started in June and the first Eligibility Rules application went in production in October.
31
© 2015 IBM Corporation
Agenda
� Introduction
� A New Need: Analytics in Business Applications
� Focus On Analytics & Decision Management
� Focus on IDAA with your IMS Data
� Focus on Hadoop & Spark
Hybrid Analytics Solution using IBM DB2 Analytics Accelerator for z/OS: http://www.redbooks.ibm.com/redbooks/pdfs/sg248151.pdf
32
© 2015 IBM Corporation
Hybrid Transaction & Analytical Data Processing
The hybrid computing platform on z Systems
Supports transaction processing
and analytics workloads concurrently, efficiently and
cost-effectively
Delivers industry leading performance for mixed workloads
The unique heterogeneous scale-out platform in the industry
Superior availability, reliability and security
TransactionProcessing
AnalyticsWorkload
33
© 2015 IBM Corporation
Deep DB2 integration within z Systems & z/OS
Applications
Application Interfaces(standard SQL dialects)
DBA Tools, z/OS Console . . .
Operational Interfaces(e.g. DB2 Commands)
DataManager
BufferManager
IRLMIBM DB2 Analytics
Accelerator
. . .Log
Manager
DB2 for z/OS
Superior availability,
reliability, security
Workloadmanagement
z/OS onz Systems
Superior
performance
on analytic
queries
PureData for
Analytics
34
© 2015 IBM Corporation
SPEED• Dramatically improve query response – up to
2000X faster – to support time-sensitive decisions• Right-time. Low latency. Trusted. Accurate.
SIMPLICITY• Simplify infrastructure, reduce ETL and data movement
off-platform • Non-disruptive installation
SAVINGS• Minimize data proliferation• Lower the cost of storing and managing historical data• Free up compute resources
SECURITY• Safeguard valuable data under the control and security
of DB2 for z/OS • Protected. Secured. Governed.
A workload optimized, appliance add-on to DB2 for z/OS that enables the integration of analytic insights into operational processes to drive business critical analytics & exceptional business value.
IBM DB2 Analytics Accelerator for z/OS
35
© 2015 IBM Corporation
IBM DB2 Analytics Accelerator Query Execution Process Flow
DB2 for z/OS
Optimizer
IDA
A D
RD
A R
equesto
r
DB2 Analytics Accelerator
Application
Application
Interface
Queries executed with DB2 Analytics AcceleratorQueries executed without DB2 Analytics Accelerator
Query execution run-time for
queries that cannot be or should
not be off-loaded to IDAA
SPU
CPU FPGA
Memory
SPU
CPU FPGA
Memory
SPU
CPU FPGA
Memory
SPU
CPU FPGA
Memory
SM
P H
ost
36
© 2015 IBM Corporation
DB2 Analytics Accelerator Loader for z/OS
� An IBM Branded product– PID: 5639-OLA, S&S:5639-OLB
� A Utility that supports loading “Any Data” into DB2 for z/OS front-end and the DB2 Analytics Accelerator, or only into the IDAA
� Allows loading of data from– DB2 image copy files
• Also: DB2 image copy + log files (i.e., “play an image copy forward”)
– Data from other sources • IMS, VSAM• DB2 for LUW, Oracle, SQL Server• And more – if you can get the data into a sequential file, the Loader can get it into the Accelerator
� Example: IMS Database• Initial Implementation: Routing IMS Queries thru DB2 only with data movement optimization
IMS
Catalog
IDAA
IMS
Databases
Define IMS tables & load data into IDAA
DB2
Catalog
DB2
OptimizerQueries
Define IMSTable Metadata
• Complete table refresh
• No Trickle Feed
37
© 2015 IBM Corporation
DB2 Analytics Accelerator Loader for z/OS - Intended direction
DB2 for z/OS
DB2 LogsDB2 Image
CopiesDB2 Load
FilesVSAM
Other non-
relationalIMS DB
Accelerator Only TablesAccelerator Only Tables
IBM DB2 Analytics Accelerator
AcceleratorTables
AcceleratorTables
DB2 for z/OS
DB2 Base Tables
IBM DB2 Analytics Accelerator Loader for z/OS intended directionIBM DB2 Analytics Accelerator Loader for z/OS intended direction
DRDA and Federated
• Load DB2 data and Non-DB2 data directly to IDAA• Easy consolidation of enterprise data on secure z
platform• Exploit Accelerator to join DB2 non-DB2 data
• New analytic workload on DB2 for z/OS• Trending, Fraud detection, Capacity Planning, etc.• Analytics of SMF data, DB2/IMS performance data,
etc.
Use Cases
38
© 2015 IBM Corporation
More Use Cases than Ever
Hardware
evolution
More query
acceleration
Capabilities for
more use cases
Improved
transparency and
management
Examples
• Faster hardware
• Self-encrypting disks
• Static SQL support
• Closing gaps in
unsupported SQL
• Accelerator-only
tables
• Improved data
maintenance
• Call home
• Management of
failures
39
© 2015 IBM Corporation
DB2 Analytics Accelerator – Usage scenariosHow organizations leverage the Accelerator today
Accelerate
existing
workload
Reduce
IT sprawl
Derive new
business insightInclude external
& historical data
Reduce IT sprawl for analytics
If the data is offloaded to a distributed data warehouse or data mart
• Simplify and consolidate
complex infrastructures, low latency, reliability, security and TCO
Rapid acceleration of Business Critical Queries
If the data is analyzed on the mainframe
• Performance improvements
and cost reduction while retaining z Systems security and reliability
Derive business insight from z/OS transaction systems
If the data is not being analyzed yet
• One integrated, hybrid platform,
optimized to run mixed workload
• Simplicity and time to value
Improve access to historical data and lower storage costs
If the analysis is based on a lot of historical data
• Performance improvements
and cost reduction
40
© 2015 IBM Corporation
DB2 Analytics Accelerator Version 5.1Adding new dimensions in functionality to expand use cases
Rapid acceleration of Business Critical Queries
Adding application support fortemporary objects (QMF, Multi-step Reporting, IBM Campaign, etc.)
Individual ad-hoc analysis that provides a Data Scientist Work Area
• Insight into now to maximize
business opportunities in today’s
dynamic environment
Accelerate
existing
workload
Reduce
IT sprawl
Derive new
business insightInclude external
& historical data
Reduce IT sprawl for analytics
• Business agility through
simplified architecture
Improve access to historical data and lower storage costs
• Simplified access to information
– when you need it
Integrate more data sources
for analytics, using DB2 Analytics Accelerator Loader for z/OS to
assimilate with IMS data or data from other sources
In-database transformation to
support Data Stage Balanced Optimization and the
consolidation of ETL/ELT processing in DB2 for
z/OS
Derive business insight from z/OS transaction systems
In-database analytics to accelerate predictive analytics applications; SPSS/INZA data mining and in-database modeling can be processed within the Accelerator
• Real-time, actionable business processes
• Environment to efficiently, continuously test and improve analytic results to drive better customer understanding
Deliver Right-
Time Analytics to drive better business outcomes
41
© 2015 IBM Corporation
� IBM DB2 Analytics Accelerator – Primary Product Page
– Prerequisites and Maintenance
– Guides and manuals
– Knowledge Center
� Customer Testimonials– https://engage.vevent.com/index.jsp?eid=556&seid=68284&code=brand
� Redbooks and Redpapers– Reliability and Performance with IBM DB2 Analytics Accelerator Version 4.1
– Optimizing DB2 Queries with IBM DB2 Analytics Accelerator for z/OS
– Hybrid Analytics Solutions using IBM DB2 Analytics Accelerator for z/OS V3.1
– IBM DB2 Analytics Accelerator: High Availability and Disaster Recovery
– SAP Integration with IBM DB2 Analytics Accelerator for z/OS
� All TechDocs available at the following link.
Sources of Information
42
© 2015 IBM Corporation
IBM DB2 Analytics Accelerator Strategy
• Complement DB2's industry leading transactional processing capabilities
• Provide specialized access path for data intensive queries
• Enable real and near-real time analytics processing
• Execute transparency to the applications
• Operate as an integral part of DB2 and z Systems
• Reusing industry leading PDA's query and analytics capabilities and take advantage of future enhancements
• Extend query acceleration to new, innovative usage cases, such as:
– in-database transformations
– advanced analytical capabilities
– multi-temperature and storage saving solutions
Enable DB2 transition into a truly universal DBMS that provides best
characteristics for both OLTP and analytical workloads.
Ultimately allow consolidation and unification of transactional and analytical data stores
DB2 for z/OS
In-databaseTransformation
QueryAccelerator
StorageSaver
OLTP
AdvancedAnalytics
43
© 2015 IBM Corporation
Agenda
� Introduction
� A New Need: Analytics in Business Applications
� Focus On Analytics & Decision Management
� Focus on IDAA with your IMS Data
� Focus on Hadoop & Spark
44
© 2015 IBM Corporation
Big Data, Hadoop & Spark History
Developed in 2009 at UC Berkeley AMPLab, open sourced in 2010, Spark has since become one of
the largest OSS communities in big data, with over 200 contributors in 50+ organizations
“Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and
analytics – should consider Spark for its in-memory performance and the breadth of its model.
It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for
machine learning and graph analysis.”
Gartner, Advanced Analytics and Data Science (2014)
45
© 2015 IBM Corporation
Hadoop Introduction
� Open source software framework from the Apache Software Foundation that supports data-intensive highly parallel applications
– High throughput, batch processing
� Designed to run on large clusters of commodity hardware
– Lots of cores – inexpensive cores working all the time
• Processors fail – that’s ok – just replace them
– Lots of redundant disks – really inexpensive disks
• Disks crash – that’s ok – just replace them
– But nothing in Hadoop requires commodity cores and disks!
� Two main components– Hadoop Distributed File System (HDFS)
• Self-healing, high-bandwidth clustered storage
– MapReduce engine• A simple, powerful framework for parallel
computation
46
© 2015 IBM Corporation
Spark processes and analyzes data from ANY data source
� Apache Spark is an open source, in-memory processing engine designed for Big Data.
– in-memory processing capability, – interfacing with multiple data sources, – ability to be written with multiple
programming languages.
� It was designed with 3 key tenants in mind– Fast, simple, able to run in many
environments.
� Apache Spark is NOT– A data store
• Spark attaches to other data stores but does not provide its own
– Only for Hadoop• Spark can work with Hadoop (especially
HDFS), but Spark is a separate, standalone system
– Only for machine learning• Spark includes machine learning and does
it very well, but it can handle much broader tasks equally well
– A replacement for Streams• Spark Streaming is micro-batching, not true
streaming, and cannot handle the real-time complex event processing that true streams do
Hadoop Database MainframeData-
warehouse
Business Applications and Business Intelligence
Source : TypeSafe , Apache Spark Survey 2015, Databricks - How Companies are! Using Spark
47
© 2015 IBM Corporation
Spark Advantages over MapReduce
� Spark is complementary to Hadoop, but much faster with in-memory performance
� MapReduce limitation– Suitable only for batch processing jobs, implementing interactive jobs and models
becomes challenging.
– Iterative jobs and SQL queries involves a lot of disk I/O for each repetition!
� Spark Processing– Innovative execution engine that support cyclic data flow and in-memory computing.
– Based on a functional programming model similar to MapReduce with the ability to load
data into memory which makes it suitable for interactive analysis, SQL queries and
complex jobs.
Query 1
Query 2
Query 3
Query 4
Result 1
Result 2
Result 3
Result 4Distribute memory
One-time processing
…
48
© 2015 IBM Corporation
z Systems & Apache Spark: Strategic Direction
Spark node
Spark node
Spark node
Linux on z
Systems
Leverage LoZ
virtualization benefits
Power
Spark node
Spark node
Spark node
Spark node
x86
Leverage call center,
external, social,
sentiment data …
Unified Analytics PlatformFlexibility & Agility with multi-language support
Efficient Structure – 100x vs. in-memory map reduce
Rich set of built-in functions with consistent APIs: Spark SQL, Spark Streaming, GraphX, …
DB2 VSAM
CICS
WAS
Spark node
z/OS
Spark node
Spark node
Leverage z/OS data and transactions
IBM DB2 Analytics Accelerator
IMS
Fast, expressive, cluster computing system leveraging in-memory framework for analytics
49
© 2015 IBM Corporation
Hadoop & z Systems Integration – 2 Use cases
1 - Mainframe clients want to incorporate sensitive mainframe data
into exploratory analytic models
What has been holding them back?
z/OS
DB2 VSAM
QSAM IMS
SMF RMF
Logs
2 - Mainframe clients want to incorporate into zApps analytics based on non-z data like
social media, machine generated data, e-mail
What has been holding them back?
There is risk associated with having copies of sensitive data existing outside the mainframe
Performance & Integration are key inhibitors for real-time analytics.
50
© 2015 IBM Corporation
Use Case 1: Challenges
� Address governance, security, and other operational practices
� Leverage Big Data without losing control of data
Challenge How to address?
Clients are worried about data
governance as the data moves off of
z. Data is considered secure as long
as it is on z. How do you secure
sensitive data once it has left z?
z needs to be in "control" of the data.
How can existing security policies be
applied?
The ingestion of data from z into the
Hadoop environment is turning into a
bottleneck
Need high speed / optimized
connectors between traditional z/OS
LPARs and a z-controlled-Hadoop
environment
zData
51
© 2015 IBM Corporation
Use Case 1: Populating System z Hadoop clustersIBM & Veristorm partnership
� A secure pipe for data– Data never leaves the box– RACF integration – no need for separate or
special credentials– Data streamed over secure channel using
hardware crypto, SSL
� Easy to use ingestion engine– Native data collectors accessed via graphical
interface– Light-weight; no programming required– Wide variety of data sources supported,
including JDBC for non-mainframe data sources
– Automatic code page conversions– COBOL copybook Parsing and presentation,
Metadata translation– Automated job scheduling
� Fast and low resource utilization– HiperSockets and 10 Gbps internal transfer– Streaming technology does not load z/OS
engines or require DASD for staging
IBM InfoSphere System z
Connector for Hadoop
IBM and Veristorm are collaborating on tighter product integrations for System z
customers
52
© 2015 IBM Corporation
Use Case 2: ““““Augmented Analysis””””
� Very large amounts of non-relational data originate outside z Systems– e.g. e-mails sent by customers, tweets, posts to company Facebook page
� Analyze sentiments and identify customers who are dissatisfied with company– Words ‘cancel’, ‘terminate’, ‘switch’ or synonyms thereof– Names of competitors
� Gather names and e-mail addresses of customers at risk
� Join these results with operational data– Alert agents of at-risk customers– Agents work with customer and offer a promotion to stave off defection
All of the
operational
applications &
data are here
There is also
potentially relevant
data here
zData
zApps
53
© 2015 IBM Corporation
Business View - Claiming disability allowance
Hadoop or agency
Data from Social Media
sites analyzed with
Text analytics
“Unable to work”
Work Status
-
z Systems
“Dude – awesome
vacation”
Facebook Post
Core Business
Processes
DB2 for z/OSIMS DB
Deterrent for fraudsters -Cost Savings for the business
Make payment or investigate
Gateway to
Hadoop
54
© 2015 IBM Corporation
Business View - Claiming disability allowanceStep by Step
� The claimant submit a transactional request to claim for disability.
� Later a batch application identifies all entitled parties for a disability payment.– This complex query could benefit of the DB2 Analytics Accelerator.
� We add in this batch a “fraud detection” step as a request for information on claimants from an Hadoop cluster or an amalgamator agency.
– A z/OS based Apps kicks off a Hadoop job by using Co:z product from Doventail for example.
– Same zApps can soon use Spark services to interrogate any source of data.
� Information on claimants is gathered from social media and other sources – and the results is sent to the requester.
– Any Hadoop Solution ingests data that usually is not ingested by established structured data analysis systems like DB2 for z/OS, .e.g. email from all clients sent to an insurance company, facebook, …
– Analytics can search for family of words, can identify customer sentiment from key words in emails or based on comments in social media.
� Data is sent back to z/OS in a predefined format and can be “joined” to the claimants main record.
– Then it can decide what to do as Next Best Action.
55
© 2015 IBM Corporation
Agenda
� Introduction
� A New Need: Analytics in Business Applications
� Focus On Analytics & Decision Management
� Focus on IDAA with your IMS Data
� Focus on Hadoop & Spark
Are you ready to modernize your IMS IT Environment to answer Business needs?
56