© 2016 IBM Corporation
IBM Analytics on z Systems
NRB Mainframe Day 2016
Analytics on z Systems
Focus on Real-Time
Hélène Lyon – Distinguished Engineer & CTO, z
Analytics & IMS for Europe
IBM Systems, zSoftware Sales, Europe
Email: [email protected]
Tel: +33 6 84 64 19 39
© 2016 IBM Corporation
Executive Summary
Analytics is one of the hot-button topics for agile enterprises.
– End users are looking more than ever for relevant and accurate information for
which their enterprise assets are the fundamental enabler.
– Challenges are about removing the time lag between analytics and business
impact.
One way is to augment Data Warehouse capabilities to provide Right-Time
Data Analytics including Big Data support.
– Business use case example: Improved product development and sales by
getting improved knowledge of customers past behavior.
Another way is to implement in-line analytics, aka in-transaction analytics
aka “Analytics in Business Applications” aka Real-Time Analytics
– Add analytics in z/OS based workload to improve its business value
– Business use case example: Fraud detection at the time of the fraud
2
Bring the analytics to the data for Right-Time Insight
Bring the analytics in the application for Real-Time Decision Management!
© 2016 IBM Corporation
Market Trends & Analytics Needs
End customers /
consumers
Aim to improve level of
detail and frequency of
existing analytics
Have innovative ideas for
new analytics
Are driven by increasing
demand in regulatory
requirements
Lines of business/
Data scientists
both have a high demand in secure
data management
• Transactional
workload runs on the
mainframe
• Very bad perception
of the mainframe
with regards to
modernity
• DW / analytical data environment on distributed platforms
• Huge amount of analytics applications
Business IT
Overnight
batch / ETL
for analytical
purposes
Scoring Rules
A
Analytics ComponentsA
Analytics
executed
outside of
core
business
applications
zDatazApps
3
Drive increased
transactional workload
through mobile
Ask for more
personalized services
and offering
© 2016 IBM Corporation
z Systems Analytics Areas – Evolution & Extension
zData
zApps
Core Business Applications and Data
ODS
EDW &
Data
Mart
ETL or ELT
ETL or ELT
IT Data
A
Today
Only Data
Analytics
z/OS based Transaction & Batch Workloads
“Transaction & Apps” Data
External Data & Master Data
“IT” Data
© 2016 IBM Corporation
z Systems Analytics Areas – Evolution & Extension
Optimized
Analytics
Infrastructure
with z in mind
zData
zAppsz/OS based Transaction &
Batch Workloads
“Transaction & Apps” Data
Core Business Applications and Data
ODS
EDW &
Data
Mart
ETL or ELT
External Data & Master Data
ETL or ELT
“IT” Data IT Data
New Business Critical AnalyticsAdd, Extend, Modernize Analytics Services
Analytics in Business
Applications
or
In-Transaction Real Time
Analytics
Right-Time Data Analytics
including “Big Data”
for both structured and
unstructured data
Analytics on “IT” Data
Enhanced
data
Analytics
Platform
ODS, DW,
DM
Data lakes
…
A
A
A
A
A Analytics Components
Unstructured Data
Co
mp
ete
in
th
e c
og
nitiv
e e
ra w
ith
hig
h q
ua
lity,
rea
l tim
e in
sig
hts
ba
se
d o
n IB
M z
Syste
ms S
olu
tio
ns
Link to IBM zRTA page
5
© 2016 IBM Corporation
z Systems Analytics Areas - How it complements existing Analytics Environments?
IBM
DB
2 A
naly
tics
Accele
rato
r
In transaction rules and
score executionIntraday capability for ad-hoc
queries & predictive analytics
Availability of historical data (in
raw format)
Accelerated reporting to
fulfill internal and regulatory
requirements
Ability to transform
data before offload to
DWH or reportingAbility to create new
SPSS models at any time
Quasi Real Time
availability of data
for analytics
Instant access to raw data
for new report generation in
hours instead of days
Load and merge of ANY non
DB2 z/OS data
Scoring Rules
A
zDatazApps
Scoring
Rules
Explore data to
uncover hidden
insights
6
A
© 2016 IBM Corporation
Focus on “Analytics in Business Applications” What is “in-transaction real-time analytics?”
This is the time during which a transactional
event is still occurring
–Someone is shopping at a store
–Someone is on the phone with a customer
service representative
–An electronic payment is being processed
–…
By acting on threats or opportunities as they
arise…
–Revenues can be increased (up-sell, cross-
sell)
–Customer churn can be reduced
It’s about leveraging the power of analytics “in the transactional moment” to achieve a
more favorable outcome for a transactional event, while the event is in progress
The person visiting a
store buys more than he
or she otherwise would
have
What would have been an
over-payment is stopped
before it gets out the door
The person on the phone,
who was about to cancel
a service, instead re-ups
A commission of fraud is
stopped before it is
effected
7
© 2016 IBM Corporation
Focus on “Analytics in Business Applications”What are Data Scientists doing?
DATA
Design
Build
Train
TestDeploy
Score
Act (rules)
BUILD / TRAIN / TEST phases– Data scientists choose their modeling
program (SPSS Modeler, SAS, R, Python) and the data to analyze. They train & test the models.
– They can use Spark to be more productive and benefit from more algorithms.
DEPLOY phase– Data scientists export model and deploy it
in any runtime environment, potentially z/OS.
SCORE phase– “in-transaction” scoring with an optimized
response time can ONLY be done in the z/OS transactional system.
– Frequently scoring is executed in batch, or as a service close to the data warehouse, but certainly not with a 200 milliseconds response time.
ACT phase– Business rules can be deployed to
replace or complement real-time scoring.– Colocation of rules with transaction is
needed to optimize performance.
The scoring and the rules execution can
take place in the transaction.
8
© 2016 IBM Corporation
Business Challenge:
Improve fraud detection capabilities through added insight: real-time access to most current data,
integrated advanced analytics within the business flow, access to aggregated data across geographic
locations, merchants, issuer and history
Solution:
Enhance existing fraud detection with predictive scoring integrated with fraud detection transaction at
bank-transaction speeds and SLAs for *preventive* capabilities before the transaction completes
Benefit:
Institution can save significant amounts on fraud detected prior to transaction completion
Improved customer service using more advanced analytics, avoiding unnecessary account deactivations
Reduced call center costs
Able to integrate with institution’s existing fraud capabilities to provide enhancement
Transaction
Pre-
Authorization
SystemDetermine Fraud Risk
Continue, Pend/Investigate, etc.
Banking
Transaction
Initiated
Internal /
External
Fraud Rules
Engine
Transaction Data
Historical
Transaction
Data
IBM DB2
Analytics
Accelerator•Models are built outside z
Systems or on z Systems
•Models are deployed to
z/OS
Use Case: z/OS Optimized Fraud Detection for Banking Transaction
Enhanced with
Outcome from
Fraud Analytics
Alert
Data
Existing or Enhanced
Fraud Analytics
Rules mining &
predictive analytics
Other Data Other Data
Account Data
9
© 2016 IBM Corporation
Use case: In-transaction, synchronous predictive analytics
• Validate inputs and channel
• Prepare for submission to System of Record
Initiate Transaction
• Invoke *existing* risk scoring mechanisms
• Get Predictive Score for transaction risk
• Determine Action
Determine Risk & Disposition • Continue
processing
• Stop Transaction
• Initiate alternate actions …
Determine Transaction Disposition
Risk Scoring Example
Customer
Account Data
Transaction
Data
Transaction
HistoryMerchant Data
In-transaction predictive analytics can address issues such as data latency, timely
execution, tight SLAs, governance of sensitive data, access to both transaction and
external data, skills gaps.
10
© 2016 IBM Corporation
Use Case: IT Simplification to support growing demand with same SLAs
z Systems PR/SM
z/OS
CICS
PL1 App
DB2
Ru
ntim
eD
eve
lop
me
nt
Linux / VMware / Intel
Blaze Advisor
(WAS)
Binning Logic
PMML(Scoring model)
SOAP/HTTP
Linux / VMware / Intel
SAS
Deployment of PMML & Binning Logic
Binning Logic
PMML
(Scoring model)
Linux / Intel
JEE
Application Server
Java Split Logic
RMI/IIOP
z/OS
z Systems PR/SM
JVM
ODM Rules
Execution
Server
(zRES)
DB2
Java Split Logic
Binning Logic
CICS AOR
PL1 App
Trx
Data
Cust
Data
Acct
Data
z/O
S c
olo
ca
ted
Ru
ntim
e
PMML
(Scoring model)
Transaction Fraud Analysis done by FICO/Falcon &
Blaze Advisor on distributed servers
Scoring Model Creation with SAS
Transaction Fraud Analysis on z/OS
© 2016 IBM Corporation
Model creation outside z/OS or in IDAA
Potential z/OS or non-z/OS service integration
In-Transaction Real-time Analytics technology pillars – RuntimeA 100% z/OS based solution – colocation & performance
Business
Rules
Policy
Regulation
Best Practices
Know-how
Business Critical
Queries
Tra
nsaction &
Ba
tch
Wo
rklo
ad
IBM DB2 Analytics Accelerator augment
analytics capabilities on historical data.
Predictive modeling, business rules and
orchestration together enable the most effective
decisions
API Programmable orchestration to coordinate
all activity in the same unit of work and ensure
best performance
1
2
3
Add-on
Other Analytics services
Orc
he
str
ati
on
wit
h S
imp
le
AP
I
Risk
Clustering
Segmentation
Propensity
Predictive Analytics
Model Execution
Predictive
Analytics
Model
Creation
Federated Data Analytics
12
© 2016 IBM Corporation
Three technology pillars enable real-time analytics in a z/OS application runtime environment
– (1) Predictive analytics with Zementis partner solution deployed on z/OS – (2) Business Rules with Operational Decision Management – ODM on z/OS– (3) Business critical queries with IBM DB2 & DB2 Analytics Accelerator
Each of the three pillars just mentioned delivers significant value on its own.
Combined, they can work together with the data & event based exploration tools– Uncover patterns of events and behaviors– Use those patterns to develop models to predict future occurrences and identify threats
and opportunities as they arise
Optionally adding when SLAs for specific use case permit– Predictive analytics execution outside z/OS, for example in SPSS Collaboration and
Deployment Services– Integration with Spark solutions for federated data analytics, CPLEX mathematical
algorithm for Optimization and Watson-based cognitive services
In-Transaction Real-Time Analytics technology pillars - Runtime
An organization can go right to “ultimate” in-transaction real-time analytics
capability, or take a phased approach – and realize benefits at each intermediate
stage (and the order of enabler implementation is up to the client)
13
© 2016 IBM Corporation
Business Benefits – Help LOB Innovate!
– Capability to richly analyze in real-time 100% of the transactions without performance
impact
– Agility to accept changes in business conditions by updating on the fly rules and models
– Auditability of business rules
– Easy to incorporate scoring in applications
IT Benefits
– When Performance Matters!
– No additional skills needed for COBOL/PLI development team
– Continous availability with non-disruptive operational analytics inherited from the core
systems (Hardware, operating system, CICS, IMS, DB2).
– Highest security level by keeping exchanged data within z Systems (no outbound data
transit)
– Simplified system management - the integrated architecture leverages existing
environment
In-Transaction Real-Time Decision solutions on z SystemsValue of deploying components on z/OS
Simple Performant AdaptiveIterative
14
© 2016 IBM Corporation
Access to Mainframe data without moving it outside z Systems secured environment.
Reduce data latency Minimize Cost & Complexity Improve Data Governance & Security
A new approach for Enterprise Analytics with z Systems
Enrich transactions with real-time analytics allowing optimized decision management
Improve Business value of mainframe applications
Conviction 3: z Systems is the Most Securable for Data!
Conviction 4: Integrate Open Source technology in the z
Systems Environment
Conviction 2: Accelerate insight and simplify implementation with z13 & IBM
DB2 Analytics Accelerator
Centralized data security Tracking of activity to address audit and compliance requirements Highly available cryptography
Did you miss the LinuxONE announcement? Linux Your Way, Linux Without Limits, Linux Without Risk
Conviction 1: Bring the analytics to the data for Right-Time insight
Bring the analytics in the application for Real-Time Decision Management!
15
© 2016 IBM Corporation
Video: In-transaction Analytics
https://www.youtube.com/watch?v=uPDfhYoQlHk
2.5 minutes
16
© 2016 IBM Corporation
Focus on Transactional Decision Management ODM, a Business Rules Engine (and more)
Operational Decision Manager can be used to extend application
functionality with improved agility, productivity, and consistency through
centralization of business rules and decision management
– What if you wanted your application to decide on how to react on a score
returned by a predictive model invoked in-transaction?
Operational
decisions
ApplicationApplication
Decision logic
Without ODM With ODM
Hard coded decisions: difficult to change
Impedes rules re-use by other systems
Externalized decisions: easy to change
Centralized decisions enable reuse and consistency
17
© 2016 IBM Corporation
Focus on Transactional Decision Management ODM brings the IT and Business World together
“customer”
• the name of …
• the birthday of …
• the number of accidents
of …
• the … is a high risk driver
Business Object Model Rule Vocabulary Business Rule Language
Developer IT / Business Rule Developer / Business User
“client”
• le nom du ...
• l’anniversaire du ...
• Le nombre d’accidents du
...
• le ... est un conducteur à
risque …
01 CUST
05 NAME
05 AGE
05 NUMACCIDENTS
05 RISKLEVEL
Rule: High risk driver
if
the birthday of customer is after 12/9/1975 and
the number of accidents of customer is at least
3
then
set the customer as a high risk driver
Règle: Conducteur à risque
si
L’anniversaire du client est après le 12/9/1975
et
le nombre d’accident du client est au moins 3
alors
Classer le client comme conducteur à risque
Automatic generation
of the rule vocabulary.
Comprehensive industry
focused business terms
to define its data and
associated actions.
Localizable vocabulary
18
© 2016 IBM Corporation
Focus on Predictive Analytics
Reduce Fraud /
Loss with detect &
prevent
Increase customer
retention
Drive up-sell and
cross-sell
Mitigate risk based
on predictions
A Universe of Data Example
• Pattern of consumer behavior over last 5 years of purchase
history: card present at time of sale
• Current transaction:
- Over $1000, card not present
- Merchant falls into category typical for this consumer’s
spend
• Likelihood of fraudulent transaction
• Develop insight to questions that are not binary ‘yes’ or ‘no’ likelihood of a pattern
• Predictive scores provide complementary insight to rules-based systems
19
© 2016 IBM Corporation
Predictive Scoring - A 2 steps approach
AnalysesSegments
Profiles
Scoring models
...
Scoring Customer
Service
Center
Data:Demographics
Account activity
Transactions
Channel usage
Service queries
Renewals
…
Identify predictive
models/patterns found in
historical data
Use those predictive models
with variables to score
transactions & identify the
best possible future
outcomes
Practical scoring approachesOff-line: Batch Scoring
On-line: External scoring function
On-line: Within a transaction, in real time
Step 1 – Build the predictive model Step 2 – Execute the predictive model
SPSS Model
Creation enhanced
with IDAA V5SPSS
Model
Model Execution on
z/OS with Zementis
20
© 2016 IBM Corporation
SPEED• Dramatically improve query response – up to
2000X faster – to support time-sensitive decisions• Right-time. Low latency. Trusted. Accurate.
SIMPLICITY• Simplify infrastructure, reduce ETL and data movement
off-platform • Non-disruptive installation
SAVINGS• Minimize data proliferation• Lower the cost of storing and managing historical data• Free up compute resources
SECURITY• Safeguard valuable data under the control and security
of DB2 for z/OS • Protected. Secured. Governed.
A workload optimized, appliance add-on to DB2 for z/OS that enables
the integration of analytic insights into operational processes to drive
business critical analytics & exceptional business value.
IBM DB2 Analytics Accelerator for z/OS
© 2016 IBM Corporation
IBM DB2 Analytics Accelerator – Four Main Usage Scenarios
On average, 70% of the data that feeds data warehousing and
business analytics solutions originates on the z Systems
platform (financial information, customer lists, personal
records, manufacturing…)
Understand your workload and data:
Where transaction source data
is being analyzed todayUse Case Benefits
If the data is analyzed on the
mainframe
Rapid Acceleration of
Business Critical Queries
Performance improvements and cost
reduction while retaining System z
security and reliability
If the data is offloaded to a distributed
data warehouse or data mart
Reduce IT Sprawl for
analytics
Simplify and consolidate complex
infrastructures, low latency, reliability,
security and TCO
If the data is not being analyzed yetDerive business insight from
z/OS transaction systems
One integrated, hybrid platform,
optimized to run mixed workload.
Simplicity and time to value
If the analysis is based on a lot of
historical data
Improve access to historical
data and lower storage
costs
Performance improvements and cost
reduction
1
2
3
4
22 © GUIDE SHARE EUROPE 2016
© 2016 IBM Corporation
IBM DB2 Analytics Accelerator Strategy
• Complement DB2's industry leading transactional
processing capabilities
• Provide specialized access path for data intensive
queries
• Enable real and near-real time analytics processing
• Execute transparency to the applications
• Operate as an integral part of DB2 and z Systems
• Reusing industry leading PDA's query and analytics
capabilities and take advantage of future
enhancements
• Extend query acceleration to new, innovative usage
cases, such as:
– in-database transformations
– advanced analytical capabilities
Enable DB2 transition into a truly universal DBMS that provides best
characteristics for both OLTP and analytical workloads.
DB2 for
z/OS
In-database
Transformation
Query
Accelerator
Storage
Saver
OLTP
Advanced
Analytics
23
Ultimately allow consolidation and unification of transactional and
analytical data stores.
© 2016 IBM Corporation
Imperatives to implement z Systems-centric Data Lake Scenarios
• Reduce complexity of information supply chain, e.g.
Avoid data movement
Simplify data transformation
Use in-DB transformation
Use temporary tables structures
• Leverage state-of-the-art technology, e.g.
HW accelerators
Special-purpose appliances
In-memory processing
• Use federation technique whenever possible, e.g.
Federated SQL queries, leaving data in place
Federated analytical processing, leaving data in place
• Adhere to innovative and novel BI/DWH concepts, e.g.
Limit # of data marts and data cubes
Use aggregation on the fly
Allow for agile usage patterns
1
3 4
2
for Apache Hadoop
© GUIDE SHARE EUROPE 201624
© 2016 IBM Corporation
Focus on Federated Data Analytics with Apache Spark
Addressing limitations of Hadoop MapReduce programming model
– No iterative programming, latency issues, ...
Using a fault-tolerant abstraction for in-memory cluster computing
– Resilient Distributed Datasets (RDDs)
Can be deployed on different cluster managers
– YARN, MESOS, standalone
Supports a number of languages
– Java, Scala, Python, SQL, R
Comes with a variety of specialized libraries
– SQL, ML, Streaming, Graph
Enables additional use cases, user roles, tasks, e.g.
– Data scientist tasks: developing analytical models
– Using languages other than Cobol or Java: R, Scala ...
© 2016 IBM Corporation
Enterprise-grade, native z/OS distribution of the Apache Spark in-memory analytics engine
coupled with optimized data abstraction services that enable efficient access to a broad set
of structured and unstructured data sources in place with no data movement.
Avoid latency, costly processing and security concerns associated with data movement.
IBM z/OS Platform for Apache Spark is a no fee program product. Subscription and Support is available for purchaseLearn more Ecosystem of tools - Jupyter projecthttps://www.ibm.com/developerworks/java/jdk/spark/
Simplify data access
Reduce data access complexity with seamless Spark access to enterprise data.
Speed development
Take advantage of your expertise with popular programming languages such as Scala, Python, and SQL.
Accelerate results
Use in-memory processing for fast results. Apply a range of analytics including machine learning, iterative, streams and batch.
Developers can collaborate and build tools around IBM z/OS Platform for Apache Spark in the new GitHub organization and obtain pre-built distributions of open source components via IBM developerWorks.
Focus on Federated Data AnalyticsIBM z/OS Platform for Apache Spark
26
IBM EMEA Announcement ZP16-0103
March 22, 2016
IBM z/OS Platform for Apache Spark,
V1.1 offers an enterprise-grade platform
to enable Apache Spark on z/OS with
integrated data abstraction and
virtualization services
NEW!!
Link to Forrester paper
© 2016 IBM Corporation
VSAM
z/OS
and *many* more . . .
Key Business Transacti
on Systems
Spark Applications: IBM and Partners
AdabasIMSDB2 z/OS
Distributed
Teradata
HDFS
Apache Spark Core
Spark Stream
Spark SQL
MLib GraphX
RDDDF
RDD
DF
Optimized data access
IBM z/OS Platform for Apache Spark
and *many* more . . .
Spark can run on z/OS close to z/OS-based Applications & Data
Values:Data-in place analytics, without need to ETL or move data for analytic purposes
Optimized access and z/OS governed ‘in-memory’ capabilities for core business data
Unique capability to access almost all z/OS sources with Apache Spark SQL & many non-z data sources
Almost all zIIP eligible
Integration of analytics across core systems, social data, website information, etc.
27
© 2016 IBM Corporation
Spark can access z/OS based Data – Accelerated or enabled by IBM DB2 Analytics Accelerator
IMS
VSAM
z/OS
DB2
load
load
accelerated table
accelerator-only table
DB2 API
Application
SQL
val results =
sqlContext.sql("SELECT name FROM people")
JDBC to retrieve data from database
Application
Spark could run
on any platform
• z/OS
• Linux on z
• Linux on x86
• …
IBM Data Server Driver for JDBC and SQLJ
SQL access to DB2 z/OS can be
transparently accelerated by IDAA
Other data sources (IMS, VSAM,…)
loaded into IDAA accelerator-only tables
can be accessed using DB2 APIs
IDAA
sync
28
Top Related