Building a smarter planet: Financial Services

Exploding Demands for Big Data, Analytics, Risk Management, Ultra-low Latency and Compute Power Requires Optimized HPC Infrastructures

Building a smarter planet: Financial Services

Robert BrinkmanInfrastructure Architect for Banking and Financial MarketsIBM Banking Center of Excellence

Let’s build a smarter planet

2 © 2009 IBM Corporation2 © 2012 IBM Corporation

Panel

Dino VitaleDirector, Cross Technology Services

Morgan Stanley

Vikram MehtaVice President, IBM System Networking

IBM Corp.

Emile Werr VP, Global Data Services

Global Head of Enterprise Data Architecture NYSE Euronext

Nick WerstiukProduct Line Executive

IBM Platform Computing



Workload Optimized Stacks

Financial Markets Industry Imperatives• Re-engineer for profitable growth: Renewed focus on the customer, Near real

time analytics• Improve the trade life cycle: Cloud and business process outsourcing• Optimize enterprise risk management: Data driven transformation and common

industry services

Appliances& Packages

ApplicationsIBM Provided, ISVs, Partners, Custom

Grid Stack

Low Latency

Stack

TransactionStack

High Message Rates

Data Value Decay

Big Data

Big Compute

High TransactionRates

Complex Data Models

Specialized Workloads

Packaged Hardware and Software

CloudStack

Discrete Components or Applications

Variable Workload

Messaging and Security

Nothing below this

point

Nothing below this

point

Dino VitaleDirector, Cross Technology Services

Morgan Stanley

Nothing below this

point

Nothing below this

point

Morgan Stanley: Road to Compute As a Service Trends

• Maximize efficiency of compute infrastructure • Cost / run-rate• Utilization – more with less, linear scale, sharing• Operational normalization

Challenges• Phasing• Dynamic provisioning and scaling on-demand of resources to applications according to

varying business needs and SLA• Multi-tenant workload protection• Application design and dependency management• Utility charge-back model options: pay-per-use, fixed allocation, hybrid approach • Sharing resources based on work load supply and demand • BCP

Convergence opportunities with “Big Data”• Increasing data volumes • Adaptive/real-time Scheduling• Resource management• Metrics / Data mining

ON-DEMAND DATA IN HIGH PERFORMANCE ENVIRONMENTEmile Werr, VP, Global Data Services

Global Head of Enterprise Data Architecture & Identity Management

Big Data (billions of transactions and multi-terabyte captured daily) Speed and business agility are essential to our business Different viewpoints and data patterns need to be analyzed Data coming out of a Trading Plant is not user-friendly Correlating disparate data & integration Moving large data around is expensive and complex System Capacity requirements need to efficiently handle 5x of our Avg daily volume. Data Spikes – the day after Flash Crash volume peaked over18.4 Bn transactions for NYSE Classic Matching engine (this excludes Options and other markets like Arca, Amex, Liffe, Euronext, etc.) Transaction volume growth sustained year-over-year Data needs to be readily available for a min of 7 years for Compliance It is too expensive to keep it all online Change is constant

Technology Challenges

Global Data Services 4

8

Data Architecture Practice

Financial Services, Regulatory & Compliance Expertise

Order arrived:BUY 10 @ 20.09

Full Quote Size- Best Quote size from the last published best quote

Price level for calculating Shares Ahead & Shares Available

Trading systems generate vast transaction volumes at high speeds The GRID is utilized to transform, normalize and enrich the time-series data using massive parallel computing. This is done as EOD or Intra-Day batch processing. Date-Level Table scans (Queries) need also massive parallel processing (MPP) Appropriate technologies need to be utilized (10gb Network, Virtualized CPU/MEM, Appliance Databases, Scalable Storage Pools)

USE CASE: Market Reconstruction for Trading Surveillance

The Electronic Book (NYSE DBK) and Market Depth needs to be reconstructed and accessible via Fast Database

Who Traded Ahead or Interpositioning ? This can be answered by a Database Query

Data Lifecycle Management Methodology

Data Capture

End-User Workflow

Data Transformation & Archive

User Analytics“Business Intelligence”

On-Demand Data (ODD)

Trading DataMarket DataRef DataUser Generated Data

Transform, Normalize, EnrichPartition, compress and archive in storage poolsCreate Metadata (mappings)

EnterpriseSystems

Secure Data Access & NavigationLoad, Extract, Stream, Filter, Transform, PurgeUser-driven Data Mart Provisioning (“Sandboxing”)Schema Change Capture (“Data Structure Lineage”)

Utilize MPP Databases & HDFSIntegrate Reporting ToolsFacilitate User CollaborationCapture Knowledge (KM)Automate Data Archive & Purge

Global Data Services 3

FeedHandler

FeedHandler

Continuous Flow (Trickle Batch)

files

Data PumpData Pump

Data Capture Data Virtualization & Abstraction Business Demand

Managed Data Services & Data Flow Automation

10

Transformation & Archive

Scale-Out Grid Fabricdistributed CPU/MEM

MessageBus

Storage Pools

AppsAdmins

AnalystsData ScientistsResearchers

Fast Processing & Data Movement

Scalable

Reliable

Simplified Access & Administration

File & Database Virtualization

Common Secured Access

Automation & Workflow

Standardization & Consistency

Agile Framework – Metadata Driven

Metering, Monitoring &Tracking

HadoopNetezzaAnalytics Data Warehouse

DataProvisionin

g

DataTools

Data Services



Vikram MehtaVice President, IBM System Networking

IBM Corporation



Nick WerstiukProduct Line Executive

IBM Platform Computing



Workload Compute Intensive Data IntensiveCompute and DataIntensive

Data type StructuredRDBMS, Fixed records

UnstructuredVideo, E-Mail, Web

All – Structured + Unstructured

ApplicationUse Case

Pricing

BIReportingStreaming

Risk Analytics

AML/Fraud

Sentiment Analysis/CRM

Genomics ETL

Gaming

Trading

CEP

Simulation

Characteristics“Real Time” QuarterlyDailyIntraday Monthly

Infrastructure Dedicated servers,Appliances, FPGAs

Disk & Tape, SMP & Mainframe,

SAN/NAS InfrastructureData Warehouses

Compute grid,Data caches, In-memory grid,Shared services CPU + GPU

Commodity processors + storage

Convergence of Compute and Data



14

Resource Orchestration

C

Workload Manager

C C C C C

C C C C C C

D

D

D

D

D

D

D

D

D

D

D

D

C C C C C C

Metadata generation,File classification,

Batch analysis

Search, Analysis, Concept Recognition

Data Intensive Apps

A A A A

A A A A

A A A A

A A A A

B

B

B

B

B

B

B

B

B

B

B

BB B B B B B

A B C DGeo-spatial integration,

Name classificationSignal processing

Support for Diverse Workloads & Platforms



Latency

Scale

Inefficient scheduling, polling model & heavy-weight transport

protocols limit scalability.

OtherGrid Servers

Symphony

With a zero-wait time “push model” and efficient binary protocols, Symphony

scales until the “wire” is saturated

Why IBM Platform Symphony is faster and more scalable



HPC Cloud – Multiple Approaches and Paths to Value

Infrastructure Management

Infrastructure Management

• Cluster consolidation into an HPC Cloud

• Self-service cluster provisioning and management

• Workload-driven dynamic cluster

Build out a more dynamic HPC infrastructure as their HPC Cloud

HPC “In the Cloud”

HPC “In the Cloud”

• ‘Bursting’ to Cloud Providers• Hosted HPC in the cloud• Enable HPC Cloud Service

Providers

Leverage the public cloud opportunity, either to tap into additional resources, or offer their own HPC cloud services

Questions

Building a smarter planet: Financial Services

Building a smarter planet: Financial Services

Documents

Transcript of Building a smarter planet: Financial Services