Machine Learning in Finance

Dmitry Petukhov,Researcher & Developer @ OpenWay

𝑃 (𝐴 𝑗|𝐵 )=𝑃 (𝐴 𝑗 )𝑃 (𝐵∨𝐴 𝑗)

∑𝑖=1

𝑁

𝑃 (𝐴𝑖 )𝑃 (𝐵∨𝐴𝑖)

⟨Ω ,𝔘 ,ℙ ⟩ #AzureMLHackathon

Machine Learning in Finance. Machine Learning in our Life

Web

Social Networ

ksScience

Healthcare

Finance

Telecom

RetailLogisti

cSecurity

Financial MarketsInvestment BankingRetail BankingInsurance

Electronics

Proof: https://www.kaggle.com/wiki/DataScienceUseCases

Machine Learning in Finance. Machine Learning Use Cases in Finance

Financial Markets & etc. Retail Banking Insurance

Real-time Batch processingDuration

Market Assets Price

Prediction

Social Network Analysis

Fraud Detection

Risk Analysis

Compliance &

Regulatory Reporting

Advertising Campaign Optimizati

on

News Analysis

Customer Loyalty & Marketing

Improving operation

al efficiencie

s

Credit Scoring

Brand Sentiment Analysis

Personalized Product

Offering

Customer Segmentati

on

Reference: http://0xcode.in/big-data-in-banking

Machine Learning in Finance. Need & Opportunity

The Need for Banks.The Opportunity for You!

Machine Learning in Finance. Opportunities for all

For Banks’ IT departments, and…

For FinTech Startups…FinTech Incubators & Accelerators

Startupbootcamp Fintech AlfaCampBarclays AcceleratorMasterCard Start PathVisa Europe CollabQIWI Universe 2.0InspirAsia (Life.SREDA)Future Fintech

For Researchers & Enthusiasts…Competitions & Hackathons

SberbankAlfabankTinkoffBeeline…

«Венчурный фонд Life.SREDA прогнозировал, что объем инвестиций

в финтех ежегодно будет удваиваться. И ошибся: в прошлом году

вложения в сектор утроились по сравнению с 2013 годом, составив $6,8 млрд.

»Rusbase, 2015

Machine Learning in Finance. Concepts

Data

Hadoop 1.0

Infrastructure

Applications

Data flow

Computation

Closed

Expensive

Big Volume,Legal restrictions

Managed by bank

Machine Learning in Finance. Implementation

Azure Storage, NoSQL & RDBS storage as a Service

Hadoop 1.0

IaaS: Azure VM, Cloudera/HortonworksPaaS: HDInsight, Web/Worker role, Azure

Batch

Intelligent Systems

Data flow

Computation

Azure Machine Learning

You manage

Managed by AzureAzure Data Factory (Data pipe)

1. Retrieve data / 4. Consume prediction

Azure

Transactions Log

Raw Transactions DataProcessing System(Gate)

DMZ

Raw Transactions DataAzure Blob Storage

Transactions Processing Jobs QueueAzure Service Bus

Transactions Processing NodesAzure Worker Roles

Calculated Data Storage NoSQL Storage

Prediction System

HDInsight

Fraud PredictionAzure Machine Learning

Commands flow

Data flow

Auxiliary Services CRMs Data

Transactions Batch Processing System

Machine Learning in Finance. Antifraud: Architecture

Bank Antifraud System h(θ0, θn)

Fraud prediction APIAzure ML Web Services

POST, httpsREST APIJSON

Final Model

2. Pre-processing data 3. Create prediction modelSource: http://0xcode.in/antifraud-insights

1. Retrieve data

Machine Learning in Finance. Antifraud: retrieve data

External Services: geolocation, currency exchange rate, etc.Support Service Data Social Network & News aggregatorPlastic Card, Accounts, Merchants, IP-hosts, etc. black/white lists

Number of customer grows fast… Number of operations grows even faster…

Transactions Logwith request information

Banking CRM DataMerchant CRM DataWeb-clicks StreamWeb/mobile-applications & Backend services Log Data for Model

Join data

Problems

2. Pre-processing data 3. Create model

Machine Learning in Finance. Antifraud: processing data

Integration problems:Heterogeneous systems are often complexDifferent format (RDBS, NoSQL, text logs)Relationship inside data not explicitly

specifiedBig volume, grows fast

But this is not enough:Missing valuesInvalid valuesOutlinersPrivate Data

But and this is not enough:Legal restrictions: local & international (PCI

DSS)Different security policies inside bankFuzzy problem formulation

Integrat

ion

Quality

Policy

1. Retrieve data 2. Pre-processing data 3. Create model

Storage

ResourceManagement

ML Framework

Execution Engine

Local OS

Local Disc

Pyth

on R

untim

e

Yet A

noth

er

Runt

ime

scikitlearn

HDFS

YARN

MapReduce

Mahout

HDFS / S3

YARN / Apache Mesos

Spark

MLlib

HDFS / S3

YARN / Apache Mesos

Python / R on Spark

Python / Rtools

Spark

Local PC Hybrid Model Cluster (on-premises/on-demand)

somelibrar

y

Machine Learning in Finance. Infrastructure for Data Scientist

Low HighCost of deployment/ownership

Distributed FS

Dark Magic…

ML as a Service

Python / Rtools

1. Retrieve data 2. Pre-processing data 3. Create model

Machine Learning in Finance. Antifraud: Create model

1. Retrieve data 2. Pre-processing data 3. Create model

Demo

References

Machine Learning in Finance. References

Start for free from azure.com/ml Read Microsoft Machine Learning BlogExamine Azure ML documentation +free booksTake free MOOCs on MVA & EdXCommunicate on «Microsoft Azure Russia» group Make the world better place with Azure for Researchers Award program

© 2015 Dmitry Petukhov All rights reserved. Microsoft Azure and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

Thank you!

Q&ANow or later (send on email)

Ping meHabr: @codezombieFacebook: @code.zombi

LinkedIn: @dpetukhov

Read my tech code instinct blog (on http://0xCode.in/)

Machine Learning in Finance. Stay connected!

Download presentation from http://0xcode.in/azureml-hackathon-2015 or