Machine Learning in Finance
-
Upload
dmitry-petukhov -
Category
Data & Analytics
-
view
753 -
download
0
Transcript of Machine Learning in Finance
Machine Learning in Finance
Dmitry Petukhov,Researcher & Developer @ OpenWay
𝑃 (𝐴 𝑗|𝐵 )=𝑃 (𝐴 𝑗 )𝑃 (𝐵∨𝐴 𝑗)
∑𝑖=1
𝑁
𝑃 (𝐴𝑖 )𝑃 (𝐵∨𝐴𝑖)
⟨Ω ,𝔘 ,ℙ ⟩ #AzureMLHackathon
Machine Learning in Finance. Machine Learning in our Life
Web
Social Networ
ksScience
Healthcare
Finance
Telecom
RetailLogisti
cSecurity
Financial MarketsInvestment BankingRetail BankingInsurance
Electronics
Proof: https://www.kaggle.com/wiki/DataScienceUseCases
Machine Learning in Finance. Machine Learning Use Cases in Finance
Financial Markets & etc. Retail Banking Insurance
Real-time Batch processingDuration
Market Assets Price
Prediction
Social Network Analysis
Fraud Detection
Risk Analysis
Compliance &
Regulatory Reporting
Advertising Campaign Optimizati
on
News Analysis
Customer Loyalty & Marketing
Improving operation
al efficiencie
s
Credit Scoring
Brand Sentiment Analysis
Personalized Product
Offering
Customer Segmentati
on
Reference: http://0xcode.in/big-data-in-banking
Machine Learning in Finance. Need & Opportunity
The Need for Banks.The Opportunity for You!
Machine Learning in Finance. Opportunities for all
For Banks’ IT departments, and…
For FinTech Startups…FinTech Incubators & Accelerators
Startupbootcamp Fintech AlfaCampBarclays AcceleratorMasterCard Start PathVisa Europe CollabQIWI Universe 2.0InspirAsia (Life.SREDA)Future Fintech
For Researchers & Enthusiasts…Competitions & Hackathons
SberbankAlfabankTinkoffBeeline…
«Венчурный фонд Life.SREDA прогнозировал, что объем инвестиций
в финтех ежегодно будет удваиваться. И ошибся: в прошлом году
вложения в сектор утроились по сравнению с 2013 годом, составив $6,8 млрд.
»Rusbase, 2015
Machine Learning in Finance. Concepts
Data
Hadoop 1.0
Infrastructure
Applications
Data flow
Computation
Closed
Expensive
Big Volume,Legal restrictions
Managed by bank
Machine Learning in Finance. Implementation
Azure Storage, NoSQL & RDBS storage as a Service
Hadoop 1.0
IaaS: Azure VM, Cloudera/HortonworksPaaS: HDInsight, Web/Worker role, Azure
Batch
Intelligent Systems
Data flow
Computation
Azure Machine Learning
You manage
Managed by AzureAzure Data Factory (Data pipe)
1. Retrieve data / 4. Consume prediction
Azure
Transactions Log
Raw Transactions DataProcessing System(Gate)
DMZ
Raw Transactions DataAzure Blob Storage
Transactions Processing Jobs QueueAzure Service Bus
Transactions Processing NodesAzure Worker Roles
Calculated Data Storage NoSQL Storage
Prediction System
HDInsight
Fraud PredictionAzure Machine Learning
Commands flow
Data flow
Auxiliary Services CRMs Data
Transactions Batch Processing System
Machine Learning in Finance. Antifraud: Architecture
Bank Antifraud System h(θ0, θn)
Fraud prediction APIAzure ML Web Services
POST, httpsREST APIJSON
Final Model
2. Pre-processing data 3. Create prediction modelSource: http://0xcode.in/antifraud-insights
1. Retrieve data
Machine Learning in Finance. Antifraud: retrieve data
External Services: geolocation, currency exchange rate, etc.Support Service Data Social Network & News aggregatorPlastic Card, Accounts, Merchants, IP-hosts, etc. black/white lists
Number of customer grows fast… Number of operations grows even faster…
Transactions Logwith request information
Banking CRM DataMerchant CRM DataWeb-clicks StreamWeb/mobile-applications & Backend services Log Data for Model
Join data
Problems
2. Pre-processing data 3. Create model
Machine Learning in Finance. Antifraud: processing data
Integration problems:Heterogeneous systems are often complexDifferent format (RDBS, NoSQL, text logs)Relationship inside data not explicitly
specifiedBig volume, grows fast
But this is not enough:Missing valuesInvalid valuesOutlinersPrivate Data
But and this is not enough:Legal restrictions: local & international (PCI
DSS)Different security policies inside bankFuzzy problem formulation
Integrat
ion
Quality
Policy
1. Retrieve data 2. Pre-processing data 3. Create model
Storage
ResourceManagement
ML Framework
Execution Engine
Local OS
Local Disc
Pyth
on R
untim
e
Yet A
noth
er
Runt
ime
scikitlearn
HDFS
YARN
MapReduce
Mahout
HDFS / S3
YARN / Apache Mesos
Spark
MLlib
HDFS / S3
YARN / Apache Mesos
Python / R on Spark
Python / Rtools
Spark
Local PC Hybrid Model Cluster (on-premises/on-demand)
somelibrar
y
Machine Learning in Finance. Infrastructure for Data Scientist
Low HighCost of deployment/ownership
Distributed FS
Dark Magic…
ML as a Service
Python / Rtools
1. Retrieve data 2. Pre-processing data 3. Create model
Machine Learning in Finance. Antifraud: Create model
1. Retrieve data 2. Pre-processing data 3. Create model
Demo
References
Machine Learning in Finance. References
Start for free from azure.com/ml Read Microsoft Machine Learning BlogExamine Azure ML documentation +free booksTake free MOOCs on MVA & EdXCommunicate on «Microsoft Azure Russia» group Make the world better place with Azure for Researchers Award program
© 2015 Dmitry Petukhov All rights reserved. Microsoft Azure and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
Thank you!
Q&ANow or later (send on email)
Ping meHabr: @codezombieFacebook: @code.zombi
LinkedIn: @dpetukhov
Read my tech code instinct blog (on http://0xCode.in/)
Machine Learning in Finance. Stay connected!
Download presentation from http://0xcode.in/azureml-hackathon-2015 or