Post on 20-May-2020
Adaptive Presentation onMachine Learning
atDAMA NYC
October 18, 2018James Cerrato
Chief Product Evangelistjames.cerrato@adaptive.com
1Copyright © 2018 Adaptive, Inc. All Rights Reserved.
Jeff Goins
Presidentjeff.goins@adaptive.com
Confidentiality & Disclosure Agreement
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 2
This is an unpublished work, the copyright in which vests in Adaptive, Inc. (“Adaptive”). All rights reserved.
The information contained herein is confidential and the property of Adaptive, Inc. and is supplied without liability for errors or commissions. No part may be reproduced, disclosed or used, except as authorized by contract or other written permission. The copyright and the foregoing restriction on reproduction and use extend to all media in which the information may be embodied.
• All product names used herein are for identification purposes only and may be trademarks of their respective companies.
What is Machine Learning?
• Machine Learning is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention
• The Pattern: Understand, learn, predict, adapt… and repeat
• When systems start acting autonomously they can produce surprising resultso Google is using deep learning algorithms to produce better,
translations between languages. This led to the system creating its own “language” to serve as a canonical representation of the meaning of word. This machine language is used as an intermediate translation between human languages.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 3
Why has Machine Learning become hot?
• Produces analytical outcomes and insights not possible using traditional development
• Significantly reduces labor costs• Leverages the growing volume and velocity of data from
e-transactions, sensors, Internet of Things, social media…
• Create new predictive models from raw data• Ever increasing power of computing and miniaturization
of computing devices is making it more feasible• Availability of skilled resources is increasing
4Copyright © 2018 Adaptive, Inc. All Rights Reserved.
Cont.
• “The biggest reason is because it works… Take the simple problem of recognizing traffic signs in different weather and lighting conditions… The best deep learning models (committees of convolutional neural networks) can now achieve about 99.5% average accuracy, compared to about 99% for the best humans, and a bit lower than 99% for average humans.”
Matthew Lai, Research Engineer @ Google DeepMind
• “Machine Learning provides insights and solutions for business process optimization and operational enhancements. It helps sorting through vast amount of data by analysing and identifying patterns that can help almost every business.”
Nishant Poojary
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 5
What are examples of it being used successfully?
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 6
Natural Language Processing
Recommendation Systems
Algorithmic Trading
Source: Iliya Valchanov
Cont.
• Predict if an employee will stay with your company or leave.
• Decide if a customer is worth your time, if they are likely to buy from a competitor, or not buy at all. You can optimize processes, predict sales, and discover hidden opportunities.
• And many more…
Discover the business value hidden in your data!
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 7Source: Iliya Valchanov
Cont.
It’s all around us…
• Many of our day-to-day activities are powered by machine learning algorithmso Fraud detection, Web search results, Text-based sentiment analysis, Credit
scoring and next-best offers, Prediction of equipment failures, Email spam filtering, license plate readers, facial recognition etc.
• Machine Learning can be applied in more complex applications o Self-parking cars, Guiding robots, Airplane navigation systems (manned
and unmanned), Space exploration, Medicine etc
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 8
Machine Learning vs Traditional Application Development
• Software engineers use their ingenuity to come up with a solution and formulate it as a precise program a computer can executeo Some problems require creative problem solving
• Machine learning systems, collect input data and desired target values. They instruct a computer to build a program or model that computes an optimal output for each input value.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 9
Machine
Learning
Outcomes
Data
Quality
Enterprise
Architecture
Data Governance
OntologiesData
Mapping
Rules
Management
Big Data
Algorithms
Predictive Models
Objective Function
OptimizationAlgorithm
OptimizedModels
Metadata
Machine Learning is a catalyst for many IT domains
How does Machine Learning apply to other IT domains?
Enterprise Architecture o Process Automation o Sales Optimization o Customer Service o Security
Metadata Managemento Data gathering o Modeling & Indexing o Determining dependencies o Automating mapping o Data Lineage o Data retention policies
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 11
Big Data o Data visualization o Data integration o Dashboards and BI o Automated, data-driven decisions
Data Quality o Cleaning and Maintaining data o Data Selection o Feature Extraction o Creating Analytical datasets
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 12
Cont.
Ontologieso Automate mapping between ontologieso Automate mapping from technical data elements to ontologyo Make ontologies executable
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 13
Cont.
Data Governance is still required
Data Governance teams are needed to perform these tasks• Identify sources for training and validation data sets• The Machine Learning models created need to be cataloged and governed
o Track where they are deployedo Record dependencies (e.g. the platform such as TensorFlow, Watson)o Assign stewards, stakeholders, consumers…
• Verify the quality of the data sets (e.g. size/variability)• Perform change management for models• Treat the models as part of the lineage for any output data• Map the data sets to well specified semantic data models• Map the operational data the ML models will be used on to data with the
same meaningo Don’t do training on one set of data and apply it to operational data with a different
meaning!
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 14
What are the critical success factors for a Machine Learning project?
• Find new team members with the right skills• Train existing team members in the new skills• More data, more data, MORE DATA!• Understand what the key features are in the data
o What are the characteristics of good training data?
• The quality of the data, and that it is representativeo Data Quality is still important!
• Stay focused on the business problem to be solved• Find the right parameters of the model• Define the best Objective Function for measuring the
outcomes
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 15
What are the risks to be aware of?
• The design of the algorithms is not as transparent• Algorithms are created “inside the black box”, which can
lead to intentional or unintentional biaseso Why is Facebook putting this in my News Feed?
• If the design is not apparent, monitoring is more difficulto For example, forensic analysis of how a result was achieved
becomes difficult or impossible
• The data collected for Machine Learning algorithms by companies may violate the privacy of users
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 16
Machine Learning Algorithms
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 17
Source: Analytics vidhya
Supervised Learning
• First train the model with lots of training data
• Then we apply the model to new data to form predictions
• This process is called Supervised Learning which is really fast and accurate.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 18
Source: Madhu Sanjeevi
• In Unsupervised Learning the training data does not include Targets, we don’t tell the system where to go , the system has to understand itself from the data we give. The training data is not structured.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 19
Source: Madhu Sanjeevi
Unsupervised Learning
• Reinforcement learning is really powerful and complex to apply for problems.
• It is a type of Machine Learning technique that enablesan agent to learn in an interactive environment by trialand error using feedback from its own actions andexperiences.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 20
Source: Madhu Sanjeevi
Basic explanation of how Machine Learning works
Datao Prepare a certain amount of data to train on. Usually, this is
historical data, which is often readily available.
Modelo The simplest model to train on is a linear model. A linear
model is just the tip of the iceberg, though it lets us create complicated non-linear models. They usually fit the data much better than a simple linear relationship.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 21
Source: Iliya Valchanov
Objective functiono The third ingredient is the objective function. After feeding the
data to the model, we want to obtain an output that is as close to reality as possible. That’s where the objective function comes in.
o The objective function allows you to estimate how accurate the model’s outputs are, on average.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 22
Source: Iliya Valchanov
Optimization algorithmo The final ingredient is the optimization algorithm, or the mechanics
through which we vary the parameters of the model in order to optimize the objective function.
o For each set of parameters, we would calculate the objective function.
o Then choose the model with the highest predictive power — the one with an optimal objective function.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 23
Source: Iliya Valchanov
Tools and Technologies
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 24
Cont.
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 25
Copyright © 2018 Adaptive, Inc. All Rights Reserved. 26