Analyzing BigData with Machine Learning and Hadoop Clusters

Sudhir Rawat, Data Engineer, Microsoft (@rawatsudhir)

Agenda

• Capabilities of HDInsight and AML

• Build solution using HDInsight and AML

• Demos

BREAK APART….

Azure HDInsightManaged 100% Apache Hadoop

99.9%

availabilityAzure SLA

Terabytes to

PetabytesScale-out

Deployed in

minutesWithin a few clicks

Azure HDInsight running Windows/Linux

– Managed & supported by Microsoft

– Re-use common tools, documentation, samples from Hadoop/Linux ecosystem

– Add Hadoop projects that were authored on Linux to HDInsight

– Easier transition from on-premises to cloud

Big Data @ Microsoft - Options

Demo

Azure Data Lake Analytics

AZURE ML

Let’s start with a game…

Business users access results from anywhere, on any device

Delivering Advanced Analytics

• HDInsight

• SQL Server VM

• SQL DB

• Blobs & Tables

Devices Applications Dashboards

Data Microsoft Azure Machine Learning

Storage space

Integrated development environment for Machine

Learning

ML

Studio

Business problem Business valueModeling Deployment

• Desktop files

• Excel spreadsheet

• Other data files on PC

Cloud

Local

Data to model to web services in minutes

http://studio.azurem

l.net

Web

Clients

API

Model is now a web service

Monetize this API

EXAMPLE SOLUTIONS

How can Advanced Analytics help you?

Demo

BRING IT TOGETHER….

Business Scenarios

Recommendations,

customer churn,

forecasting, etc.

Perceptual Intelligence

Face, vision

Speech, text

Personal Digital Assistant

Cortana

Dashboards and

Visualizations

Power BI

Machine Learning

and Analytics

Azure

Machine Learning

Azure

Stream Analytics

DATA

Business apps

Custom apps

Sensors and devices

INTELLIGENCE ACTION

People

AutomatedSystems

Big Data Stores

AzureSQL Data Warehouse

Information

Management

Azure

Data Factory

Azure

Data Catalog

Azure

Event Hub

Azure

Data Lake Store

Azure

HDInsight (Hadoop)

Azure

Data Lake Analytics

ConceptualCortana Analytics Suite - Layer Stack

Background

Solution

Problem

Logistics – IoT use case

TransformationCollection Presentation and action

Event Queuing System

Long-term storage

Fleet Management – Data Flow

Search and query

Data analytics (Excel)

Web/thick client dashboards

Devices to take action

Event hub

Event producers

Applications

Web and social

Devices

Sensors

Live Dashboards

Apache HBase onHDInsight

DocumentDBSolr Azure SearchMongoDB SQL

Cloud gateways

(web APIs)

Field

gateways

Kafka/RabbitMQ/ActiveMQ

Event hubs

Azure ML

Storage

adapters

Stream processing

1

7

Apache Storm

on HDInsight

Demo