Hortonworks Data In Motion Series Part 4

Presentation Title Goes Here with a Maximum of Three Lines of Copy

Harnessing Data-in-Motion with Hortonworks DataFlow

A Paradigm Shift to Business as Usual: Real World Use Cases of Real-Time DataFlows in Record TimeAnna YongProduct MarketingHaimo LiuProduct Manager

# Hortonworks Inc. 2011 2016. All Rights Reserved

Hortonworks: Powering the Future of Data1

AgendaOverview of Hortonworks DataFlow (HDF)How HDF transforms data movement months to minutesHDF Use CasesReal-World HDF Use CasesApache NiFi Case Studies from Hadoop Summit Other Apache NiFi/HDF Use Cases


# Hortonworks Inc. 2011 2016. All Rights ReservedConnected Data Platforms

# Hortonworks Inc. 2011 2016. All Rights ReservedHortonworks DataFlow is part of a complete connected system of data in motion and data at rest, on prem on in the cloud. Both platforms support a set of open source Apache projects but with a different focus.


4 Hortonworks Inc. 2011 2016. All Rights ReservedPayment TrackingDueDiligenceSocialMappingProductDesignM & ACallAnalysisMachineDataDefectDetectingFactoryYieldsCustomerSupportBasketAnalysisSegmentsCustomerRetentionSentimentAnalysisOptimizeInventoriesSupplyChainCross-SellVendorScorecardsAdPlacementCyberSecurityDisasterMitigationInvestmentPlanningAdPlacementRiskModelingProactiveRepairInventoryPredictionsNextProduct RecsOPEXReductionHistoricalRecordsMainframeOffloadsDevice DataIngestRapid ReportingDigitalProtectionDataas aServiceFraudPreventionPublicDataCaptureINNOVATERENOVATE

EXPLORE

OPTIMIZE

TRANSFORM

ACTIVEARCHIVE

ETLONBOARD

DATAENRICHMENT

DATADISCOVERY

SINGLEVIEW

PREDICTIVEANALYTICS


4Hortonworks: Powering the Future of Data

ConstrainedHigh-latencyLocalized contextHybrid cloud / on-premisesLow-latencyGlobal context

CoreInfrastructure

Hortonworks DataFlow Manages Data in Motion

RegionalInfrastructure

Sources



Hortonworks DataFlow Manages Data in Motion

CoreInfrastructureSources

ConstrainedHigh-latencyLocalized contextHybrid cloud / on-premisesLow-latencyGlobal context




451 Analyst Report on Stream Processing and Streaming Integrationhttp://hortonworks.com/info/value-streaming-integration/


7

Dataflow Management




Problems Today: Timely Access to Data and Decisions

http://diginomica.com/2016/04/22/royal-mail-starts-to-deliver-on-hortonworks-data-in-motion-promise

HDF helps us to streamline the flowof data and build models andvisualisations quickly, so that my teamcan work iteratively with business colleagues on building solutionsthat work for the business.Royal Mail



HDPHORTONWORKSDATA PLATFORMPowered by Apache HadoopHDF Makes Big Data Ingest Easy

Complicated, messy, and takes weeks to months to move the right data into Hadoop

HDPHORTONWORKSDATA PLATFORMStreamlined, Efficient, Easy

HDPHORTONWORKSDATA PLATFORMPowered by Apache Hadoop



Hortonworks DataFlow, Powered by Apache NiFi. Demo Time



Create a live dataflow in minutesHow would that change your business?


Add processor for data intake. Time: 1 minute

1Drag and drop processor from top menu


13

Choose the specific processor2Choose one of the processors currently 170+ available

# Hortonworks Inc. 2011 2016. All Rights ReservedExample: Pick Twitter Processor


15

Configure the processor. Time: 2 minutes3

4Select processor and choose option to ConfigureAdjust parameters as required


Another processor for data output. Time: 1 minute

5

6Filter for and select a Put processorDrag and drop processor from top menu

# Hortonworks Inc. 2011 2016. All Rights ReservedConfigure second processor. Time: 1 minute7Configure 2nd processor


Connect processors, configure connection. 2 minutes

Configure Connection8

Note: Sample Flow is different from previous example of PutHDFS. This dataflow is PutFile. Same concepts apply.


Click Start to Begin Processing. Time total: 7 minutes

9Click start play to being processing (will run continuously until you select stop)

# Hortonworks Inc. 2011 2016. All Rights ReservedHDF Use Cases


CoreInfrastructure

Hortonworks DataFlow Use Cases


Sources

Dataflow ManagementOn-ramp into HadoopLog Collection / Splunk OptimizationCyber SecurityIoT IngestionDeliver data into stream processing engines Real-time Event Processing (Kafka, Storm)Move data between from on-prem and cloud environments



Optimize Log Analytics with Content Based Routing


23

IoT Data IngestionConstrainedHigh-LatencyLocalized ContextHybrid Cloud/On-PremiseLow-LatencyGlobal ContextResolves real world connectivity and transmission issues often overlooked by assuming connectivity is always perfect


24

Enterprise Data Movement and Hybrid CloudSeamlessly fuse dataflows between data centers Data center to data center, Remote location to data center, Data center to cloud

HDF

Between Data Centers

HDF

HDF

Remote to Data CenterHDF

HDF

HDF

HDF

Between Data Centers & Cloud

HDF


25

Stream ProcessingPage 26

Data AcquisitionEdge Processing

Real Time Stream AnalyticsRapid Application Development

IoT ANALYTICS CLOUD

Hortonworks Inc. 2011

# Hortonworks Inc. 2011 2016. All Rights ReservedReal World Use Cases



Royal Mails Journey

Satisfied customers with better service levels & improved retentionActionable intelligence improves customer experienceData management transformed to deliver specific, actionable insights to line of business departments Analysis delivered within days & weeks rather than months per project deadlinesChurn modelling project identified customers at risk by vertical sector in order to take preventative actionImproved accuracy of delivery times for business customers & highlighted trends related to volumes of mail expectedGovernance & compliance simplified due to central data platform

Satisfied customers with better service levels & improved retention

PREDICTIVEANALYTICS

ACTIVEARCHIVE

SINGLEVIEW

ACTIVEARCHIVE

SINGLEVIEW

ETLONBOARD

DATAENRICHMENT

PREDICTIVEANALYTICS

DATAENRICHMENTParcel distributionCustomer Acquisition

SINGLEVIEWCustomer SupportInventory PredictionsInvestment PlanningData-as-a-ServicePublic data captureRapid reportingEDW offloadOPEX reductionInnovateRenovate

New Data Products



Open Energi Uses HDF for Electricity Demand ResponseAs a result of its investment in Hortonworks Dataflow, Open Energi is Already:Reducing costs thanks to 10-15% less data being transmitted across a mobile networkCreating a full transparent trail for data provenance that Open Energi can sharewith customersEnabling line of business teams to contribute to building dataflow rules and processesStandardizing the output of data across various end point devicesOpen Energi: hortonworks.com/blog/data-fuel-open-energi-virtual-power-station-hortonworks-dataflow/



Prescient Traveler transformed the travel risk management market

$0.5MMSavings in development costs due to Hortonworks HDF700%Improvement in analyst productivity in determining actual threats49,000Number of data sources currently being analyzed to identify threats

# Hortonworks Inc. 2011 2016. All Rights ReservedCentricas Journey

1.3 Million Smart MetersEDW OffloadMobile App for Customer SitesIngest 300 GB per DayProduct Cross-SellBuilding a Data-Driven Energy Utility Business Self-service analytics for 3 million customers in UK & North America HDP and HDF simplify IT estateIngest of 300 GB/day rationalizes maintenance workPersonalized customer communications replaced impersonal up-sell messagesLegacy EDWs decommisioned

InnovateRenovate

Smart, Efficient Homes

DATADISCOVERY

DATAENRICHMENT

PREDICTIVEANALYTICS

SINGLEVIEW

ACTIVEARCHIVE

ETLONBOARD

SINGLEVIEW

SINGLEVIEW

PREDICTIVEANALYTICSOn-site customer data captureOptimized engineering scheduleTailored servicingCustomer sentiment


31

Apache NiFi Case Studies: Hadoop Summit San Jose

# Hortonworks Inc. 2011 2016. All Rights ReservedFrom Zero to DataFlow: http://www.slideshare.net/HadoopSummit/from-zero-to-data-flow-in-hours-with-apache-nifi-64032731


33

Make Streaming Analytics Work For Youhttp://www.slideshare.net/HadoopSummit/make-streaming-analytics-work-for-you-the-devil-is-in-the-details

34

# Hortonworks Inc. 2011 2016. All Rights ReservedHadoop Summit Keynote: Apache MetronIngest log data into their cyber security data lakehttps://youtu.be/Nffx8SKn7l4?t=1h37m50s

# Hortonworks Inc. 2011 2016. All Rights ReservedHortonworks: Powering the Future of Data35

Hadoop Summit Keynote: Improving Customer Experience

https://youtu.be/BY_0HB9uyXQ

# Hortonworks Inc. 2011 2016. All Rights ReservedOther HDF/Apache NiFi Use Cases

# Hortonworks Inc. 2011 2016. All Rights ReservedData Hacks & Demos Keynote: Retail Simulation at Hadoop SummitLive Voting, Electronic Conversation, Real-Time Facial Recognition

Intro, Demo 1, Demo 2, Demo 3, Demo 4.https://www.youtube.com/watch?v=BY_0HB9uyXQ&feature=youtu.be&t=49m10s

# Hortonworks Inc. 2011 2016. All Rights ReservedGENIVI Alliance: Open Source, In-Vehicle Infotainment Software

The GENIVI Alliance is a nonprofit industry alliance committed to driving the broad adoption of specified, open source, In-Vehicle Infotainment software.

# Hortonworks Inc. 2011 2016. All Rights ReservedMore Use CasesUsing Apache NiFi to read childrens books

https://twitter.com/KayLerch/status/721455415456882689

# Hortonworks Inc. 2011 2016. All Rights ReservedYet More Use Caseshttps://twitter.com/4everfusiongal/status/735158522539855872

https://www.linkedin.com/pulse/making-rain-apache-nifi-jeremy-dyer?trk=prof-post

www.linkedin.com/hp/update/6138493082129149952

https://community.hortonworks.com/articles/30636/how-to-simulate-a-sales-executive-with-hdf.html

# Hortonworks Inc. 2011 2016. All Rights ReservedEven More Use Caseshttps://community.hortonworks.com/articles/47854/accessing-facebook-page-data-from-apache-nifi.html

https://community.hortonworks.com/content/kbentry/32605/running-nifi-on-raspberry-pi-best-practices.html

Accessing Facebook Data

# Hortonworks Inc. 2011 2016. All Rights ReservedAnd Even More Use Caseshttps://community.hortonworks.com/articles/20318/visualize-patients-complaints-to-their-doctors-usi.html

Visualize patients' complaints to their doctors using NiFi and Solr/Banana

http://hortonworks.com/blog/qualcomm-hortonworks-showcase-connected-car-platform-tu-automotive-detroit/

Connected Car


Questions?

Hortonworks Community Connection:Data Ingestion and Streaminghttps://community.hortonworks.com/Contact Us:http://hortonworks.com/contact-us/



Hortonworks Data In Motion Series Part 4

Technology

Transcript of Hortonworks Data In Motion Series Part 4