Apache Spark in Cloud and Hybrid: Why Security and Governance Become More Important: keynote by Arun...

1 ©Hortonworks Inc.2011– 2017.AllRightsReserved1 ©Hortonworks Inc.2011– 2017.AllRightsReserved

Arun C.MurthyHortonworksCo-Founder@acmurthy


DataCenter CloudModernData

Applications

©Hortonworks Inc.2011– 2016.AllRightsReserved

Exceptions thatDrivePredictiveMaintenance

360ViewofOperations,

Equipment FailureAnalytics, Recalls,

etc.

DeepHistoricalAnalysis

DATA CENTER

Stream Analytics

Mfg LineData

ConnectedCity,Infrastructure, and

Devices

MachineLearning

CLOUD Sensors,Infotainment,ControlSystems

EdgeAnalytics

Time –basedRouteOptimization

ConnectedDataPowersModernDataApplications

5 ©Hortonworks Inc.2011– 2017.AllRightsReserved

PrescientTravelerIsaModernDataApplication“Weknowthatwhenwedefineahigh-threatareainagivenareaoftheworld,thatitisunderpinnedbyveryspecificdatasources.It’sdata-driven,andwecanpointtothosesources—ifeverasked—andsay,‘Here’swhy.’”

MikeBishop,ChiefSystemsArchitect,Prescient


PrescientHarnessesMachineLearningforTravelerSafetyWarnings

S ITUAT ION

Couldonlyproduceoneassessmentevery3-4days

Performsriskmanagement

Useshumanstoidentifyfalsepositives

Neededefficientwaytostorerawdataforanalytics

49,500DataSources

700%ProductivityImprovement

5PetabytesofData

HybridArchitecture

ingestedbyHDFintoHDP

forgeospatialanalysts

storedinHDPconnectedEMC

HDFconnectsdatacentertocloud

ETLOFFLOADSensor Data

Ingest

DATADISCOVERY

ThreatAssessments

SINGLEVIEWGlobal

Threat Map

PREDICT IVEANALYT ICSThreat-Proximity MobileAlerts

ACTIVEARCHIVEStreaming

Threat Archive

DATAENRICHMENTProvenanceMetadata

“Weknowthatwhenwedefineahigh-threatareainagivenareaoftheworld,thatit isunderpinnedbyveryspecific datasources. It’s data-driven,andwecanpointtothosesources—if everasked—andsay,‘Here’s why.’”MikeBishop,ChiefSystems Architect


Symantec:DataScienceSpeedsTimetoCyberSecurityProtection

From4hoursto2-Seconds

5000x

10sofPBS

CloudFlexibility

threatdetectionlatency

improvedtime-to-protection

ofhistoricaldataformachinelearning

tomeetpeakdemandforanalysis

S ITUAT ION

3-4hrprocessinglatenciestoanalyzedigitalthreats

Networkhas+57Mattacksensorsin157countries

Datastreamsfrom75Muserson120Mdevices

Longopenwindowsofexposuretocyberattacks

DATADISCOVERYSecurityLogs

DATADISCOVERY

ThreatDetection

DATADISCOVERYAttacker

Identification

PREDICTIVEANALYTICSUnifiedSecurity

PREDICTIVEANALYTICSProtectiveSafeguard

ETLOFFLOADDevice Data

Ingest

ACTIVEARCHIVEThreatArchive

ETLOFFLOADGreenplumOffload

ETLOFFLOADMetadataCapture

PREDICT IVEANALYT ICS

Threat Predictions

“Onanygivenday,we’llbeprocessing40billionmessagesintooursystem…It usedtobethatqueueswouldbackup.Wewouldseetimes toanalysisontheorderof4hours.Onaverage,we’ve gottenthatdowntotwoortwoandahalfseconds.”DavidLin,SeniorDirectorofEngineering, Symantec CloudPlatform


Progressive:RewardSafeDriversandImproveTrafficSafety

ETLOFFLOADSensor Data

Ingest

DATADISCOVERYWeb LogAnalysis

ACTIVEARCHIVEIndividual

Driving Histories

100%in2-3Days

+12Billion

WebApp-Enabled

$2.6Billion

drivingdetailcapturedfromSnapshot,inHDF

milesdrivenstored

customersseedrivingdetailandimprovesafety

in2014Premiums

ExistingDataSystemsDidNotScaleEfficiently

Usage-Based“Snapshot”InsuranceProgram

In-CarSensorCapturesIoT Data

~7DaystoTransformOnly25%ofUBIData

S ITUAT IONDATA

DISCOVERYOnlineAdPlacement

DATADISCOVERYClaimNotes

Mining

PREDICT IVEANALYT ICSUsage-Based Insurance (UBI)

“We’re lookingatdatasetsthatweneverdreamedwecouldlookat…It’s joiningdotsthatinthepastwedidn’tevenknowwecouldjoin.”Pawan Divakarla, Data&Analytics Business Leader


HDCSharedServices:Secure&GovernallModernDataApplications

CentralizedManagement• DatainMotionoratRest• AcrossDataCenterand

Cloud• Manageend-to-enddata

lifecycle

Cloud

DataCenter

Security

Governance

DataLifecycle

SharedServices


Demo


ThankYou

Apache Spark in Cloud and Hybrid: Why Security and Governance Become More Important: keynote by Arun...

Data & Analytics

Transcript of Apache Spark in Cloud and Hybrid: Why Security and Governance Become More Important: keynote by Arun...