From Big Data to AI

FromBigDatatoAIBuildingMachineLearningApplications

MaloyManna

Abstract

• ThenewestbuzzwordafterBigDataisAI.FromGooglesearchtoFacebookmessengerbots,AIisalsoeverywhere.

• Machinelearninghasgonemainstream.OrganizationsaretryingtobuildcompetitiveadvantagewithAIandBigData.

• But,whatdoesittaketobuildMachineLearningapplications?BeyondtheunicorndatascientistsandPhDs,howdoyoubuildonyourbigdataarchitectureandapplyMachineLearningtowhatyoudo?

• Thistalkwilldiscusstechnicaloptionstoimplementmachinelearningonbigdataarchitecturesandhowtomoveforward.

Yourpresenter

MaloyMannaDataengineeringPM,AXADataInnovationLab

Ibuilddataproducts&services.- Solutionarchitecture|Design|Lean-Agile

linkedin.com/in/maloymaloymanna.github.io

Aboutyou

Howmanyofyouare:

• Business?• Datascientists?• Dataanalysts?• DevOpsorIT?• Dataengineering?

Whatwe’lltalkabouttoday

• EvolutionofDataAnalytics… andproductstrategies• BigDataarchitectures• MachineLearning• MLUseCases• DeepLearning• APIs&Chatbots• Challenges&Takeaways

Search

MessengerBots

GoogleAssistant

TeslaAutopilot

Homeautomationplatform

Don’tlookatAIinisolation

DataScience

Datasciencetobuilddataproducts

• Whatistheproblemwearetryingtosolve?• Howdoweknowwhenwe’vewon?(keysuccessmetrics)• Isituseful?

• Whatdatadowehave?• Whatdatashouldwehave?• WhichassumptionscanIverify?• Askquestions

• https://www.datasciencecentral.com/profiles/blogs/the-data-science-project-lifecycle

Wherewillyouintegratethisprediction?

• Strongbusinessobjectivesandgoals• Specificproblems• Targetedbenefits

Datalakearchitecturecomponents

• Platform• Dataacquisition• Dataprocessing• Dataaccess• Security

Influences:• Objectives:Agile/Fail-fastandopen-source• ITstrategy- infra• Interplayoffactors

Dataplatform

Dataacquisition Dataprocessing Dataaccess

Security&Governance

Streamingusecases

• Clickstreamanalysis• Processpoint-of-saletransactions• Mobilegaming• IoT data

MachineLearning

Howmanyofyouhave:

• Usedanopen-sourceML?• Built/plantobuildyourownML?• UsedMLinthecloud?• Use/plantouseMLon-prem?

MachineLearning

MachineLearning

• Atitscore,asetofalgorithms• Canlearnfromandmakepredictionsbasedonrecordeddata• MLmodelisasetofexplanationsoftherelationshipsbetweeninputfeaturesandoutputpredictions• deployedwhereexplicitprogramingistoorigidorisimpractical

https://biguru.wordpress.com/2015/02/05/a-gentle-introduction-to-machine-learning/

MachineLearning

• Atitscore,asetofalgorithms• Canlearnfromandmakepredictionsbasedonrecordeddata• MLmodelisasetofexplanationsoftherelationshipsbetweeninputfeaturesandoutputpredictions• Predictsfutureoutcomesbasedonpasttrendsandtransaction• Deployedwhereexplicitprogramingisimpactical

https://biguru.wordpress.com/2015/02/05/a-gentle-introduction-to-machine-learning/

MachineLearningUseCases• Healthcare– Virtualassistants,wearables,lifestylemanagementandmonitoring,ER&surgery,drugdiscovery,mentalhealth,patientdataandriskanalytics,hospitalmanagement

• Fintech– creditscoring/directlending,regulatorycompliance,frauddetection,personalfinanceassistants,robo-advisors,insurance

• Security– cybersecurity,datasecurity

• Marketing&Advertising

• Facialrecognition– Image&Videoclassification/Segmentation• Speechrecognition• NaturalLanguageUnderstanding– NLP• Personalization- Recommendationengines

DeepLearning

• SubfieldofMLthatusesseverallayersofalgorithmscalledneuralnetworks(algorithmsthatmimicthehumanbrain)• Algorithmsnotlimitedtocreateanexplainablesetofrelationships• ConvolutionalNNsaregoodforimagerecognition• Longshort-termmemorynetworksaregoodforspeechrecognition• Steeplearningcurvebecauseoflow-levelAPIs

DeepLearning

Neuralnetworks

Neuralnetworkarchitectures

https://medium.com/@xenonstack/overview-of-artificial-neural-networks-and-its-applications-2525c1addff7

Usingdeeplearning

Whatwouldinterestyoumore:

• In-depthtutorialsondeeplearning• HowtouseMLframeworks• HowtouseMLAPIsinpractice

APIs

• Vision• Speech• Language

MLframeworks

• MLLib – ApacheSparkMLlibrary• Scikit-Learn(Pythonmathandscience)• Tensorflow - dataflowgraphs,neuralnetworks• ApacheMXNet• Caffe – deeplearning• H20.ai– extendtoHadoopandSpark(Sparklingwater)

• MicrosoftCognitiveToolkit(vision,speech,language,searchapis)• Amazonservices• Googleservices

Spark&Deeplearningpipelines

https://databricks.com/blog/2017/06/06/databricks-vision-simplify-large-scale-deep-learning.html

Chatbots

• Botframeworks– MicrosoftBotFramework,Api.ai,Facebookbotengine• Botplatforms– Chatfuel,BeepBoop,Botengine.ai

AIinproductstrategy

AIacrossindustries

Takeaways

• ProductstrategywithAIinyourproduct• Specificbusinessgoalsandsuccessmetrics• Agilemodelnotalwayssuitablefordata/AIproducts• Peopleandprocessasimportantastechnology• Investinuser-centricdesign

From Big Data to AI

Technology

Transcript of From Big Data to AI