Dataflow with Apache NiFi - Crash Course - HS16SJ

DataflowwithApacheNiFiAldrinPiri- @aldrinpiriApacheNiFi CrashCourseHadoop Summit2016– SanJose

29June2016

2 ©HortonworksInc.2011–2016.AllRightsReserved

Key:'ApacheNiFi’Value:'PMCMember'

Key:'Work’Value:’Sr.MemberofTechnicalStaff@Hortonworks'

Key:'WorkingwithNiFi Since’Value:'2010’


AgendaWhatisdataflowandwhatarethechallenges?

ApacheNiFi

Architecture

LiveDemo

Community



ApacheNiFi

Architecture

LiveDemo

Community


Let’sConnectAtoBProducersA.K.AThings

AnythingAND

Everything

Internet!

Consumers• User• Storage• System• …MoreThings


Movingdataeffectivelyishard

Standards: http://xkcd.com/927/


Whyismovingdataeffectivelyhard?

Ã Standards

Ã Formats

Ã “ExactlyOnce”Delivery

Ã Protocols

Ã VeracityofInformation

Ã ValidityofInformation

Ã EnsuringSecurity

Ã OvercomingSecurity

Ã Compliance

Ã Schemas

Ã ConsumersChange

Ã CredentialManagement

Ã “That [person|team|group]”

Ã Network



Let’sConnectLotsofAstoBs toAstoCstoBs toΔs toCstoϕsLet’sconsidertheneedsofacourierservice

PhysicalStore

GatewayServer

MobileDevices

Registers

ServerCluster

DistributionCenter CoreDataCenteratHQ

ServerCluster

OnDeliveryRoutes

Trucks Deliverers

DeliveryTruck: CreativeStall,https://thenounproject.com/creativestall/

Deliverer:RigoPeter,https://thenounproject.com/rigo/

CashRegister:SergeyPatutin,https://thenounproject.com/bdesign.by/

HandScanner:EricPearson,https://thenounproject.com/epearson001/


Great!Iamcollectingallthisdata!Let’suseit!Findingourneedlesinthehaystack

PhysicalStore

GatewayServer

MobileDevices

Registers

ServerCluster

DistributionCenter

Kafka

CoreDataCenteratHQ

ServerCluster

Others

Storm/Spark/Flink /Apex

Kafka


OnDeliveryRoutes

Trucks Deliverers






Whyismovingdataeffectivelyhardwhenscopedinternally?

Ã Standards

Ã Formats


Ã Protocols



Ã EnsuringSecurity


Ã Compliance

Ã Schemas

Ã ConsumersChange



Ã Network



Let’sConnectLotsofAstoBs toAstoCstoBs toΔs toCstoϕsOh,thatcourierserviceisglobal


Whyismovingdataeffectivelyhardwhenscopedglobally?

Ã Standards

Ã Formats


Ã Protocols



Ã EnsuringSecurity


Ã Compliance

Ã Schemas

Ã ConsumersChange



Ã Network



TheUnassumingLine:ACaseStudyWe’veseenafewlinesshowupinthewildthusfar

Internet! Inter- &Intra- connectionsinourglobalcourierenterprise

Spotlight:ArthurLacôte,https://thenounproject.com/turo/


DataflowLineAnatomy101Let’sdissectwhatthislinetypicallyrepresents

Fig1.Lineus Worldwidewebus.CommonName:Internet!

ScriptorApplication

ScriptorApplication

Data Data

DisparateTransportMechanisms


DataflowLineAnatomy201Sometimesthattransportisjustmorelines

Fig1.Lineus Worldwidewebus.CommonName:Internet!

ScriptorApplication

ScriptorApplication

LineInception

Data Data


DataflowLineAnatomy301Butthoselinescouldalsohavecomponents…

Fig1.Lineus Worldwidewebus.CommonName:Internet! Fig2.Good RecursionJoke

NoSuchJokeException

footagenotfound



ApacheNiFi

Architecture

LiveDemo

Community


ApacheNiFiKeyFeatures

• Guaranteeddelivery

• Databuffering- Backpressure

- Pressurerelease

• Prioritizedqueuing

• FlowspecificQoS- Latencyvs.throughput

- Losstolerance

• Dataprovenance

• Supportspushandpullmodels

• Recovery/recordingarollinglogoffine-grainedhistory

• Visualcommandandcontrol

• Flowtemplates

• Pluggable/multi-rolesecurity

• Designedforextension

• Clustering


ApacheNiFi Subproject:MiNiFi

Ã LetmegetthekeypartsofNiFi closetowheredatabeginsandprovidebidrectionalcommunication

Ã NiFi livesinthedatacenter.Giveitanenterpriseserveroraclusterofthem.

Ã MiNiFi livesasclosetowheredataisbornandisaguestonthatdeviceorsystem


Let’srevisitourcourierservicefromtheperspectiveofNiFi

PhysicalStore

GatewayServer

MobileDevices

Registers

ServerCluster

DistributionCenter

Kafka

CoreDataCenteratHQ

ServerCluster

Others


Kafka


OnDeliveryRoutes

Trucks Deliverers





ClientLibraries

ClientLibraries

MiNiFi

MiNiFiNiFi NiFi NiFi NiFi NiFi NiFi

ClientLibraries


ApacheNiFi ManagedDataflowSOURCES REGIONAL

INFRASTRUCTURECORE

INFRASTRUCTURE


NiFi isbasedonFlowBasedProgramming(FBP)

FBPTerm NiFi Term Description

InformationPacket

FlowFile Each objectmovingthroughthesystem.

Black Box FlowFileProcessor

Performsthework, doingsomecombinationofdatarouting,transformation,ormediationbetweensystems.

BoundedBuffer

Connection Thelinkage betweenprocessors,actingasqueuesandallowingvariousprocessestointeractatdifferingrates.

Scheduler FlowController

Maintainstheknowledgeofhowprocessesareconnected, andmanagesthethreadsandallocationsthereofwhichallprocessesuse.

Subnet ProcessGroup

Asetofprocessesandtheirconnections,whichcanreceiveandsenddataviaports.Aprocess groupallowscreationofentirelynewcomponentsimplybycompositionofits components.


FlowFiles &DataAgnosticism

Ã NiFi isdataagnostic!

Ã But,NiFi wasdesignedunderstandingthatusers

cancareaboutspecificsandprovidestooling

tointeractwithspecificformats,protocols,etc.

ISO8601 - http://xkcd.com/1179/

Robustnessprinciple

Beconservativeinwhatyoudo,beliberalinwhatyouacceptfromothers“


FlowFiles arelikeHTTPdataHTTPData FlowFile

HTTP/1.1200OK

Date:Sun,10Oct201023:26:07GMT

Server:Apache/2.2.8(CentOS)OpenSSL/0.9.8g

Last-Modified:Sun,26Sep201022:04:35GMT

ETag:"45b6-834-49130cc1182c0"

Accept-Ranges:bytes

Content-Length:13

Connection: close

Content-Type: text/html

Helloworld!

StandardFlowFile AttributesKey:'entryDate’ Value:'FriJun1717:15:04EDT2016'Key:'lineageStartDate’Value:'FriJun1717:15:04EDT2016'Key:'fileSize’ Value:'23609'FlowFile AttributeMapContentKey:'filename’ Value:'15650246997242'Key:'path’ Value:'./’

BinaryContent*

Header

Content



ApacheNiFi

Architecture

LiveDemo

Community


Extension/IntegrationPoints

NiFi Term Description

Flow FileProcessor

Push/Pull behavior.CustomUI

ReportingTask

Used topushdatafromNiFi tosomeexternalservice(metrics,provenance,etc..)

ControllerService

Usedtoenablereusablecomponents/ sharedservicesthroughouttheflow

RESTAPI Allowsclientstoconnecttopullinformation,changebehavior,etc..


OS/Host

JVM

FlowController

WebServer

Processor1 ExtensionN

FlowFileRepository

ContentRepository

ProvenanceRepository

LocalStorage

OS/Host

JVM

FlowController

WebServer


FlowFileRepository

ContentRepository


LocalStorage

Architecture* OS/Host

JVM

NiFiClusterManger– RequestReplicator

WebServer

MasterNiFiClusterManager(NCM)

OS/Host

JVM

FlowController

WebServer


FlowFileRepository

ContentRepository


LocalStorage

SlavesNiFiNodes


NiFiArchitecture– Repositories- Passbyreference

FlowFile Content Provenance

F1à C1 C1 P1à F1

Excerptofdemoflow… What’shappeninginsidetherepositories…

BEFORE

AFTER

F2à C1 C1 P3à F2 – Clone(F1)

F1à C1 P2à F1 – Route

P1à F1 – Create


NiFiArchitecture– Repositories– CopyonWrite

FlowFile Content Provenance

F1à C1 C1 P1à F1- CREATE

Excerptofdemoflow… What’shappeninginsidetherepositories…

BEFORE

AFTER

F1à C1F1.1à C2 C2(encrypted)

C1(plaintext)

P2à F1.1 - MODIFY

P1à F1- CREATE



ApacheNiFi

Architecture

Demo

Community


Learn,ShareatBirdsofaFeatherStreaming,DataFlow&Cybersecurity

ThursdayJune306:30pm,BallroomC


WhyNiFi?

Ã Movingdataismultifacetedinitschallengesandthesearepresentindifferentcontextsatvaryingscopes– Thinkofourcourierexampleandorganizationslikeit:intervs intra,domestically,internationally

Ã Providecommontoolingandextensionsthatarecommonlyneededbutbeflexibleforextension– LeverageexistinglibrariesandexpansiveJavaecosystemforfunctionality– Alloworganizationstointegratewiththeirexistinginfrastructure

Ã Empowerfolksmanagingyourinfrastructuretomakechangesandreasonaboutissuesthatareoccurring– DataProvenancetoshowcontextanddata’sjourney– UserInterface/Experienceakeycomponent


Learnmoreandjoinus!

Apache NiFi sitehttp://nifi.apache.org

Subproject MiNiFi sitehttp://nifi.apache.org/minifi/

Subscribe to and collaborate [email protected]@nifi.apache.org

Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFI

Follow us on Twitter@apachenifi


OurLabforToday

Ã WewillbeexploringsomeexamplestoworkthroughcreatingadataflowwithApacheNiFi

Ã UseCase:Anurbanplanningboardisevaluatingtheneedforanewhighway,dependentoncurrenttrafficpatterns,particularlyasotherroadworkinitiativesareunderway.Integratinglivedataposesaproblembecausetrafficanalysishastraditionallybeendoneusinghistorical,aggregatedtrafficcounts.Toimprovetrafficanalysis,thecityplannerwantstoleveragereal-timedatatogetadeeperunderstandingoftrafficpatterns.NiFi wasselectedforforthisreal-timedataintegration.

Ã Labsareavailableathttp://tinyurl.com/nificrashcourse


ThankYou

Dataflow with Apache NiFi - Crash Course - HS16SJ

Technology

Transcript of Dataflow with Apache NiFi - Crash Course - HS16SJ