BENDINGSPARKTOWARDSENTERPRIZE NEEDS

BORISTROFIMOV@SIGMASOFTWARE

USERINTERFACE

SERVICELAYER

DATASTORE

CLASSIC APPROACH

PRESENTATIONLAYER

APPLICATIONSERVICES

DOMAINLAYER

DAL

INFRASTRUCTURE

CLASSIC LAYERS

CLASSIC APPROACH

USERINTERFACE

SERVICELAYER

DATASTORE

CQRS

USERINTERFACE

SERVICELAYER

DATASTORE

COMMAND/WRITEMODEL READMODEL

CQRS NAKEDUSERINTERFACE

SERVICELAYERWRITESTORE

COMMANDBUS

COMMANDHANDLER

REPOSITORY

DomainModel Domain

Model

EVENTBUS

EVENTHANDLER

Commands

READMODELSTORE

QUERYFACADE

Query DTO

WRITEMODEL READMODEL

USERINTERFACE

MULTIPLE READMODELSUSERINTERFACE

SERVICELAYERWRITESTORE

COMMANDBUS

COMMANDHANDLER

REPOSITORY

DomainModel Domain

Model

EVENTBUS

EVENTHANDLER

Commands

EVENTHANDLER

READMODELSTORE1

READMODELSTORE2

QUERYFACADE QUERYFACADE

WRITEMODEL READMODEL

BULLETIN BOARDAPP

APPLICATIONFEATURES

•ADDBulletinwithspecificauthornameandmessage•VIEWlistofpublishedbulletins

SERVICELAYER

SPARKISCOMINGUSERINTERFACE

MONGOREAD/WRITESTORE

KAFKA

COMMANDHANDLER

REPOSITORY

DomainModel Domain

Model

Commands

QUERYFACADE

Query DTO

WRITEMODEL READMODELSPARK

SERVICELAYER

SPARKISCOMINGUSERINTERFACE

MONGOREAD/WRITESTORE

KAFKA

COMMANDHANDLER

REPOSITORY

DomainModel Domain

Model

Commands

QUERYFACADE

Query DTO

WRITEMODEL READMODELSPARK

SHAREDSTORE

TECHNOLOGY STACK

FAÇADE• Java8• SpringBoot&MVC• Mongo

COMMANDPROCESSOR• Kafka0.9.0.1• Scala2.11• Spark2.0.0Streaming

DEPLOYMENT VIEW

DOCKER

KAFKAZOOKEEPER

MONGO

DEVMACHINE

LOCALSPARK

FACADE

DEMO

https://github.com/btrofimov/spark-enterprise-example/tree/master/nonblocking-bulletinboard

WAIT

WELCOMETOASYNCHONOUSHELL

ATLEASTFOURSOLUTIONS

• C'mon,it’sOK

• Pullingserviceuntilentityisadded

• Addpushchannelanddelivereventsupstairs

• MakeRESTmethodsblockeduntilcommandisfinished

MAKEITSYNCHRONOUS

DEMO2

https://github.com/btrofimov/spark-enterprise-example/tree/master/blocking-bulletinboard

RETROSPECTIVE

READMODELinstances

WRITEMODELinstances

EFFICIENT SCALEOUTSEPARATEDSCALEOUTDEPENDINGONMODEL

NEEDS

dataStream.foreachRDD { rdd =>

val values = rdd.values

values.map { cmdMessage =>…

}.foreach { cmdMessage =>

…}

}

DESERIALIZATION ERRORS

MIGHTFAILDURINGLAUNCHIFPIPELINEHASBEENCHANGED

RISK: AFFECTSCHECKPOINTS

DESERIALIZATION ERRORS

• Hardtokeepcodetoleranttotheseerrorsandtestableatthesametime.• SparkwithKafkaconnector0.10allowstosaveKafkaoffsetsanditispossibletorestorefromtheminsteadofcheckpoints.

THROUGHPUT VSLATENCY

Spark-basedserviceshavenicethroughputhoweverduetointernaloverheadstheyhavelowerlimitonlatency

IDENTIFIER ASSIGNRESPONSIBILITY

• Clientgeneratesuniqueidentifiersandpassesittoservice• Servicegeneratesuniqueidentifierandpassesitbacktoclient

EVENTUALVS STRONG CONSISTENCY

• CQRSappsusuallyareeventuallyconsistent• ChoosewiselybetweenCQRSandtraditionalapproach• Consideroptimisticorpessimisticlockingtoachievestrongconsistency• Insomecasesdefinedreasonabletimeoutsorin-progresslistsmighthelptosolveraceconditions(dependsoncommandroundtrips)

FAULTTOLERANCE

FAULTTOLERANCE

• Sparkautorerunstaskstomitigatenetworkorotheroutages• Easyandsafeapplicationrestartbasedoncheckpoints

THANKYOU

slides code