MESSAGEBROKER

BY:HALUANMOHAMMADIRSAD

WhatisMB?

Messagebrokerisamiddlewaretoexchangemessageintelecommunicationorcomputernetworks.

WhentouseMB?

1. Decouplingmessagepublisherandconsumer(receiver)2. Storingthemessage3. Routingofmessage4. Monitoringandmanagementofmessages

HowMBworks?

• Queuing• Publish-Subscribe

Queuing

“ Apoolofconsumersmayreadfromaserverandeachrecordgoestooneofthem”(+)Allowtodivideupthedataprocessingovermultipleconsumerinstances,whichletsyouscaleuptheprocess.(-)Notmulti-subscriber—onceoneprocessreadsthedatait'sgone.

Publish-Subscribe

“Broadcastmessagestoallconsumersthatsubscribedtoaspecificchannel”(+)Allowgreaterscalabilityandamoredynamicnetworktopology.(-)Messagespecificationandparticipantrulesaddssomemorecomplexitytothesystem.(-)Hasnowayofscalingprocessingsinceeverymessagegoestoeverysubscriber

MBTools?

• RabbitMQ• ActiveMQ• ApacheKafka

ThroughputBenchmark

ThroughputBenchmark(Batch)

NotesnottoUseMessageBroker(MB)

MeasureSystemCouple• LooseCoupling:don’tuse MB• TightCoupling:use MB

NotesnottoUseMB

ThesystemalreadyimplementObserverDesignPattern(readmore:https://sourcemaking.com/design_patterns/observer)

APACHEKAFKA

Kafkaisawesome(ehemmm..)

• Buildingreal-timestreamingdatapipelinesthatreliablygetdatabetweensystemsorapplications• Buildingreal-timestreamingapplicationsthattransformorreacttothestreamsofdata

Waittt,STREAM????

Streamisthewaydatatransfer aspiece bypiece datarather thanasawhole.

Concepts

• Kafkaisrunasaclusterononeormoreservers.• TheKafkaclusterstores streamsof records incategoriescalled topics.• Eachrecordconsistsofakey,avalue,andatimestamp

CoreAPIs

• The ProducerAPI allowsanapplicationtopublish astreamrecordstooneormoreKafkatopics.• The ConsumerAPI allowsanapplicationtosubscribe tooneormoretopicsandprocessthestreamofrecordsproducedtothem.• The StreamsAPI allowsanapplicationtoactasa streamprocessor,consuming aninputstreamfromoneormoretopicsandproducing anoutputstreamtooneormoreoutputtopics,effectivelytransformingtheinputstreamstooutputstreams.• The ConnectorAPI allowsbuildingandrunningreusableproducers orconsumers thatconnect Kafkatopicstoexistingapplicationsordatasystems.Forexample,aconnectortoarelationaldatabasemightcaptureeverychangetoatable.

KafkaTopic

Atopic isacategory orfeednametowhichrecords arepublished.Topics inKafkaarealwaysmulti-subscriber;thatis,atopiccanhavezero,one,ormanyconsumersthatsubscribetothedatawrittentoit.

KafkaPartition

Foreachtopic,theKafkaclustermaintains apartitioned log.

KafkaLogs

Eachpartition isanordered,immutable sequenceofrecordsthatiscontinuallyappendedto—astructuredcommitlog.

Distribution

• Eachpartitionhasoneserverwhichactsasthe"leader"andzeroormoreserverswhichactas"followers".Theleaderhandlesallreadandwriterequestsforthepartitionwhilethefollowers passivelyreplicate theleader• UseZooKeeper (morenextpart)

Producers&Consumers

Guarantees

• Messagessentbyaproducertoaparticulartopicpartitionwillbeappendedintheordertheyaresent.Thatis,ifarecordM1issentbythesameproducerasarecordM2,andM1issentfirst,thenM1willhavealoweroffsetthanM2andappearearlierinthelog(FIFO).• Aconsumerinstanceseesrecordsintheordertheyarestoredinthelog.• ForatopicwithreplicationfactorN,KafkawilltolerateuptoN-1serverfailures withoutlosing anyrecordscommittedtothelog.

KafkaasaStorageSystem

• Datawritten toKafkaiswrittentodisk andreplicated forfault-tolerance.• Kafkaallowsproducerstowaitonacknowledgementsothatawriteisn'tconsideredcompleteuntilitisfullyreplicatedandguaranteed topersisteveniftheserverwrittentofails.

KafkaforStreamProcessing

• Handlingout-of-orderdata,reprocessing inputascodechanges,performingstateful computationsinrealtime.(Readmorehttps://kafka.apache.org/documentation.html#streams)

ZOOKEEPER

ZooKeeper

Atoolstohelpcontrolandmonitoringdistributedsystems.

Howitworks?

• ZooKeeper ManagedKafkaclusterconfiguration.• ZooKeeper canelectKafkanodetobecomethemaster.• ProducersandConsumersarenotifiedbyZooKeeper serviceaboutthepresenceofnew brokerinKafkasystemorfailure ofthebrokerintheKafkasystem.• ZooKeeper SynchronizedKafkaacrossthecluster.

SingleNodeforMultipleBrokerCluster

MultipleNodeforMultipleBrokerCluster

Notes

• Kafkatopicreplicationhandled byKafka

PuttingthePiecesTogether(DO)

• WriteaScalaserviceastheproducer• Writearubyserviceastheconsumer• Exchangedatainrealtime