Message System Fundamentals

By John Staveley27/07/2017

https://uk.linkedin.com/in/johnstaveley/@johnstaveley

Who am I?

John StaveleyContract Mvc.net developerOrganiser @LeedsSharpUser of messaging systems for the last 4 years (MSMQ,

Rabbit, Azure Service Bus etc)

Overview

Why use messaging systems?What are they? Benefits?Demos (Azure Storage Queues, Azure Service Bus,

RabbitMQ)Azure FunctionsInfrastructure as code.Net Core and DockerOther Messaging implementationshttps://github.com/johnstaveley/MessagingPresentationSlides on SlideShare

Use cases and benefits

Distributed workloads Increased reliability (cf fallacies of distributed networking) Synchronous vs Asynchronous processes Loosely coupled systems (Microservices) The promise of fast reads and fast writes LinkedIn, CompareTheMarket.com, Facebook

Demo: Azure Storage Queues

What is Azure? What is Serverless / FaaS - Azure functions? What are Azure Queues? Message patterns - One way/competing consumers Demo

What is Azure?

Second largest cloud provider in the world IaaS (Windows or Linux VMs), PaaS, FaaS Data storage, Notifications, Load Balancing, Active

Directory, Application insights SQL Azure, DocumentDb, HDInsight, Azure data lakes,

Azure machine learning, Cognitive services ARM Templates Message Queues

Azure Functions (1)

Serverless – paradigm shift FaaS = Function as a Service Azure function = Event + Code Triggers – Schedule, Http, Queues, Storage Bindings – connecting to data Programmed in JavaScript, C#, Python, PHP, Powershell Functions are stateless Try it for free - monthly grant of 1,000,000 executions and

400,000 GB-s Execution limited to 5 minutes

Azure Functions (2)

Monitoring for free Try online: https://functions.azure.com/try Can be edited inside of the Azure portal Can be edited and deployed from Visual Studio

Azure Queuing technologies

Storage Queues- Supports Get/Put/Peek- Can store over 80Gb in queue- No ordering guarantee- Cheap (100x cheaper than Azure Service Bus)

Service Bus Queues- Can support message exchange patterns e.g. Pub/Sub,Topics- Doesn't require polling- 80Gb queue limit- AMQP- Apps can support at-most-once delivery- Partitions to overcome broker limitation

Event Hubs- Can handle huge scale or event streaming needs- AMQP or HTTP

Messaging pattern – One way

One-way

S Q C

Message Broker

Azure Storage Queue Demo

Workload: Calculate factors of numbers90 = 1, 2, 3, 5, 6, 9, 10, 15, 18, 30, 45, 90

Source RestAPI Queue

FactorCalc Queue Consumer

Azure Queues Demo

Azure Function auto-scaling

Scales using heuristics

Source RestAPI Queue

FactorCalc Queue Consumer

Messaging – Competing Consumers

Competing Consumers (a.k.a Worker queues)

S Q

C

CMessage Broker

Types of messaging pattern

One-way Competing consumers RPC Pub-Sub Routing Topics Scatter-Gather

Messaging – Pub sub

Publish-subscribe

S

Q C

CQ

E

Message Broker

Messaging – Routing

Routing

S

Q C

CQ

E

Message Broker

Large

Medium

Small

Messaging – RPC

Remote Procedure Call

S

Q

C

Q

Message Broker

Request queue

Response queue

Messaging – Scatter-Gather

Scatter-Gather

S

Q C

CQE

Message Broker

Dynamic response queue

Q

Demo: Azure Service Bus

Infrastructure as code: What are ARM templates? Azure Service Bus Topics message pattern Demo

ARM Templates (1)

Azure Resource Manager templates Infrastructure as code JSON Schema

https://github.com/Azure/azure-quickstart-templates

Azure Service Bus

Queues → Single Consumer Topics & Subscriptions → Multiple consumers Patterns– Topics– Routing– Pub/Sub

Messaging – Topics

Topics

S

Q C

CQ

E

Message Broker

*.*.mail

*.small.*

high.*.*

Q C

High.small.mail

Low.large.post

Low.small.post

Azure Service Bus Demo

S

All Products C

COrange, White, Yellow

E

Azure Service Bus

Large C

All Products

SubscriptionsTopics

Expensive

Light Colour

Large

Cost > 80 C

Products:Colour.Size.Cost

Azure Service Bus Demo

Break!

Demo: RabbitMQ in Docker

Introduction to Docker Introduction to RabbitMQ Setup .Net core on docker Routing message pattern Demonstration of RabbitMQ Management UI Creation of Exchange and Queue using UI and powershell Running rabbitmq, producer, receiver in docker

Infrastructure

Infrastructure: How we used to do it– Windows image, Packer, Boxstarter, Chocolatey, Vagrant = 6Gb

What is Docker?– Lightweight containers– Run on a host operating system (either Linux or Windows)– Inside of VirtualBox or HyperV host– Extensive support - lots of images available on docker hub– Very fast to setup and start images– Docker is scriptable

What is kitematic?

RabbitMQ

AMQP – Advance Message Queuing Protocol Built on Erlang Easy to setup - runs on Linux, Windows and containers Exchanges Queues Routing Key Virtual Hosts Web based Management UI EasyNetQ (https://www.infoq.com/presentations/RabbitMQ-NET-EasyNetQ) Powershell management

Docker and .Net Core

.Net Core runs on Linux Choose the right NuGet package Build .Net core to run inside docker:

– Build a container image to build the C# solution (DockerFile.build)– Create an instance of the build container– Copy the .Net core build output from the docker build container– Use the .Net core build output to create images with .Net core (DockerFile)– Run the containers (docker run or docker-compose)

Messaging – Routing

Routing

S

Q C

CQ

E

Message Broker

Large

Medium

Small

RabbitMQ Demo

S

Q C

E

Large Products

Products:Colour.Size.Cost

*

Large

Rabbit MQ Demo

… and now!

Reliability patternsWhat is a service bus?Messaging jargonTypes of messaging systemsHow to select a messaging system?Event sourcing

Reliability patterns (1)

S Q C

Message Broker

What could possibly go wrong?

Reliability patterns (2)

Acknowledgement levelsConnection failures -> Client automatically reconnect, use heartbeatsFailure at broker/queue -> Persist to diskMessaging hardware failure -> Clustering (e.g. RabbitMQ and MSMQ)Producer failure -> Ack will not be received so need Idempotent messagingBad Config (Message not routed) -> Set ack to confirm message routedEasyNetQ - Error handling, connection management, publisher confirms for reliable messaginghttps://www.rabbitmq.com/reliability.html

What is a service bus?

Layer on top of messaging systems– Handles Retries, transaction management

nServiceBus and MassTransit Sagas (aka transactions across messaging)

Messaging fundamentals

Brokered vs non-brokered Persisted vs non-persisted Dead letter queue

Messaging systems

Azure Storage Queue Azure Service Bus RabbitMQ MSMQ ZeroMQ WebSphereMQ / IBM MQ Kafka Azure Event Hubs AWS Simple Queue Service

ZeroMQ

Concurrency framework using socketsFaster than RabbitMQ, but you have to work harder for itUsed by Cisco, AT&T, NASA, Spotify, Microsoft

MSMQ

Store and forward: can do point to point or fan outBrokerless, but for reliability you need a cluster plus MSDTCLimited to windowsUsed by Microsoft, nServiceBus

Kafka

Distributed streaming platform (but can do queuing and pub/sub)Used by thousands of companies like LinkedIn, Yahoo, Twitter, Netflix, Spotify, Pinterest, Uber, Paypal etc etcStreams data efficiently in real time to a replicated commodity cluster running LinuxCan form ecosystems with Hadoop, Spark etcApache projectAppend Onlyhttps://kafka.apache.org/uses

How do I select a messaging system?

Performance Messaging patterns Brokered/Non-brokered Platform Other features

− Duplicate detection− Can it be used with a Service Bus?

Ecosystem

Event Sourcing

Append Only – Store the history of events Don't hold state, build up state from the history of events Useful when holding a log of everything is useful EventStore LMAX architecture:

https://martinfowler.com/articles/lmax.html

Further Resources

Pluralsight₋ https://app.pluralsight.com/library/courses/rabbitmq-by-example₋ https://app.pluralsight.com/library/courses/rabbitmq-dotnet-developers₋ https://app.pluralsight.com/library/courses/message-queue-

fundamentals-dotnet₋ https://app.pluralsight.com/library/courses/docker-web-development

SyncFusion RabbitMQ succinctly bookhttps://github.com/Azure/azure-quickstart-templates

Summary

Demos of Messaging patterns– One Way– Competing Consumer– Topics– Routing

Messaging systems– Azure Storage Queues– Azure Service Bus– RabbitMQ

Hipster tech– FaaS – Build auto-scaling apps without the hassle of any infrastructure– ARM Templates – Infrastructure as code– Docker – fast setup of infrastructure

Any Questions?

@johnstaveley