Automated Build System with Docker, Jenkins and Azure: Intro
Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
-
Upload
john-staveley -
Category
Software
-
view
864 -
download
8
Transcript of Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions
Message System Fundamentals
By John Staveley27/07/2017
https://uk.linkedin.com/in/johnstaveley/@johnstaveley
Who am I?
John StaveleyContract Mvc.net developerOrganiser @LeedsSharpUser of messaging systems for the last 4 years (MSMQ,
Rabbit, Azure Service Bus etc)
Overview
Why use messaging systems?What are they? Benefits?Demos (Azure Storage Queues, Azure Service Bus,
RabbitMQ)Azure FunctionsInfrastructure as code.Net Core and DockerOther Messaging implementationshttps://github.com/johnstaveley/MessagingPresentationSlides on SlideShare
Use cases and benefits
Distributed workloads Increased reliability (cf fallacies of distributed networking) Synchronous vs Asynchronous processes Loosely coupled systems (Microservices) The promise of fast reads and fast writes LinkedIn, CompareTheMarket.com, Facebook
Demo: Azure Storage Queues
What is Azure? What is Serverless / FaaS - Azure functions? What are Azure Queues? Message patterns - One way/competing consumers Demo
What is Azure?
Second largest cloud provider in the world IaaS (Windows or Linux VMs), PaaS, FaaS Data storage, Notifications, Load Balancing, Active
Directory, Application insights SQL Azure, DocumentDb, HDInsight, Azure data lakes,
Azure machine learning, Cognitive services ARM Templates Message Queues
Azure Functions (1)
Serverless – paradigm shift FaaS = Function as a Service Azure function = Event + Code Triggers – Schedule, Http, Queues, Storage Bindings – connecting to data Programmed in JavaScript, C#, Python, PHP, Powershell Functions are stateless Try it for free - monthly grant of 1,000,000 executions and
400,000 GB-s Execution limited to 5 minutes
Azure Functions (2)
Monitoring for free Try online: https://functions.azure.com/try Can be edited inside of the Azure portal Can be edited and deployed from Visual Studio
Azure Queuing technologies
Storage Queues- Supports Get/Put/Peek- Can store over 80Gb in queue- No ordering guarantee- Cheap (100x cheaper than Azure Service Bus)
Service Bus Queues- Can support message exchange patterns e.g. Pub/Sub,Topics- Doesn't require polling- 80Gb queue limit- AMQP- Apps can support at-most-once delivery- Partitions to overcome broker limitation
Event Hubs- Can handle huge scale or event streaming needs- AMQP or HTTP
Messaging pattern – One way
One-way
S Q C
Message Broker
Azure Storage Queue Demo
Workload: Calculate factors of numbers90 = 1, 2, 3, 5, 6, 9, 10, 15, 18, 30, 45, 90
Source RestAPI Queue
FactorCalc Queue Consumer
Azure Queues Demo
Azure Function auto-scaling
Scales using heuristics
Source RestAPI Queue
FactorCalc Queue Consumer
Messaging – Competing Consumers
Competing Consumers (a.k.a Worker queues)
S Q
C
CMessage Broker
Types of messaging pattern
One-way Competing consumers RPC Pub-Sub Routing Topics Scatter-Gather
Messaging – Pub sub
Publish-subscribe
S
Q C
CQ
E
Message Broker
Messaging – Routing
Routing
S
Q C
CQ
E
Message Broker
Large
Medium
Small
Messaging – RPC
Remote Procedure Call
S
Q
C
Q
Message Broker
Request queue
Response queue
Messaging – Scatter-Gather
Scatter-Gather
S
Q C
CQE
Message Broker
Dynamic response queue
Q
Demo: Azure Service Bus
Infrastructure as code: What are ARM templates? Azure Service Bus Topics message pattern Demo
ARM Templates (1)
Azure Resource Manager templates Infrastructure as code JSON Schema
https://github.com/Azure/azure-quickstart-templates
Azure Service Bus
Queues → Single Consumer Topics & Subscriptions → Multiple consumers Patterns– Topics– Routing– Pub/Sub
Messaging – Topics
Topics
S
Q C
CQ
E
Message Broker
*.small.*
high.*.*
Q C
High.small.mail
Low.large.post
Low.small.post
Azure Service Bus Demo
S
All Products C
COrange, White, Yellow
E
Azure Service Bus
Large C
All Products
SubscriptionsTopics
Expensive
Light Colour
Large
Cost > 80 C
Products:Colour.Size.Cost
Azure Service Bus Demo
Break!
Demo: RabbitMQ in Docker
Introduction to Docker Introduction to RabbitMQ Setup .Net core on docker Routing message pattern Demonstration of RabbitMQ Management UI Creation of Exchange and Queue using UI and powershell Running rabbitmq, producer, receiver in docker
Infrastructure
Infrastructure: How we used to do it– Windows image, Packer, Boxstarter, Chocolatey, Vagrant = 6Gb
What is Docker?– Lightweight containers– Run on a host operating system (either Linux or Windows)– Inside of VirtualBox or HyperV host– Extensive support - lots of images available on docker hub– Very fast to setup and start images– Docker is scriptable
What is kitematic?
RabbitMQ
AMQP – Advance Message Queuing Protocol Built on Erlang Easy to setup - runs on Linux, Windows and containers Exchanges Queues Routing Key Virtual Hosts Web based Management UI EasyNetQ (https://www.infoq.com/presentations/RabbitMQ-NET-EasyNetQ) Powershell management
Docker and .Net Core
.Net Core runs on Linux Choose the right NuGet package Build .Net core to run inside docker:
– Build a container image to build the C# solution (DockerFile.build)– Create an instance of the build container– Copy the .Net core build output from the docker build container– Use the .Net core build output to create images with .Net core (DockerFile)– Run the containers (docker run or docker-compose)
Messaging – Routing
Routing
S
Q C
CQ
E
Message Broker
Large
Medium
Small
RabbitMQ Demo
S
Q C
E
Large Products
Products:Colour.Size.Cost
*
Large
Rabbit MQ Demo
… and now!
Reliability patternsWhat is a service bus?Messaging jargonTypes of messaging systemsHow to select a messaging system?Event sourcing
Reliability patterns (1)
S Q C
Message Broker
What could possibly go wrong?
Reliability patterns (2)
Acknowledgement levelsConnection failures -> Client automatically reconnect, use heartbeatsFailure at broker/queue -> Persist to diskMessaging hardware failure -> Clustering (e.g. RabbitMQ and MSMQ)Producer failure -> Ack will not be received so need Idempotent messagingBad Config (Message not routed) -> Set ack to confirm message routedEasyNetQ - Error handling, connection management, publisher confirms for reliable messaginghttps://www.rabbitmq.com/reliability.html
What is a service bus?
Layer on top of messaging systems– Handles Retries, transaction management
nServiceBus and MassTransit Sagas (aka transactions across messaging)
Messaging fundamentals
Brokered vs non-brokered Persisted vs non-persisted Dead letter queue
Messaging systems
Azure Storage Queue Azure Service Bus RabbitMQ MSMQ ZeroMQ WebSphereMQ / IBM MQ Kafka Azure Event Hubs AWS Simple Queue Service
ZeroMQ
Concurrency framework using socketsFaster than RabbitMQ, but you have to work harder for itUsed by Cisco, AT&T, NASA, Spotify, Microsoft
MSMQ
Store and forward: can do point to point or fan outBrokerless, but for reliability you need a cluster plus MSDTCLimited to windowsUsed by Microsoft, nServiceBus
Kafka
Distributed streaming platform (but can do queuing and pub/sub)Used by thousands of companies like LinkedIn, Yahoo, Twitter, Netflix, Spotify, Pinterest, Uber, Paypal etc etcStreams data efficiently in real time to a replicated commodity cluster running LinuxCan form ecosystems with Hadoop, Spark etcApache projectAppend Onlyhttps://kafka.apache.org/uses
How do I select a messaging system?
Performance Messaging patterns Brokered/Non-brokered Platform Other features
− Duplicate detection− Can it be used with a Service Bus?
Ecosystem
Event Sourcing
Append Only – Store the history of events Don't hold state, build up state from the history of events Useful when holding a log of everything is useful EventStore LMAX architecture:
https://martinfowler.com/articles/lmax.html
Further Resources
Pluralsight₋ https://app.pluralsight.com/library/courses/rabbitmq-by-example₋ https://app.pluralsight.com/library/courses/rabbitmq-dotnet-developers₋ https://app.pluralsight.com/library/courses/message-queue-
fundamentals-dotnet₋ https://app.pluralsight.com/library/courses/docker-web-development
SyncFusion RabbitMQ succinctly bookhttps://github.com/Azure/azure-quickstart-templates
Summary
Demos of Messaging patterns– One Way– Competing Consumer– Topics– Routing
Messaging systems– Azure Storage Queues– Azure Service Bus– RabbitMQ
Hipster tech– FaaS – Build auto-scaling apps without the hassle of any infrastructure– ARM Templates – Infrastructure as code– Docker – fast setup of infrastructure
Any Questions?
@johnstaveley