RabbitMQ & HutchPete Hamilton - GoCardless

About Me

• Developer at GoCardless

• CompSci Student at ICL

• Github: https://github.com/petehamilton

• Blog: pete-hamilton.co.uk

• Twitter: @peterejhamilton

SOA CommunicationA (very) brief overview

Awesome Service

SOA CommunicationA (very) brief overview

Awesomer Services

SOA CommunicationA (very) brief overview

Awesomer Services

SOA CommunicationA (very) brief overview

?Client - Server?

Message Queues?

RabbitMQ

RabbitMQWhy?

• Nice Learning Curve

• Reliable

• Active Community

• Polyglot

• Scalable

• Proven itself in battle production

RabbitMQWhy?

• Payments are pretty serious stuff

• Async = Potential problems

• Network Failure?

• Consumer Crashes?

• Duplication/redelivery?

• RabbitMQ helps with confirms, acknowledgements, mandatory flags

RabbitMQ Routing

P X

C1

C2

Q1

Q2

Fanout Exchange

RabbitMQ Routing

P X

C1

C2

Q1

Q2

Direct Exchange

RabbitMQ Routing

P X

C1

C2

type=direct

fireflystar_trek

big_bang_theory

Q1

Q2

Topic Exchange

RabbitMQ Routing

P X

C1

C2

type=topic

tv.scifi.*

tv.comedy.*

C3

tv.#

Q1

Q2

Q3Sample Message Routing Keys:- tv.comedy.big_bang_theory- tv.scifi.firefly- tv.scifi.star_trek

RabbitMQ

• Very Flexible, but...

• Which data structure do you use?

• How do you serialise your data?

• Which fields do you require in your messages?

• How should messages be routed? fan out? direct? topically?

• How do new nodes set themselves up?

• What form should “Boiler Plate” code take?

A system for processing RabbitMQ messages in Ruby

• Our Conventions (Warning: Opinions!)

• Topic Exchange Routing

• Serialisation w/ JSON

• Message IDs and Timestamps

• Simple to use

• Configurable

• Specific ruby files or full Rails apps

Publisher

Consumer

https://github.com/gocardless/hutch

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

S S

M

C1

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

S S

M

x C1

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

S S

M

x C1

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

S S

M

C1

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

S S

M

C1

x

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

M S

M

C1

x

RabbitMQScaling

• Good guide to clustering online

• Potential for issues with multiple nodes

• Clients exist for load testing, but...

• We wanted ++ parallelism

• Built “AMQPC”

AMQPC

• Written in Go

• Producer & Consumer modes

• Traffic Modes

• One off burst

• Intermittent burst

• Consistent load

AMQPC

https://github.com/gocardless/amqpc

Useful Links

• Hutch - www.github.com/gocardless/hutch

• AMQPC - www.github.com/gocardless/amqpc

• RabbitMQ - http://www.rabbitmq.com/getstarted.html

Questions?