Massaging the Pony:Message Queues and You

Shawn RiderPBS Education

DjangoCon 2010

http://www.flickr.com/photos/funtik/1175522045

What is a Message Queue?1. A system for enabling asynchronous

processing of discrete tasks.2. Super awesome.

http://www.flickr.com/photos/gadl/89650415

What are they good for?

Alleviate server issues by offloading processing

http://www.flickr.com/photos/islandgyrl/3197469932

What are they good for?

Make better user experiences

http://www.flickr.com/photos/bbaltimore/6779682

What are they good for?

Increase the reliability of third party service integrations.

What are they good for?

Background media processinghttp://www.flickr.com/photos/factoryjoe/2882992091

What are they good for?

• Repeating Tasks can replace Cron jobs • Keep all your code in your project

Available MQ Solutions

There are many solutions out there. To name a few of them:

• RabbitMQ• Amazon Simple Queue System• Apache MQ• Gearman• OpenAMQ• Peafowl• Q4M• Starling• Sun Java System Message Queue

Message Queue Protocols

• AMQP: Advanced Message Queuing Protocol• JMS: Java Messaging Service• STOMP: Streaming Text Oriented Messaging

Protocol

Criteria for Broker Selection

The following criteria were important to us in selecting a solution. You may have some different needs.

• Handling exceptions and recovery• Low level of required maintenance• Ease of deployment• Persistent queuing• Community support• What language is it written in? How compatible is that with

our current systems?• Cluster support

Failsafe MQ Brokers

• Must be able to survive a server failure.• Upon restarting the MQ Broker, processing

should resume from the last un-processed messages in queues

Queue Durability

The Messages Queue Broker must be able to reconstruct the queues when it is restarted.

http://www.flickr.com/photos/raul/846318014

Message Persistence

The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server

Our Choice

• Celery (a Django task management app)• Carrot (Django middleware)• PyAMQP (Python module)• RabbitMQ (MQ Broker software)

The Production System

Webapp

DatabaseWorker

Server(s)

CeleryRabbit

MQ

MQ in Development Environs

It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.

MQ in Development Environs

• Carrot can be set up to use different queuing backend for development purposes

• GhettoQ provides database backend for carrot where a database serves as the queue storage

• The broker is a simple Django application that monitors the queue in DB

• Inefficient, therefore only suitable for development environments

CARROT_BACKEND = ‘qhettoq.taproot.Database’

Ease a future MQ implementation

Isolate functionality into reusable components.

Recycle Existing Code

Django, HTTP Requests & MQ

• Most functionalities provided through Django applications are tightly coupled with HTTP requests e.g. form data processing

• In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters

• HTTP requests are usually large and not easily serializable. e.g. WSGI requests

The PickleableHTTPRequest Object

Using PickleableHttpRequest

if request.GET.get(‘download’): try: result = EnrollmentCSVExportTask.delay( PickleableHttpRequest( request, attributes=\ [‘user’, ‘admin_current_organization’] ) )

Making Message Queue Tasks

It is fast and easy to add a decorator to specific functions to create Message Queue tasks.

from celery.decorators import task

@taskdef add(x, y): return x + y

Explicitly Inherit from Celery’s Task Object

Then What?

Some Message Queue tasks can complete with no notification required.

For other tasks:• Notify by email• Use an AJAX listener to notify users• Some other messaging

In Review

• Choose your MQ solution wisely• Set up a robust system• Leverage your existing code• Replace trouble spots as they come up• Profit!

Message Queues Make Life Better

References and Links

Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

Ask Solem Hoel’s Github Repos: http://github.com/askCelery Project: http://celeryproject.org/

PickleableHttpRequest by Nowell Strite (@nowells)http://gist.github.com/571027

Special thanks to Tareque Hossain (http://codexn.com)