Running Asynchronous Workloads in Windows Azure

NameTitleMicrosoft Corporation

Why Use Asynchronous Patterns?Improve apparent responsivenessPerform long running work in backgroundReturn immediately to the clientAbsorb spikes in load

Basic Asynchronous Pattern

Web Role Worker Role

StorageQueue

LB

LB

Worker RoleWorker RoleWeb RoleWorker Role

Blob Container

Table

1. User uploads large image file2. Image inserted into blog storage3. Message placed on queue incl BLOB URI and metadata4. Worker role is polling queue. Reads message from queue5. Worker role processes message, reads from BLOB storage,

generates thumbnail6. Thumbnail and metadata stored in Table storage7. Message deleted from queue

30mb JPEG

Idempotencyf(x) = f(f(x))

Messages Process At Least Once

Web Role Worker Role

StorageQueue

LB

LB

Worker RoleWorker RoleWeb RoleWorker Role

1. Debit bank account $100 message2. Worker role reads message3. Balance debited $1004. Worker role is torn before message can be

deleted5. 3 minutes later, message re-appears on queue6. Worker role reads message7. Balance debited $1008. Message deleted from queue9. Chaos ensues.....10.Customer calls bank.....

Balance = $1000Balance = $900Balance = $800

Solving the Idempotency Problem

Web Role Worker Role

StorageQueue

LB

LB

Worker RoleWorker RoleWeb RoleWorker Role

1. Debit bank account $100 message with transaction ID2. Worker role reads message. Checks transaction ID not present.3. Writes transaction ID with state ‘Started’ to ‘Replay Log’4. Balance debited $1005. Worker role is torn before message can be deleted6. 3 minutes later, message re-appears on queue7. Worker role reads message. Checks transaction ID. It is present in state

started.8. Compensating message written to another queue9. Message deleted from queue10.Compensatory message processed.Balance = $1000Balance = $900

Table

QueryQuery

Queue

Replay Log StorageWhere possible store within same transaction boundary as primary operationSQL Azure

Simple as we have database wide transactionsWindows Azure Tables

Store replay log in same partition as primary dataUse ‘Entity Group Transaction’ to insert/update as one atomic operation.

http://tinyurl.com/EntityGroup

Poison Message Handling

Poison MessagesWindows Azure Queues provide a simple and fast async mechanismSome messages may not be able to be handledPoison messages may become zombiesEven idempotent messages can be poisonousIs a poisonous message idempotent though?

Spitting the Poison Pill<QueueMessagesList> <QueueMessage> <MessageId>string-message-id</MessageId> <InsertionTime>insertion-time</InsertionTime> <ExpirationTime>expiration-time</ExpirationTime> <PopReceipt>opaque-string-receipt-data</PopReceipt> <TimeNextVisible>time-next-visible</TimeNextVisible> <DequeueCount>integer</DequeueCount> <MessageText>message-body</MessageText> </QueueMessage></QueueMessagesList>

Spitting the Poison PillGet message from queueCheck dequeue countSet thresholds before a message is placed into a Poison Queue and deleted

Ensure that your Poison test is at the top of the batch (i.e. avoid reliance on parsing message)Poison queue should clean up state and/or escalate

Avoid writing your own poison tracker…

Poison Message HandlingPoisons messages should be moved to another queue and loggedPotential to resolve blocking issue

Code fixRestore missing resourcesEtc…

Re-inject message back into original queue

Dynamic Worker Roles

Problem and SolutionProblem

Worker Roles Most Cost Effective at 100% Utilization

Often have many work types, none of which requires 100% of 1 instance

May want to add new work types without redeploying

Solution

Use a generic queue.

Encode message with info to resolve work type

Load assembly to process message from blob storage

Dynamically instantiate and execute

1. User uploads large image file2. Image inserted into blog storage3. Message placed on queue incl BLOB URI and metadata such as image

type4. Worker role is polling queue. Reads message from queue5. Worker role parses message. Pulls appropriate type from blob storage6. Worker dynamically loads type into new AppDomain, dynamically

invokes type7. Code reads image from BLOB storage, generates thumbnail8. Thumbnail and metadata stored in Table storage9. Message deleted from queue

AppDomain

Web Role Worker Role

StorageQueue

LB

LB

Worker RoleWorker RoleWeb RoleWorker Role

Blob Container

Table

30mb JPEG

30mb JPEG

Key Points for Dynamic WorkersUse a smart polling approach for queues

Don’t poll every few milliseconds. Poll based on processing latency.

Use a new AppDomain to separate loaded typesAvoid tearing down entire worker when updating work typesCan also use RoleEnvironment.RequestRecycle

May be value in including Replay Log and Poison tracking as part of Dynamic Worker framework

Handle poison messages & kill the child AppDomain rather than the whole worker roleSupport automatic or manual message re-injection

The MapReduce Pattern

MapReduceSometimes a data set is to large to be handed to a single workerSimple in concept:

Take large problemBreak into small partsDistribute parts to multiple nodes for solvingAggregate small solutions into whole answer

Suited to problems that are;ParallelizableNon real time Data intensive

MapReduce – An ExampleGigapixel Image Processing - Solarize

Upload Web Role

Map Worker Role

Reduce Worker Role

Image Uploaded

Image Split

SlicesSaved To

Blob Storage

Slices Pulled from Blob Storage

Images Processedin Worker Instances

Processed Slices Pushed Back to Blob Storage

Splits ReducedBack to Full Image

Final ImageSaved as BLOBReady for Use

Using Queues EffectivelyTradeoff between IO and ‘chunkiness’ of work

Affects Retry WindowsAffects Load Distribution

Uses Queues to batch or ‘chunk’ work in low CPU compute scenariosUse Multiple Queues for high IO apps

Per worker or by Queue PollingTune based on configuration when possible

SummaryAsynchronous processing can massively increase the perceived speed of your application Dispatch via Windows Azure Queues requires some care

Ensure that your messages are idempotent or have a compensation mechanism

Poison messages can dramatically reduce scaleImplement poison detection

Dynamics Worker roles can provide flexibility and cost effectiveness

© 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the

date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.