5 useful message queue patterns

Posted: December 30, 2011

I love message queues and find them incredibly useful. This article outlines several cases where introducing a queue can make your system simpler and more robust.

Messaging is a vast topic, but here are a few patterns that I've found most useful on real world projects.

Task Queues

By far the most common queueing pattern. Producers write messages to queues and don't wait for a reply. Consumers listen to the queue and process messages.

Useful for: * Performing work in the background that users don't need immediately * Transparently scaling work (just add more consumers)

Common implementation semantics: * Consumers receive messages in FIFO order * Messages are redelivered if the consumer doesn't explicitly ack or delete them

Delayed Jobs

Typically a minor variant on the job queue. Producers can specify that the message delivery be delayed.

Useful for: * Retrying integration with a remote system. Example: You write a job to bill customers through PayPal at the end of each month. If paypal is unreachable you requeue the task with a 24 hour delay. * Deferred state cleanup. You allocate a resource and queue a delayed message that tells a consumer to cleanup the state in a few hours. * Grace periods. Example: You want a customer to be able to cancel sending an email or placing an order for up to x minutes. You enqueue the job with a delay. If the customer cancels within the grace period you simply delete the message from the queue.

Fanout

Fanout destinations are similar to email aliases. The destination that producers write messages to is an alias that maps to one or more child queues. When the broker receives a new message it delivers that message to each child queue. Consumers dequeue from the child queues, not from the fanout destination.

With an email alias the sender doesn't need to know who is subscribed to the alias. The composition of the alias can change at any time.
Similarly a fanout queue acts as a logical destination. The child queues that the fanout queue points to can change at any time without impacting the producers.

Useful for: * Event notification. When an event occurs the producer sends a message. Subsystems that require notification are added to the fanout configuration. Each receive a copy of the message. * Logging

Message Groups

Imagine you have a system that accepts data from a 3rd party as a stream of messages. Each message represents a different write operation and is related to a single entity in your system. Each entity is independent.

So far this looks like a fine problem for a job queue. Messages come in and go into a single job queue. Consumers dequeue messages in FIFO order and process them.

But this doesn't work due to race conditions. If two consumers each dequeue a message related to the same entity and apply them in parallel then state could be modified out of order.

You could write messages to separate job queues -- one per entity. But then consumers would have to have some way of subscribing to all the queues. As the number of entities grows, this could get complicated.

This is the problem that message groups solve. The solution works like this:

  • Producers set a message group id header on each message. The details are implementation dependant. But in our example, the message group id would be something like: [entity type]:[entity id]
  • All messages are sent to a single destination
  • Consumers subscribe to the single destination, as though it were a single job queue
  • The queue groups messsages by message group id and guarantees that only a single consumer is sent messages for each group id at any given time.

You can also use this pattern to ensure fair resource utilization across multiple customers when setting up a job queue.

  • Customers enqueue background jobs. You set the group id to the customer id.
  • This ensures that each customer gets a single consumer, but no more

I first learned about this pattern when reading the ActiveMQ Message Group docs.

RPC

Queues are usually used to route messages asynchronously. However, using return queues you can turn a regular job queue into a load balanced synchronous remote service.

Here's how it works:

  • Producers set a "Reply-To" header that specifies the name of the queue to send the reply message to. This is typically a single use UUID. Some implementations hide this detail from you. Messages are sent to the job queue normally.
  • Producer blocks on the Reply-To queue, waiting for a response message
  • Consumers dequeue normally and process the message
  • Consumer sees the "Reply-To" header and sends a response to that queue
  • Producer reads response

This pattern allows the same consumer code to function asynchronously or synchronously based simply on the presence of the Reply-To header on the inbound message.

Of course the message queue can become a potential bottleneck, and only queues with very low latency are appropriate for this pattern. You probably wouldn't want to attempt this with a hosted service like SQS.


General Purpose Queues

name model wire protocol FIFO task queues delayed jobs fanout message groups rpc notes
Amazon SQS/SNS SaaS HTTP No Yes No Yes (SNS to SQS) No Not recommended due to latency Pros: Nothing to install. Good for integrating remote systems. Cons: High latency. Messages may be delivered more than once. Not FIFO
RabbitMQ Open source (erlang) AMQP or STOMP Yes Yes ? Yes No Yes, using temporary destinations Full featured message broker, but more complex to install than some solutions
beanstalkd Open source (C) Custom Yes Yes Yes No No Yes, but Reply-To must be part of your message payload Very simple and fast. Great choice if you only need job queues.
Kestrel Open source (Scala) Custom No if clustered Yes No No No Yes, but Reply-To must be part of your message payload Used at Twitter. Message queues are transparently partitioned. Designed for high volume.
Gearman Open source (C) Custom Yes Yes Not natively No No Yes Client bindings automatically route jobs to functions on worker processes, simplifying writing consumer code.
ActiveMQ Open source (Java) Custom, STOMP Yes Yes Yes Yes - virtual topics Yes JMS client (Java) supports all features. STOMP support is not as robust. In my tests message persistence was not sufficiently reliable.
HornetQ Open source (Java) Custom, STOMP Yes Yes Yes Yes, using diverts Yes JMS only Formerly JBoss MQ. Full JMS implementation. STOMP support is not as robust.

Language Specific

The solutions below offer higher level queueing features and tight integration with a single programming language. Typically this means that your producers and consumers have to use the same language, and in some cases, have to share the same code tree due to how messages are marshalled and consumed.

In addition these solutions typically include monitoring and admin features, which makes it easier to visualize the state of your system.

My notes are a bit incomplete on these systems. They all seem to be optimized for the task queue use case, but please comment with any corrections.

name language task queues delayed jobs fanout message groups rpc backends notes
celery Python Yes Yes No No Yes RabbitMQ (preferred), Redis, beanstalkd, MongoDB, CouchDB, SQL databases Very active community
resque Ruby Yes ? No? No? No? Redis Github sponsored. Good visualization tools for monitoring workers/jobs.
octobot Java, Scala, Ruby, Python Yes No? No? No? No? AMQP (RabbitMQ), beanstalkd, Redis Very limited docs. Used at Urban Airship