5 useful message queue patterns
I love message queues and find them incredibly useful. This article outlines several cases where introducing a queue can make your system simpler and more robust.
Messaging is a vast topic, but here are a few patterns that I've found most useful on real world projects.
Task Queues
By far the most common queueing pattern. Producers write messages to queues and don't wait for a reply. Consumers listen to the queue and process messages.
Useful for: * Performing work in the background that users don't need immediately * Transparently scaling work (just add more consumers)
Common implementation semantics: * Consumers receive messages in FIFO order * Messages are redelivered if the consumer doesn't explicitly ack or delete them
Delayed Jobs
Typically a minor variant on the job queue. Producers can specify that the message delivery be delayed.
Useful for: * Retrying integration with a remote system. Example: You write a job to bill customers through PayPal at the end of each month. If paypal is unreachable you requeue the task with a 24 hour delay. * Deferred state cleanup. You allocate a resource and queue a delayed message that tells a consumer to cleanup the state in a few hours. * Grace periods. Example: You want a customer to be able to cancel sending an email or placing an order for up to x minutes. You enqueue the job with a delay. If the customer cancels within the grace period you simply delete the message from the queue.
Fanout
Fanout destinations are similar to email aliases. The destination that producers write messages to is an alias that maps to one or more child queues. When the broker receives a new message it delivers that message to each child queue. Consumers dequeue from the child queues, not from the fanout destination.
With an email alias the sender doesn't need to know who is subscribed
to the alias. The composition of the alias can change at any time.
Similarly a fanout queue acts as a logical destination. The child queues
that the fanout queue points to can change at any time without impacting
the producers.
Useful for: * Event notification. When an event occurs the producer sends a message. Subsystems that require notification are added to the fanout configuration. Each receive a copy of the message. * Logging
Message Groups
Imagine you have a system that accepts data from a 3rd party as a stream of messages. Each message represents a different write operation and is related to a single entity in your system. Each entity is independent.
So far this looks like a fine problem for a job queue. Messages come in and go into a single job queue. Consumers dequeue messages in FIFO order and process them.
But this doesn't work due to race conditions. If two consumers each dequeue a message related to the same entity and apply them in parallel then state could be modified out of order.
You could write messages to separate job queues -- one per entity. But then consumers would have to have some way of subscribing to all the queues. As the number of entities grows, this could get complicated.
This is the problem that message groups solve. The solution works like this:
- Producers set a message group id header on each message. The details are implementation dependant. But in our example, the message group id would be something like:
[entity type]:[entity id]
- All messages are sent to a single destination
- Consumers subscribe to the single destination, as though it were a single job queue
- The queue groups messsages by message group id and guarantees that only a single consumer is sent messages for each group id at any given time.
You can also use this pattern to ensure fair resource utilization across multiple customers when setting up a job queue.
- Customers enqueue background jobs. You set the group id to the customer id.
- This ensures that each customer gets a single consumer, but no more
I first learned about this pattern when reading the ActiveMQ Message Group docs.
RPC
Queues are usually used to route messages asynchronously. However, using return queues you can turn a regular job queue into a load balanced synchronous remote service.
Here's how it works:
- Producers set a "Reply-To" header that specifies the name of the queue to send the reply message to. This is typically a single use UUID. Some implementations hide this detail from you. Messages are sent to the job queue normally.
- Producer blocks on the Reply-To queue, waiting for a response message
- Consumers dequeue normally and process the message
- Consumer sees the "Reply-To" header and sends a response to that queue
- Producer reads response
This pattern allows the same consumer code to function asynchronously or synchronously based simply on the presence of the Reply-To header on the inbound message.
Of course the message queue can become a potential bottleneck, and only queues with very low latency are appropriate for this pattern. You probably wouldn't want to attempt this with a hosted service like SQS.
General Purpose Queues
name | model | wire protocol | FIFO | task queues | delayed jobs | fanout | message groups | rpc | notes |
---|---|---|---|---|---|---|---|---|---|
Amazon SQS/SNS | SaaS | HTTP | No | Yes | No | Yes (SNS to SQS) | No | Not recommended due to latency | Pros: Nothing to install. Good for integrating remote systems. Cons: High latency. Messages may be delivered more than once. Not FIFO |
RabbitMQ | Open source (erlang) | AMQP or STOMP | Yes | Yes | ? | Yes | No | Yes, using temporary destinations | Full featured message broker, but more complex to install than some solutions |
beanstalkd | Open source (C) | Custom | Yes | Yes | Yes | No | No | Yes, but Reply-To must be part of your message payload | Very simple and fast. Great choice if you only need job queues. |
Kestrel | Open source (Scala) | Custom | No if clustered | Yes | No | No | No | Yes, but Reply-To must be part of your message payload | Used at Twitter. Message queues are transparently partitioned. Designed for high volume. |
Gearman | Open source (C) | Custom | Yes | Yes | Not natively | No | No | Yes | Client bindings automatically route jobs to functions on worker processes, simplifying writing consumer code. |
ActiveMQ | Open source (Java) | Custom, STOMP | Yes | Yes | Yes | Yes - virtual topics | Yes | JMS client (Java) supports all features. STOMP support is not as robust. In my tests message persistence was not sufficiently reliable. | |
HornetQ | Open source (Java) | Custom, STOMP | Yes | Yes | Yes | Yes, using diverts | Yes | JMS only | Formerly JBoss MQ. Full JMS implementation. STOMP support is not as robust. |
Language Specific
The solutions below offer higher level queueing features and tight integration with a single programming language. Typically this means that your producers and consumers have to use the same language, and in some cases, have to share the same code tree due to how messages are marshalled and consumed.
In addition these solutions typically include monitoring and admin features, which makes it easier to visualize the state of your system.
My notes are a bit incomplete on these systems. They all seem to be optimized for the task queue use case, but please comment with any corrections.
name | language | task queues | delayed jobs | fanout | message groups | rpc | backends | notes |
---|---|---|---|---|---|---|---|---|
celery | Python | Yes | Yes | No | No | Yes | RabbitMQ (preferred), Redis, beanstalkd, MongoDB, CouchDB, SQL databases | Very active community |
resque | Ruby | Yes | ? | No? | No? | No? | Redis | Github sponsored. Good visualization tools for monitoring workers/jobs. |
octobot | Java, Scala, Ruby, Python | Yes | No? | No? | No? | No? | AMQP (RabbitMQ), beanstalkd, Redis | Very limited docs. Used at Urban Airship |