At points, we have a really good homegrown message queueing service, called QHub. It is a stable and mature JMS-based queue product with many important features:
- message prioritisation
- message retry
- well known, and well understood interface
There were a couple of important drawbacks though:
- clients need to implement JMS queues to use QHub
- QHub was implemented as an application running in a WebLogic container. For reasons we never fully understood, the version of WebLogic running QHub must be the same as any other application that talks to QHub. Normally this is not an issue, but it does make container upgrades needlessly risky.
One of our backend applications, which processes credit card transactions (PAYMENT) had a new requirement to communicate with an internal middle tier service called ePoch. Ideally, QHub would have been used, except that we don’t want to go down the rabbit hole of needing to keep two more applications in lockstep with QHub’s WebLogic version. To get around this in the past, we’ve implemented HTTP interfaces for passing messages between several applications. HTTP is light, easy to implement, well understood, and pretty much ubiquitous. Unfortunately, HTTP is not a message queueing service, and there are limitions, such as scalability and persistence,
We saw this as an opportunity to investigate, and potentially implement a new queueing mechanism at Points, and perhaps use it to replace QHub one day.
We had a number of requirements including:
- Java 1.4-compatible client library (not everything has a brand-new JVM)
- Capable/easy-to-use monitoring/admin features.
- Scalable architecture. Our actual needs were very modest, but it needed to be able to easily scale.
- Clustering capabilities.
- Persistence/Recovery capabilities.
- Open source is preferable to a proprietary product.
Matrix of Vendors to Features
We did some looking and benchmarking:
|candidate||Rabbit MQ||Apache QPid||ØMQ||Storm MQ||Open AMQ|
|client||Java 1.4||Java 1.4||Java 1.4||Java 1.6||Java 1.5|
|server||Erlang||Java 1.4||C++||Java 1.6||C|
|notes||clients include: NASA, Second Life, Reddit. owned by SpringSource||brokerless/ no server, FAST||cloud based|
Community activity was an interesting criteria that we hadn’t considered before. If we use a product (especially an open source one), it is important to have a community to help with the support of it.
RabbitMQ and Apache Qpid both meet the technical feature set defined above. Additionally, they both support AMPQ queueing protocol, so switching clients should be as simple as changing Jar files, classnames, and recompiling. StormMQ was not selected due to its fee-based model, when there are so many good free ones. Likewise, the ØMQ model/approach (while very interesting) doesn’t fit with our needs. Open AMQ doesn’t have our client library requirements and it does not have much community activity.
During evaluation, we learned about atomic reads (almost as exciting as it sounds), where a message is not removed from the queue until after the client sends an ack message to the queue, and we also asked a few questions to see how fast we got responses.
In the end, we selected RabbitMQ over Apache Qpid. We are moving towards our first production implementation in mid-June, and everything is looking good.
There are a few unresolved issues still to come:
- The server is written in Erlang!! While it’s cool to be able to say that we run Erlang applications, until someone starts learning Erlang, the code may as well be compiled, closed source code.
- Retry mechanism: If the message from the queue is not processed successfully due to things like database outage, then the message needs to be put back on the queue for later retry
- Priority queues: While not currently a requirement, there is a future need to (somehow) prioritise the contents of some queues.
None of these is serious, and we are looking forward to bringing RabbitMQ into Points!