Messaging - Part 3

A comparison of most popular message brokers: Amazon SQS, Azure Queues, RabbitMQ and ActiveMQ.

Hi Friends,

Welcome to the 93rd issue of the Polymathic Engineer newsletter.

This issue completes our deep dive into message queues by analyzing and comparing different implementations. The goal is to provide the information you need to choose the best one for your use case.

The outline will be as follows:

• The Evolution of Message Queues

• Amazon Simple Queue Service (SQS)

• Azure Storage Queues and Service Bus Queues

• RabbitMQ

• ActiveMQ

• A final comparison

The Evolution of Message Queues

As we've seen in previous posts, message queues have become a core part of any distributed system. They enable asynchronous communication and make the system more reliable.

However, message queues' role has changed over time, evolving from messaging middleware to platforms capable of handling real-time data streams.

IBM was the first company to work on a messaging system around 1990. Their first product, IBM MQ, already included most key elements, such as a queue manager that transferred data via channels. Messages were an abstraction for transmitting data, and their headers specified the routing, storage, and delivery.

Following IBM, other tech giants introduced their messaging solutions, such as Microsoft Message Queuing (MSMQ) in 1997 and Amazon Simple Queue Service (SQS) in 2004.

All these solutions worked well in their own ecosystems, but they could not be combined into a single computer bus because they had different APIs and protocols.

The first attempt to solve the problem was made in 1998 by the Java Message Service (JMS), which introduced standardization at the API level. The main drawback was that JMS could only be used in Java applications.

In 2003, financial institutions joined in developing a messaging protocol called Advanced Message Queuing Protocol (AMQP), which focused on wire-level standardization.

Rabbit Technologies' most famous implementation of AMQP, RabbitMQ, was released in 2007. Its design included ideas like exchanges and routing keys, which made message sending more flexible.

When LinkedIn released Kafka to the public in 2011, it changed the way messaging systems are designed. Optimized for writing and handling real-time data feeds, Kafka introduced concepts like topics and partitions and established new standards for high-throughput, low-latency event streaming.

In 2016, Yahoo introduced Pulsar, a platform that combines messaging and streaming capabilities. It incorporated many features from its predecessors but used a cloud-oriented architecture. For example, Pulsar natively lets you leverage cheaper object storage to persist messages for a longer term.

In the following sections, we will go into more detail about some traditional message queue implementations. Since streaming platforms like Kafka are a category on their own, we will discuss them in a separate newsletter.

Amazon Simple Queue Service (SQS)

Amazon SQS is a cloud-based service that stands out for how simple and pragmatic it is. The best thing about SQS is that Amazon takes care of everything, so you don't have to worry about setting it up, configuring it, or maintaining it. SQS can handle thousands of messages per second without planning or preparation. It can automatically scale based on your needs.