Messaging - Part 2
Benefits and challenges of using a message queue. Plus a look into Amazon SQS and RabbitMQ works.
Hi Friends,
Welcome to the 90th issue of the Polymathic Engineer newsletter.
This week, we continue our discussion about message queues. In the first part, we have seen the main components of a message queue and how the exchange of messages works.
This time, we analyze the benefits of using a message queue and the related challenges. The outline will be as follows:
Benefits of using message queues
Challenges of using message queues
Message queues antipatterns
Benefits of using message queues
The example we used to introduce message queues in the first part showed the many benefits of using an intermediate messaging layer between producers and consumers.
We will discuss all these aspects in more detail here to understand why message queues are an indispensable tool in scalable system design.
Asynchronous and Reliable Processing
The main benefit of using message queues is enabling asynchronous processing. This capability allows your system to quickly acknowledge incoming requests and defer the execution of time-consuming tasks, becoming more responsive and performant.
A good example of such a scenario could be an applications processing video content for different user's needs. The frontend servers upload each video to a file store and publish a processing request to the message queue.
Backend servers pick up messages later and begin the video processing. Once done, they upload the video to the file store and publish a new message to a separate queue to indicate that work has been completed.
Since message queues persist the messages, the system becomes more reliable and resistant to failures. If one of the backend servers fails, messages are not lost but retained for later processing.
Scalability
Message queues allow you to scale producer and consumer services independently based on demand. For example, you can add more front-end servers to handle more user requests without immediately needing to scale your back-end capacity or process multiple requests in parallel on different back-end servers.
Moreover, message queues often support efficient batch processing, allowing consumers to fetch multiple messages with a single read request.
Even if batching increases the processing latency of single messages, it dramatically improves the overall throughput. This is a good trade-off tradeoff if high throughput is more important than low latency.
Load Spikes Handling
Another advantage of message queues is that they allow you to handle traffic and load spikes smoothly. They work as buffers where the incoming requests are queued, preventing the system from being overwhelmed.