Happy Birthday

Celebrating the first year of the newsletter and adding a paid subscription plan. Plus a how Redis Pub-Sub works.

Hi Friends,

Welcome to the 45th edition of the Polymathic Engineer newsletter. The issue of this week is a special one since I celebrate the first year of the newsletter.

Time flies when you enjoy what you do, and I loved every minute I spent writing this newsletter. This love also came from getting your fantastic feedback. In one year, this newsletter grew from 0 to 9K readers, and you have been saying great things about it.

My mission with the Polymathic Engineer was to explain complex technical topics in the same way I wished someone else explained them to me before.

I am convinced that many concepts in computer science are easier to grasp than they seem. The problem is that often, they are not explained clearly.

Indeed, explaining things in a simple and concise way takes a lot of effort. It requires giving more value to the time of your potential readers than to yours. So, I am happy that many of you appreciated my effort.

For all those reasons, I have decided to devote more time to the newsletter, adding a premium plan. I want to give the possibility to anyone who wants to support the work I am doing.

If you’ve found value in this newsletter, consider subscribing. It would mean a lot to me.

What’s changing?

For free subscribers, almost nothing. They will continue to receive a weekly newsletter issue.

Premium subscribers will receive additional newsletter issues in which I will delve deeper into selected topics.

The initial plan is to deliver to premium subscribers at least one issue every month. But of course, the frequency will increase according to the support I receive.

Here are some of the topics I already have in mind to treat:

• a deep dive into the Apache Cassandra database

• a bulletproof method to solve every dynamic programming problem

• how to design a push notification systems

• a primer on database replication

Of course, premium subscribers can get in touch with me and tell me which topics they are more interested in.

I'll send out the first post that is only for premium subscribers in two weeks.

What will it cost?

$59 per year, or $6/month.

Redis Pub-Sub

Redis is commonly considered a key-value server, but it can also be used as a messaging server using the Publisher-Subscriber pattern.

Redis Pub/Sub uses a channel data type to support publish and subscribe operations. Clients can use three commands: SUBSCRIBE to register to a channel, UNSUBSCRIBE to unregister from a channel, and PUBLISH to send messages through the channel.

When new content is published on a channel, Redis pushes it out to the subscribers of that channel. Publishers and subscribers are loosely coupled since they interact only through channels. A client can subscribe to multiple channels, and a channel can have 0 or multiple subscribers.

Redis uses an at-most-once delivery semantic to send messages to the subscribers.

This means that only (and all) currently connected subscribers get the messages. Once Redis delivers a message to all the subscribers, it deletes it from the channel.

So, the data going through a channel is stateless and is dropped if there are 0 subscribers. Disconnected subscribers or later subscribers won't get any copy of past messages.

Redis Pub-Sub internal functioning

Redis keeps track of channels and subscribers using hash tables as underlying data structures. Two hash tables are involved in the process and kept in synch.

A first hash table with chaining collision handling is used to map channels to subscribers. Subscribers are stored using a linked list for fast iteration during a publish.

A second hash table is used as a set to store all the channels a client subscribes to. Here, choosing a hash table over a linked list allows fast retrieval for subscribe and unsubscribe operations.

Publishing a message requires the following:

• hashing the channel name to get a hash chain

• Iterate the chain in case of conflicts

• Iterate the linked list of subscribers sending the message

Subscribe a channel requires:

• find the proper linked list as before

• append the subscriber to the list

• add the channel to the subscriber's hash table.

Unsubscribing a channel is similar but more expensive since removing a subscriber requires scanning the whole list.

Redis Pub-Sub Pros and Cons

Redis Pub/Sub is an efficient way to distribute messages, but it's important to know its pitfalls and strengths.

Even if Redis Pub/Sub may look similar to a topic-based system like Kafka, it has several limitations compared to a robust brokered messaging system:

• Messages are ephemeral without persistence guarantees

• Messages can be lost without delivery assurances

• Messages may be received out of order

These are its most appropriate use cases:

1. Real-time, low-latency messaging where messages are only relevant to subscribers for a short time (i.e., simple chat applications)

2. Unreliable delivery messaging where messages are lost is not critical

3. Cases where the number of subscribers and patterns per channel is relatively small

4. Subscribers require at-most-once delivery because target systems can't detect duplicates or are not idempotent

Interesting Tweets

There is absolutely nothing wrong with enjoying spending most of your working time coding or doing technical activities. Being a skilled individual contributor is beneficial to any business, and companies are recognizing this.

Traditionally, the path to advancement was through management. But that is not for everyone. Luckily, many companies are incorporating parallel IC tracks alongside managerial tracks.

I agree that this strategy works. But it only works if the great senior engineer is willing to become a manager. In my experience, it is never good to put someone as a manager if not entirely convinced.

Reading research papers is the best habit I kept as a software engineer after my PhD. My company grants me access to the main journals, and I enjoy spending a few hours per week reading papers.