Hi Friends,
Welcome to the 189th issue of the Polymathic Engineer newsletter. In this week’s article, we start discussing latency in depth.
Latency is one of those things that every software engineer has to deal with, but few make the effort to really understand it. We all know that fast is good and slow is bad. But when we try to put some definition on what latency is, what limits it, and how to quantify it, things get a bit fuzzier. Since you can’t improve what you can’t define or measure, it’s important to know these basic concepts.
The outline is as follows:
What latency is
Units and the limits of physics
Why latency matters
The laws of latency
Latency is a distribution
To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.
What latency is
As a developer, you have likely talked about latency informally as lag or response time. That works great in everyday conversations, but if the goal is to optimize latency, we need a sharper definition. The one we will use here is: latency is the time delay between a cause and the moment we see its effect.
This definition tells us two things. First, there is an action we do and an outcome we see. Second, latency is the time between those two events. There are different ways to measure latency because what is considered cause and effect varies by situation.
This is something I began to notice after replacing my old wired doorbell with a smart one. With the old doorbell, after pressing the button, the chime rang right away. With the smart one, there is a visible delay between the button press and the notification on my phone. The reason is that the press goes over Wi-Fi to a hub, then to the vendor’s cloud, and only then back to my phone. The cause is pressing the button, the observed effect is the notification, and the latency is the period in between. If you play with one of these devices, you will also notice the delay is not always the same, a point we will come back to at the end of this article.
The exact same mechanics apply to a scenario closer to our daily work: a standard web request. When a user types a URL and presses Enter, they experience the end-to-end latency from a key press to page load. There are a lot of moving parts. The browser needs a DNS lookup before it can even send the request. The server might need to access a database to get the data needed to respond. The browser then has to render the page, which may trigger extra requests. Each step takes its own time, and the times add up.
One thing latency is not is throughput. We already compared latency, throughput, and bandwidth in a previous issue, so here it is enough to remember their differences: latency measures how long a single operation takes, while throughput measures how many operations complete per unit time. You can improve one without improving the other, and later in this article we will see a law that connects them.
Units and the limits of physics
We measure latency in units of time, but the scale changes wildly depending on where we look in the system. A read from a CPU register or cache takes about a nanosecond. Getting data from DRAM takes on the order of 100 nanoseconds, while reading from an SSD is already in the microseconds, about 100 of them. And as soon as the network is involved, we jump to milliseconds: a round trip from New York to London takes about 60 ms. This ladder spans about 7 orders of magnitude from top to bottom. Keeping a few of these numbers in mind is one of the most functional tools for back-of-the-envelope calculations.
But why does distance show up in the numbers at all? The reason is that latency has a physical lower bound. Information can’t move faster than the speed of light, around 300,000 km/sec. In practice, it is slower because the light in a fiber-optic cable doesn’t travel in a vacuum. No matter how good your code is, if the data has to cross an ocean, the laws of physics set the minimum duration of the trip.
It is challenging to grasp intuition for these small time spans. We can feel a second, but microseconds and nanoseconds are too short for our senses to detect. Grace Hopper, a pioneer of computing, had an ingenious approach to fix this. She asked her engineers to cut wires as long as the maximum distance electricity could cover in a nanosecond: about 30 centimeters. A microsecond was a coil of wire about 300 meters long. In her lectures, she handed out the nanosecond wires and the concept stuck: to achieve a latency target, sometimes machines must be nearby. If you haven’t watched her lecture, it’s worth your time.