Hi Friends,
Welcome to the 190th issue of the Polymathic Engineer newsletter. This week, we continue our series of articles on latency.
At the end of Part I, we posed two open questions. If latency is a distribution, not a single number, where does all that variance come from? And how do we measure it? This issue answers both, starting with a tour of the entire stack.
The outline is as follows:
- Sources of latency
- How latency compounds
- Measuring without lying to yourself
- Hands-on: measure your own network’s tail
To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.
Sources of latency
Once you start looking under the hood, it is surprising how many places latency can hide. Parts of code that don’t seem to do anything can have a substantial impact on tail latency, yet the root causes often lie much deeper than the code itself. Let’s walk through all the possible factors so you know what to look for:
Physics. The first source of latency is the speed of light because data can’t travel faster than that. You can’t really get around the physical distance between parts of your system. Your application logic might run close to the user, but if it needs to access a database in a cloud region, every request still incurs a round-trip cost. If that database is in New York, only users on the US East Coast will experience low latency.
Hardware. CPU and hardware introduce variability due to the optimizations they utilize, and the tricky part is that your software has almost no direct control over them. The first one is the CPU cache hierarchy. Reading from memory can take up to 100 nanoseconds, which is why CPUs keep several levels of cache in front of it. Each level has different characteristics. The L1 cache is the smallest and fastest. It can respond in 1 to 2 nanoseconds. The last-level CPU cache takes 10 nanoseconds or more. If your working set doesn’t fit in L1, you start paying for cache misses since the CPU has to fetch data from a slower cache level or from DRAM. Then there’s speculative execution. Instead of waiting for one instruction to finish before starting the next, the CPU guesses what will be needed and starts executing it. As a classic example, think about what might happen when the CPU needs to execute an if-else branch. If the prediction of which way it will go is correct, you get a speedup, but if not, you get delays and wasted work. The last source is power saving. CPUs and hardware use different mechanisms to save energy. However, they can have a noticeable impact on tail latency. CPUs can scale their frequency up or down to cut down power consumption, so the same code may run more slowly depending on the current frequency.
Virtualization. Virtualization is inevitable when your application is running in the cloud. You can either use hardware-based virtualization with hypervisors or OS-based virtualization with containers. Hypervisors add overhead because they multiplex hardware, but there are other issues as well. When a physical machine is split into virtual machines, CPU, memory, and I/O must be shared, which is one of the main causes of latency spikes. Another application on the same host that you cannot see or control can use these resources, slowing you down. Virtualized networks and storage can add even more delay, since each I/O operation must be translated by the virtualization layer into a hardware operation.
Operating system. The OS provides abstractions that hide the hardware but can be a source of latency variance. Context switches allow many threads to share a single core, but they take microseconds and can evict the previous thread’s cache. Interrupts can stop your code at unpredictable times to service a network card or disk, which is why many latency-sensitive applications use polling. Device drivers add their own delays by queuing and batching requests, trading the latency of an individual request for better overall throughput. Also, the firmware that runs in network cards and storage can interrupt the CPU or delay an I/O request without the OS even being aware of it.
Managed runtimes. Languages like Java, JavaScript, and Python don’t run your code directly, and the runtime can stop the world at the worst time. Just-in-time compilation kicks in as the program runs, so an application humming along for minutes can suddenly stall while the JIT optimizes a hot path. Garbage collection does the same when it pauses to reclaim memory.
Application. Your own code is the place you can fully control and has the biggest potential for improvement. How you can improve it is application-specific and also depends on what latency target you’re aiming for.
How latency compounds
Knowing where latency comes from is only half the story. The other half is that these delays add up. Even small delays across different parts of the system can compound to create a real latency problem. This is why building a low-latency application means thinking about the whole picture. Two things matter: the components that make up latency, and the way they compound.