Welcome to the 194th issue of the Polymathic Engineer.

Most of the code we (or our agents) write performs simple tasks, so it makes little sense to optimize them. However, some computations are too expensive to repeat if we don’t need to. Accessing a database is one of them. Calling an external service is another.

Let’s go through an example. Consider a price-comparison service. Given a product ID, the service calls the APIs of four online retailers, collects the prices, and returns the best offer. Each call goes outside your network and has its own latency. You can call the four retailers in parallel, but you need all four answers before you can decide, so your latency will be at least as high as the slowest of them.

This is where things get tricky. Suppose you promised your customers a service-level agreement (SLA) where 95% of the monthly calls return within 750 ms. Most of the time, this is not a problem. But during traffic peaks, a couple of retailers take as long as 3 s to answer. To make things worse, the load balancer kills every call that takes longer than 3.5 s. If you need around 250 ms to handle the incoming call and process the data, you risk having a lot of timeouts. And if you are a paid service, violating the SLA means giving money back.

Since the catalog is small (a few hundred products), the most natural thing to do is to compute the best offer once, store it, and look it up the next time somebody asks for the same product. After an initial warm-up period when every call has to actually compute values, most calls will be answered without hitting the retailers at all. This is precisely what a cache does.

We already discussed where to place a cache in a distributed system, which strategies to use for reads and writes, and how to handle stale data. Here, we focus on the cache itself as a data structure: how to build one so that storing, looking up, and evicting entries are all fast, no matter how many entries it holds.

The outline is as follows:

- The contract

- Memory alone is not enough

- Building an LRU cache

- When fresher data isn’t better: the LFU cache

- Code

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

The contract

As always, let’s start by defining the interface our data structure should offer. A cache stores a certain number of entries, established when the cache is created. It always allows you to add a new entry, and it decides which entries to keep based on its eviction policy. The following snippet shows the API:

class Cache:
    init(max_size)
    get(key)
    set(key, value)
    size()

When set is called with a key that already exists, the new value overwrites the old one. get returns the value stored for a key, or nothing if the key is not in the cache. size tells you how many entries are stored at the moment.

There is one thing we are sweeping under the rug: cached values can go stale. The best offer computed this morning is no longer true once the retailers update their prices. How to deal with stale data depends heavily on the context. In some cases, an old value can provide a good enough approximation if a certain margin of error is acceptable. Conversely, in a time-critical, safety-critical application, approximations and stale data are not an option (certainly not for more than a few seconds).

How can we quickly tell when data goes stale? Usually, you store a timestamp along with the value when you write it to cache and check it when you read it. If it’s too old to still be relevant, you can recompute and update the cache.

Handling stale data is an orthogonal task that enhances the base mechanism, and we covered time-to-live and invalidation in a previous issue. So, in the rest of the article, we assume that cached values don’t go stale.

Memory alone is not enough

Let’s stop and recap what we discussed so far. We have an expensive computation whose results can be reused, and we need a mechanism to remember those results so that we compute them only once. For our example, this is straightforward. There are at most a few hundred products, and for each, we store only the best offer. A very small amount of memory is enough.

What would happen, however, if we wanted to cache the raw listings retailers return, with descriptions, images, and shipping options? A popular product can have thousands of listings across sellers, and each one takes a few kilobytes. Even with a few hundred products, the cache could reach several gigabytes.

Or consider how a social media app like Facebook works. When you open your feed, an algorithm computes the best posts to show you, based on your mutuals, your preferences, and the users you follow. It’s a resource-consuming algorithm, so you want to cache its results as much as possible: there is no need to recompute the feed if the same user comes back after 5 minutes. Now consider this cache for a billion users. Even if you store just the top 50 posts per feed, and for each post just its ID, you still need hundreds of gigabytes.

These examples show how easily a cache can reach a size that is hard or impossible to keep in memory. You could move to a distributed cache, but even there, the more entries you store, the slower it gets to check the cache. And in any case, we can’t add entries forever. A cache is a finite system, and at some point, it will be full.

So, once the cache is full, adding a new entry means removing an existing one. The question is which one. The ideal answer is to remove entries that won’t be requested anymore, or at least those that will be requested less. Unfortunately, no computer can predict which elements we are going to need the most, so we can only make an educated guess.

One reasonable assumption is that an entry that hasn’t been accessed for a long time is worth less than one accessed recently. If a product hasn’t been requested in hours, it’s likely it won’t be needed soon, or ever again. But looking only at the time of last access throws away useful information. Suppose an entry was accessed hundreds of times, but not in the last few minutes. Every newer entry, even one accessed just once, would be considered more valuable. If you think about it, an entry accessed only once has been computed but never read from the cache, so caching it has brought no advantage yet. A different approach is to value entries based on how often they are accessed.

These assumptions give us two different caches. Let’s build them, starting with the first.

Building an LRU cache

The first cache we build is the LRU (least recently used) cache, which removes the least recently used entry every time it needs to make room. Before choosing the data structures, let’s list the operations we want to be fast:

Read more

Welcome to the 193rd issue of the Polymathic Engineer. This week, we are concluding our series of articles on API design.

So far, we have examined the rules and best practices for designing REST APIs. Then we transformed the API layout into a machine-readable file so that the CI/CD pipeline could verify compliance. Throughout this process, we assumed that our APIs relied exclusively on REST and JSON.

That assumption holds true for the Booking API we designed for an external partner. But the booking service also interacts with the payment service, the notification service, and the event catalog, making thousands of calls per second. Why would you want to pay the same cost per call on your own infrastructure as you would across the internet?

That is the question we will address today. The right API format depends on where the traffic comes from, and most real systems end up adopting more than one.

The outline is:

• What RPC really is

• North-south vs east-west traffic

• gRPC and Protocol Buffers

• When performance tips the scale

• Where GraphQL fits

• Can one service offer both?

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

What RPC really is

REST can provide a domain model and abstract away the underlying technology for the consumer, but it’s not the only way for services to communicate. Remote Procedure Calls (RPC) rely on a different but simple concept: you invoke what looks like a regular function in one process, but it runs in another process—usually on a different machine.

On the client side, the stub makes this work. When the caller invokes getBooking(42), it really calls generated code that packs the arguments into a message, sends it over the network, waits for the reply, and unpacks the result. The network is transparent to the caller. We dedicated an entire issue to how remote procedure calls work under the hood, so we won’t repeat the technicalities here.

The most used RPC framework nowadays is gRPC, initially created by Google and released as open source in 2015. Since 2017, the project has been hosted by the CNCF and is the de facto standard on most platforms.

The fundamental difference compared to REST lies not in the syntax, but in state management. As we saw in the first article, REST is stateless by definition. With RPC, state depends on the specific implementation. Clients and servers can maintain state across calls, which can enhance performance but makes routing and recovery more difficult. If a server container fails, its replacement doesn’t know where you left off.

The trade-off is an exchange that couples producer and consumer more tightly than a resource model does. But let's be honest: coupling is not always the enemy. Between two services owned by the same team, deployed and monitored together, it might be a price you are happy to pay for speed.

North-south vs east-west traffic

Ultimately, the API format you use depends on where the request comes from. Engineers split API traffic into two groups and named them according to how architecture diagrams are drawn.

North-south traffic describes requests originating from outside the ecosystem. It’s the partner calling the Booking API, or a mobile app hitting your public endpoints. East-west traffic describes requests originating within the ecosystem and flowing entirely inside your cluster between your own microservices. For example, the booking service asks the payment service to charge a credit card.

These two kinds of traffic live in completely different worlds and have extremely different performance characteristics.

In a north–south exchange, traffic crosses the internet. The internet adds significant latency—DNS resolutions, TLS handshakes, the user's network—and clients can be written in any language using any HTTP library. It is natural to choose a low barrier-to-entry format such as REST over JSON.

East-west exchanges stay in your infrastructure. Round-trip times are measured in fractions of a millisecond, and you control both ends of every connection. Here the priorities flip: there is no need for a low barrier to entry between two services you manage directly, whereas every microsecond of overhead matters.

The real killer here is the compounding effects of each service. North-south requests rarely map to a single internal call. For a partner request, the booking service might check availability, calculate the price, and load the user profile. A single request fans out into 5-10 internal exchanges. You will pay several times over for every millisecond you squander on an internal hop before the response makes it back to the consumer.

This lens is also a classic discriminator in system design interviews. Weak candidates usually pick one format and apply it everywhere. Stronger ones ask where the traffic comes from before choosing, and give a different answer for the public API than for the internal calls behind it.

The decision rule is therefore natural: north-south traffic values low coupling and simplicity of adoption; east-west traffic values efficiency.

gRPC and Protocol Buffers

gRPC is better suited than REST for east–west services, thanks to smaller data payloads and faster communication within the ecosystem. gRPC is schema-first. Before writing any code, you describe the API in a .proto file, including the messages being exchanged and the methods that can be invoked. The contract for our booking service looks like this:

message Booking {
    int32 id = 1;
    string displayName = 2;
    string event = 3;
}

service BookingService {
    rpc GetBooking(BookingRequest) returns (Booking);
}

Read more

Welcome to the 192nd issue of the Polymathic Engineer newsletter. This week's article continues our API design series.

Last time, we focused on the core rules for creating intuitive and easy-to-evolve APIs—things like proper resource modeling, adopting standards, and reporting errors customers can build on. We wrapped up by promising to show you how to automate all of these rules: the OpenAPI Specification (OAS).

The spec describes the API's layout (e.g., endpoints, authentication, parameters, and request and response schemas) in a single machine-readable file. The tooling built around it enforces the rules. Once the layout of an API becomes a file, you can generate code from it, validate traffic against it, and, best of all, diff it in your pipeline to catch a breaking change before any consumer sees it.

The outline is as follows:

• Why a shared spec matters

• Generating code from the spec

• Validating live traffic

• Examples and mocking

• Versioning an API

• Catching breaking changes in your pipeline

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

Why a shared spec matters

Let’s go back to the Booking API example. When the external partner's developers start integrating your APIs, they will ask a barrage of questions: What endpoints do you have? What does the payload look like? How do we handle auth?

They can’t read your code, and relying on a handwritten wiki is a bad idea. If someone changes the API and forgets to update the wiki page, your partner risks wasting hours building against a different layout.

You need a machine-readable source of truth that tooling can actually enforce. That's where the OpenAPI Specification comes in. A spec is a plain JSON or YAML file that includes every detail a consumer needs to call your API—from paths and request payloads to auth rules and code examples.

The following snippet shows a fragment describing the booking endpoint:

paths:
  /bookings/{id}:
    get:
      responses:
        '200':
          content:
            application/json:
              schema:
                type: object
                required: [id, displayName, event]
                properties:
                  id:
                    type: integer
                  displayName:
                    type: string
                  event:
                    type: string

Swagger was the original reference implementation of the OpenAPI Specification, but most tooling has now converged on using OpenAPI. Today, it’s what powers those interactive dev portals and those handy "Try it out" buttons on API marketplaces.

The caveat is that a spec defines an API’s layout, not its behavior. It guarantees that GET /bookings/42 returns an object with an id and a displayName. Not what happens when someone cancels a booking that was already canceled, or if a search yields a clean 200 OK with an empty array or a hard 404. Layout and behavior are two different parts of the consumer contract.

What to write first: the spec or the code? You can hand-craft a spec and generate server boilerplate from it, or do the reverse using annotations. What matters is that the spec and the code never evolve independently: one must be generated from the other. A spec maintained by hand next to hand-written code is the one that drifts.

Generating code from the spec

The real superpower of a specification is that it lets you automatically generate client-side code to consume the API. Anyone consuming the Booking API needs the exact same boilerplate to construct HTTP requests, serialize payloads, and parse responses.

Writing all this is tedious, and integration bugs often slip in. A misspelled field name or a wrong type may not show up until the first call at runtime. Since an OAS maps out the entire server blueprint (i.e., security, paths, parameters, and schemas), a generator can produce service objects that represent and invoke the API.

The OpenAPI Generator project supports a wide range of languages and toolchains. For example, in Java you can choose between Spring and JAX-RS; in TypeScript, you can use your favorite frontend framework. Third-party developers feed the generator with your YAML file and get a complete client SDK with typed methods for every endpoint.

You can also generate server implementation stubs, such as skeleton controllers with routes and parameter types, plus empty method bodies to fill in with logic. This is what makes the spec-first workflow from the previous section practical — the producer and the consumer agree on the layout, and generate their scaffolding from it.

The generated code acts as an automated sanity check for the integration over time. When the spec changes, regenerating the client turns structural mismatches into immediate compilation errors instead of runtime surprises. A renamed field stops the build immediately. It is annoying in the moment, but better than a parser silently returning null values in production.

Validating live traffic

You could think that if producer and consumer generate their code from the same spec, nothing can go wrong. But generation happens at build time, and the network is full of clients you don’t control. Some consumers hand-write their integration, some run an SDK generated from an outdated spec, and some traffic isn’t a well-meaning consumer at all.

Read more

Welcome to the 191st issue of the Polymathic Engineer newsletter. This week, we are starting a new series of articles on API design.

Let’s be honest: building an API is quite fast with modern frameworks. You can get a working endpoint up and running in minutes. The real nightmare starts a few months later, once consumers rely on the API. Then every name you picked and structure you chose becomes difficult to change or remove. Here, we will explore how to design APIs that are intuitive for consumers and easy for the managing team to evolve.

Whether you are designing APIs as part of your job or preparing for system design interviews, these are concepts you will use time and again.

The outline for this first article is as follows:

- API design and coupling

- What makes an API RESTful

- The Richardson Maturity Model

- Adopting an API standard

- Designing collections

- Error handling

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

API design and coupling

Much of the discussion surrounding API design gets bogged down in the weeds: “Should the field be called userName or displayName?” “Is PATCH better than PUT?” These topics are important, but they are details. They miss the real elephant in the room: coupling. The real challenge with API design is how strongly consumers are bound to the decisions of the development team.

Suppose that you’re building the booking service of an event platform. Your service owns the data and exposes a Booking API to two consumers: your company’s own legacy system, and a third-party partner reselling tickets on their own site.

The two consumers put different constraints on your design. Coordinating changes with the team managing the legacy system is straightforward: they work in the same office, and any migration can be planned together. In contrast, you have no control over the code of the external partner or their release schedule.

Yet, both consumers call the same API. Once this is published, you can refactor the backend logic as much as you want, but you can’t change the API itself.

The golden rule is to design the API from the outside in, prioritizing the consumer's perspective. An API should model what your service does, not how it implements it. If your internal architecture leaks into the interface, even the slightest backend modification risks becoming a breaking change. If, however, the architecture remains hidden, a database can be replaced or a service split into microservices without consumers even noticing.

This consumer-first mindset is also what sets apart great candidates in system design interviews. Anyone can list a few REST endpoints, but the engineers who get hired are those who start by asking who the consumers are and what level of coupling they can realistically sustain.

What makes an API RESTful

REST is the default architectural style for web APIs these days, but most developers throw around the term "RESTful" pretty loosely. Many so-called RESTful APIs are just HTTP endpoints spitting out JSON. We already compared the main API styles in a previous issue, and HTTP itself in another. Here, we focus on what the REST label actually demands.

REST models an exchange between a producer and a consumer, in which the producer exposes resources that the consumer can interact with. A resource is anything the service can name, such as a booking or a set of bookings. You use HTTP verbs to state your intent: GET to read, POST to create, PUT or PATCH to update, and DELETE to clear it out.

Here is what that looks like for a booking service:

http

GET /bookings/42

Accept: application/json

---

200 OK

Content-Type: application/json

{

    "id": 42,

    "displayName": "Jim",

    "event": "tech-conf-2026"

}

The URL names the resource, the method states the intent, and the status code informs the consumer how the operation went on the producer side. The exchange is also stateless: the producer doesn’t retain any state from previous requests, so the consumer must include everything needed to process each request. If a request depends on a prior one, it is the consumer’s job to carry that context forward.

The aspect worth exploring further is the meaning of the methods, because they hide the guarantees consumers rely on. A method is safe if it doesn’t change anything on the server. GET is safe: anyone can invoke it as often as they like without side effects. A method is idempotent if repeating it gives the same result as calling it once. PUT is idempotent because it replaces the entire resource with the payload you send. DELETE is also idempotent since deleting a booking twice leaves the system in the same state as deleting it once. POST is neither safe nor idempotent: each call makes a new resource.

You should now also be able to answer the question of when to use PATCH or PUT. Use PUT when you’re replacing a resource entirely. Use PATCH when you want to modify only some fields. The trade-off for PATCH's convenience is that idempotency is no longer guaranteed.

These properties are not just theoretical, since much of the web's machinery is built around them. For example, a cache can store the response to a GET request, and a client can retry a failed PUT without fear. If your API uses GET for writing or POST for reading, that machinery works against you, and the bugs it generates are painful to track down.

The Richardson Maturity Model

Not every API built on HTTP is equally "RESTful." Back in 2008, Leonard Richardson spent a lot of time reviewing real-world REST APIs and distilled his findings into a four-level model that classifies the extent to which those APIs were built from a REST perspective. The model is still the most practical heuristic to answer the question of how RESTful your API is:

• Level 0 just uses HTTP as a transport layer. You have a single URL, such as /bookingService, and every request goes there. Basically, this is a remote procedure call (RPC) service that happens to speak HTTP.

• Level 1 introduces the use of resources and the idea of modeling resources as URIs. Individual resources have unique URLs: /bookings/42 identifies a single booking, while /bookings identifies the entire collection.

• Level 2 brings in the verbs. GET /bookings/42 reads a booking; DELETE /bookings/42 removes it. APIs at level 2 can make the guarantees we discussed in the previous section.

• Level 3 involves navigable APIs, also known as HATEOAS. Each response contains the actions possible on the object returned from the server: the response to GET /bookings/42 would include links to update or cancel that booking.

In practice, level 3 is rarely a target. Although navigation is useful in flexible UI-style systems, it is not so valuable for interservice API calls. Using HATEOAS would only make the exchange more verbose, and it is commonly circumvented, as developers program against a specification of possible interactions that they read up front.

The best compromise for most APIs is level 2. It gives your consumers an understandable resource model, with appropriate actions available for it. In turn, this reduces coupling and keeps the backing service's details hidden.

Adopting an API standard

Reaching level 2 still leaves many everyday decisions open. “What is the best way to report errors?” “How do you avoid accidentally breaking backward compatibility?” “What is the best pagination strategy?” REST doesn’t say anything specific about any of this. Every team answers these questions slightly differently, and every consumer pays the price when integrating a new API.

The good news is that several companies open-sourced their internal API guidelines, and adopting one is the fastest way to shortcut design discussions. The Zalando RESTful API guidelines are a great starting point: actively maintained and organized as a list of MUST and SHOULD rules you can check a design against. Microsoft’s guidelines are another well-known reference. Whichever you pick, it’s important to adopt it early: aligning an API with a standard later means renaming things, and renaming things breaks your consumers' integrations.

A tangible example of how these guidelines might help you is the use of personal data in URLs. It might feel reasonable to look up bookings by email, such as /bookings?email=jim@example.com. Microsoft’s API guidelines warn against it: URLs end up in server logs, proxy caches, and browser histories, where nobody expects sensitive data to show up. The recommendation is to keep personal information in the body or in the headers, and identify resources with opaque IDs.

Naming deserves the same care. Guidelines come with a short list of standard names, but your domain needs its own common data dictionary. Is the person who books a “customer”, an “attendee”, or a “user”? Companies that provide consistency across their APIs are more likely to enable consumers to understand and connect responses.

Standards also cover other topics such as authentication, but we won’t repeat them here. We already dedicated two entire issues to those fundamentals.

Designing collections

Almost every API returns a collection of items, and the natural way to model the response is a raw JSON array:

[

    { "id": 42, "displayName": "Jim", "event": "tech-conf-2026" },

    { "id": 43, "displayName": "Sara", "event": "tech-conf-2026" }

]

This works, but the trouble starts when the collection grows. It’s impractical to return thousands of bookings in a single response, so you need pagination: returning partial results along with a pointer the consumer can follow to get the next set of results. But where does that pointer go? It can’t be stored in a raw array. To add it, you have to turn the array into an object, which breaks every consumer that parses the response.

The guidelines recommend wrapping collections in an object from day one:

{

    "value": [

        { "id": 42, "displayName": "Jim", "event": "tech-conf-2026" }

    ],

    "@nextLink": "/bookings?page=2"

}

This costs nothing when the collection is small, and you can add metadata and links later without touching what consumers already parse. We dedicated an entire issue to pagination strategies, so we won’t detail the technicalities here.

Filtering and searching follow the same logic. You don’t have to support every possible query from the start, but adopting a convention early lets the API evolve without breaking compatibility for consumers. A commonly used one is the OData syntax, used by Microsoft Graph in production: GET /bookings?$filter=event eq ‘tech-conf-2026’.

The rationale is always the same: leave APIs room to grow in predictable ways.

Error handling

Consumers build logic around your status codes, and it’s important to carefully define what should happen when things go wrong:

• 3xx status codes are for redirects: consumers follow them automatically, which lets you relocate resources

• 4xx should indicate a client-side error for which the content of the message field is extremely useful

• 5xx typically indicates a failure on the server side, and some client libraries will retry on these types of failures.

That logic works only if the codes are accurate. If some APIs return a 2xx with an error message hidden in the body, consumers end up writing parsing code that second-guesses every success.

Suppose the booking service charges for a ticket and answers with a 500. The consumer has no way to tell whether the payment went through, since the request may have failed before or after the charge. This is why accurate codes and idempotent retries go hand in hand. A well-designed API makes it safe to ask again.

The guidelines recommend consistency too. Ideally, a consumer should be able to handle all errors with a single code path, which is only possible if they have the same shape with a machine-readable code and a human-readable message:

{

    "error": {

        "code": "EVENT_SOLD_OUT",

        "message": "No seats left for tech-conf-2026."

    }

}

A specific case worth mentioning is security. Error messages sent back to an external consumer are publicly exposed, so they must not include stack traces or internal details. The details that help you debug are the exact same details an attacker aims to use to compromise the system. Keep the rich details in your logs, and send the consumer only what they can act on.

When a consumer sends more requests than you can handle, respond with a 429 and a Retry-After header indicating when to try again. Explicitly turning down requests is much better than slowing down for everybody.

Wrap Up

In this first article, we started with the observation that APIs are hard to change once published, and we have seen rules to address that concern: model resources at level 2, adopt a standard early, wrap your collections, and report errors consumers can build on.

None of them is rocket science. They just require making the correct decisions early, before the API ships and when changes are still cheap.

In the next article, we will look at the tool that automatically enforces these rules: the OpenAPI specification. We will also see how to catch a breaking change in your pipeline before any consumer sees it.

Welcome to the 190th issue of the Polymathic Engineer newsletter. This week, we continue our series of articles on latency.

At the end of Part I, we posed two open questions. If latency is a distribution, not a single number, where does all that variance come from? And how do we measure it? This issue answers both, starting with a tour of the entire stack.

The outline is as follows:

- Sources of latency

- How latency compounds

- Measuring without lying to yourself

- Hands-on: measure your own network’s tail

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

Sources of latency

Once you start looking under the hood, it is surprising how many places latency can hide. Parts of code that don’t seem to do anything can have a substantial impact on tail latency, yet the root causes often lie much deeper than the code itself. Let’s walk through all the possible factors so you know what to look for:

• Physics. The first source of latency is the speed of light because data can’t travel faster than that. You can’t really get around the physical distance between parts of your system. Your application logic might run close to the user, but if it needs to access a database in a cloud region, every request still incurs a round-trip cost. If that database is in New York, only users on the US East Coast will experience low latency.

• Hardware. CPU and hardware introduce variability due to the optimizations they utilize, and the tricky part is that your software has almost no direct control over them. The first one is the CPU cache hierarchy. Reading from memory can take up to 100 nanoseconds, which is why CPUs keep several levels of cache in front of it. Each level has different characteristics. The L1 cache is the smallest and fastest. It can respond in 1 to 2 nanoseconds. The last-level CPU cache takes 10 nanoseconds or more. If your working set doesn’t fit in L1, you start paying for cache misses since the CPU has to fetch data from a slower cache level or from DRAM. Then there’s speculative execution. Instead of waiting for one instruction to finish before starting the next, the CPU guesses what will be needed and starts executing it. As a classic example, think about what might happen when the CPU needs to execute an if-else branch. If the prediction of which way it will go is correct, you get a speedup, but if not, you get delays and wasted work. The last source is power saving. CPUs and hardware use different mechanisms to save energy. However, they can have a noticeable impact on tail latency. CPUs can scale their frequency up or down to cut down power consumption, so the same code may run more slowly depending on the current frequency.

  

• Virtualization. Virtualization is inevitable when your application is running in the cloud. You can either use hardware-based virtualization with hypervisors or OS-based virtualization with containers. Hypervisors add overhead because they multiplex hardware, but there are other issues as well. When a physical machine is split into virtual machines, CPU, memory, and I/O must be shared, which is one of the main causes of latency spikes. Another application on the same host that you cannot see or control can use these resources, slowing you down. Virtualized networks and storage can add even more delay, since each I/O operation must be translated by the virtualization layer into a hardware operation.

• Operating system. The OS provides abstractions that hide the hardware but can be a source of latency variance. Context switches allow many threads to share a single core, but they take microseconds and can evict the previous thread’s cache. Interrupts can stop your code at unpredictable times to service a network card or disk, which is why many latency-sensitive applications use polling. Device drivers add their own delays by queuing and batching requests, trading the latency of an individual request for better overall throughput. Also, the firmware that runs in network cards and storage can interrupt the CPU or delay an I/O request without the OS even being aware of it.

• Managed runtimes. Languages like Java, JavaScript, and Python don’t run your code directly, and the runtime can stop the world at the worst time. Just-in-time compilation kicks in as the program runs, so an application humming along for minutes can suddenly stall while the JIT optimizes a hot path. Garbage collection does the same when it pauses to reclaim memory.

• Application. Your own code is the place you can fully control and has the biggest potential for improvement. How you can improve it is application-specific and also depends on what latency target you’re aiming for.

How latency compounds

Knowing where latency comes from is only half the story. The other half is that these delays add up. Even small delays across different parts of the system can compound to create a real latency problem. This is why building a low-latency application means thinking about the whole picture. Two things matter: the components that make up latency, and the way they compound.

Read more

Welcome to the 189th issue of the Polymathic Engineer newsletter. In this week’s article, we start discussing latency in depth.

Latency is one of those things that every software engineer has to deal with, but few make the effort to really understand it. We all know that fast is good and slow is bad. But when we try to put some definition on what latency is, what limits it, and how to quantify it, things get a bit fuzzier. Since you can’t improve what you can’t define or measure, it’s important to know these basic concepts.

The outline is as follows:

• What latency is

• Units and the limits of physics

• Why latency matters

• The laws of latency

• Latency is a distribution

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

What latency is

As a developer, you have likely talked about latency informally as lag or response time. That works great in everyday conversations, but if the goal is to optimize latency, we need a sharper definition. The one we will use here is: latency is the time delay between a cause and the moment we see its effect.

This definition tells us two things. First, there is an action we do and an outcome we see. Second, latency is the time between those two events. There are different ways to measure latency because what is considered cause and effect varies by situation.

This is something I began to notice after replacing my old wired doorbell with a smart one. With the old doorbell, after pressing the button, the chime rang right away. With the smart one, there is a visible delay between the button press and the notification on my phone. The reason is that the press goes over Wi-Fi to a hub, then to the vendor’s cloud, and only then back to my phone. The cause is pressing the button, the observed effect is the notification, and the latency is the period in between. If you play with one of these devices, you will also notice the delay is not always the same, a point we will come back to at the end of this article.

The exact same mechanics apply to a scenario closer to our daily work: a standard web request. When a user types a URL and presses Enter, they experience the end-to-end latency from a key press to page load. There are a lot of moving parts. The browser needs a DNS lookup before it can even send the request. The server might need to access a database to get the data needed to respond. The browser then has to render the page, which may trigger extra requests. Each step takes its own time, and the times add up.

One thing latency is not is throughput. We already compared latency, throughput, and bandwidth in a previous issue, so here it is enough to remember their differences: latency measures how long a single operation takes, while throughput measures how many operations complete per unit time. You can improve one without improving the other, and later in this article we will see a law that connects them.

Units and the limits of physics

We measure latency in units of time, but the scale changes wildly depending on where we look in the system. A read from a CPU register or cache takes about a nanosecond. Getting data from DRAM takes on the order of 100 nanoseconds, while reading from an SSD is already in the microseconds, about 100 of them. And as soon as the network is involved, we jump to milliseconds: a round trip from New York to London takes about 60 ms. This ladder spans about 7 orders of magnitude from top to bottom. Keeping a few of these numbers in mind is one of the most functional tools for back-of-the-envelope calculations.

But why does distance show up in the numbers at all? The reason is that latency has a physical lower bound. Information can’t move faster than the speed of light, around 300,000 km/sec. In practice, it is slower because the light in a fiber-optic cable doesn’t travel in a vacuum. No matter how good your code is, if the data has to cross an ocean, the laws of physics set the minimum duration of the trip.

It is challenging to grasp intuition for these small time spans. We can feel a second, but microseconds and nanoseconds are too short for our senses to detect. Grace Hopper, a pioneer of computing, had an ingenious approach to fix this. She asked her engineers to cut wires as long as the maximum distance electricity could cover in a nanosecond: about 30 centimeters. A microsecond was a coil of wire about 300 meters long. In her lectures, she handed out the nanosecond wires and the concept stuck: to achieve a latency target, sometimes machines must be nearby. If you haven’t watched her lecture, it’s worth your time.

Why latency matters

Read more

Hi Friends,

Welcome to the 188th issue of the Polymathic Engineer newsletter.

Errors and exceptions are inevitable. Every code can fail when something unexpected happens, or when it runs in an environment it wasn’t designed for. Many unexpected things can occur like running out of memory errors, and so on.

That’s why handling failures should be one of the first things we think about when designing our code. However, the dangerous failures are not the ones that crash our application. They are the silent ones, where an exception gets dropped and the system keeps running in a bad state. Such problems are the toughest to discover and to fix.

In this issue, we’ll look at the patterns and anti-patterns of exception handling. The code examples are in C#, but the ideas apply to most languages.

The outline is as follows:

• A quick map of exceptions

• How wide should your catch be?

• Designing failures into your API

• The anti-patterns that hide failures

• Cleaning up when things go wrong

• Exceptions at the boundary: third-party code

• When failures go async

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

A quick map of exceptions

Every C# exception is an object. It has a message describing the problem and might wrap an inner exception. What is more important, it contains the stack trace. This is an array of elements, each identifying a line of code in a specific class that leads to an exception. It is essential diagnostic information.

Not all exceptions need to be handled. If an exception signals a critical runtime error (e.g., OutOfMemoryException or StackOverflowException), we can do little.

We care about exceptions that signal problems in the code or its environment: for example, an IOException when a file operation fails, or an InvalidOperationException if an object gets used in the wrong state.

How wide should the catch be?

A method that exports reports to disk might throw an IOException if there are file system issues, or an OperationCanceledException if the user decides to cancel the operation. The most granular option for the caller is to use a separate catch block for each exception type; this is the correct approach if you want to respond differently based on the error type. In the following example, only a disk error is an event that should be recorded in the error logs:

try
{
    ExportReport(path);  // write to a file
}
catch (IOException e)
{
    logger.LogError(e, “Could not write the report”);
}
catch (OperationCanceledException e)
{
    logger.LogInformation(”The export was canceled by the user”);
}

Another approach is to catch a more generic exception. FileNotFoundException and DirectoryNotFoundException are IOExceptions, so they can be caught together in a single catch block. We can treat both the same way, at the price that the code no longer indicates to the reader which errors we expected.

By taking the concept to the extreme, we can catch everything:

try
{
    ExportReport(path);
}
catch (Exception e)
{
    logger.LogError(e, “Something went wrong”);
}

Now if a bug in our function throws a NullReferenceException, this block swallows it together with the actual I/O failures, and a coding error is logged as a normal disk problem. An exception that ought to be propagated and resolved ends up buried in a log file instead. This is the first way in which error information can be lost.

As many other languages, C# also offers an intermediate solution known as exception filters. A when clause allows you to refine a catch block without having to duplicate it:

try
{
    
(path);
}
catch (IOException e) when (e is FileNotFoundException or DirectoryNotFoundException)
{
    logger.LogError(e, “The report path is not valid”);
}

If the filter does not match, the exception keeps propagating as if this catch block didn’t exist. We handle only those cases we are actually equipped to deal with. The general rule is to catch exceptions as specifically as possible, based on handling requirements. You should broaden the scope only for failures you genuinely treat the same way.

Designing failures into your API

When developing a component, it is likely that someone else will call it, perhaps a colleague working on another part of the system or an unknown user in the case of a library. In Java, the contract is partially enforced by the compiler: a method declares its checked exceptions, and the caller is required to handle them. In C#, however, the method signature provides no information about potential errors. The contract must be expressed elsewhere through guard clauses, the choice of exception types, and documentation.

A good practice is to validate arguments and state at the beginning of a method. For a task scheduler with a StartWorkers() method, we could write something like this:

public void StartWorkers(int workerCount)
{
    if (workerCount <= 0)
    {
        throw new ArgumentOutOfRangeException(
            nameof(workerCount), “The number of workers must be positive.”);
    }
    if (running)
    {
        throw new InvalidOperationException(
            “The scheduler is already running.”);
    }
    // start the workers...
}

These guard clauses follow an established convention: the ArgumentException family is reserved for invalid inputs provided by the caller, while InvalidOperationException is for the object being used in an invalid state. It is important to understand that such exceptions are not intended to be caught; they signal a bug in the calling code, and attempting to recover normal operation in the face of a bug makes little sense. It is better to fail fast and loud, bringing the issue to light during development rather than letting it show up in production more subtly.

All failures that are not bugs demand a different treatment. If the scheduler can fail due to a missing configuration file, the caller is actually in a position to respond; consequently, they should be informed of this possibility.

The closest thing C# has to a contract is the XML documentation comment: adding /// <exception cref=”IOException”> at the top of the method ensures that the potential failure appears in IntelliSense for anyone calling that method. It’s a weaker guarantee than a checked exception (nothing forces a developer to read it), but it remains the mechanism provided by the language. This practice should be reserved exclusively for public interfaces. Private methods do not require documented error contracts, as you control both ends of the call, and the extra verbosity adds no real advantage.

The anti-patterns that hide failures

Now that the API signals problems explicitly, callers need to react appropriately, and this is where things typically go wrong. Let’s discuss the most common anti-patterns, all variations of the same mistake: losing information about a failure.

The first and worst one is swallowing the exception:

try
{
    ExportReport(path);
}
catch (Exception e)
{
    // this cannot happen
}

It is often tempting to analyze the above code and conclude that such an exception cannot be thrown under any circumstances. Perhaps that might even be true for the time when the code is analyzed. However, the underlying behavior may change in the future. Even if it does not throw an exception when we are writing our caller code, the implementation might change over time. In that case, the exception that is swallowed never propagates up in the call stack. The system continues to operate in a bad state without anyone noticing. Those are some of the hardest problems to debug.

Read more

Welcome to the 187th issue of the Polymathic Engineer newsletter.

This week, we continue our series of articles on machine learning. In a recent article, we addressed the perceptron, the oldest and simplest classification model. We saw how it can count the words in a review, compute a score, and determine whether the review is “happy” or “angry” based on the sign of the score.

At the end of that article, we observed that the perceptron has one limitation: it can only answer a hard yes/no. A review with a score of 0.1 and a review with a score of 100 are both labeled as happy. But the two cases are distinct. The first review is right on the happy/angry edge, and the model is just guessing. The second one is deep in the happy zone, and there is no doubt about it.

We usually want a model that can tell us how confident it is. A spam filter that informs us an email is spam with 99% probability is far more valuable than one that simply tells us it is spam. We gain enough confidence to do the right thing: delete the email immediately or move it to a folder to look at it again later.

This is exactly what logistic regression does. Instead of predicting a specific label, it gives a number between 0 and 1 that we can interpret as a probability. The nice thing is that very little changes with respect to the perceptron: weights, bias, and score all stay the same. All we do is swap one function at the end.

Logistic regression is worth knowing well for two reasons. It is one of the most widely used classification models in practice, and its underlying function is present in every neural network. Understanding it makes deep learning much easier to grasp.

The outline is as follows:

• From the Step Function to the Sigmoid

• Scoring Reviews with Probabilities

• The Log Loss

• Comparing two classifiers

• The Logistic Trick

• The Logistic Regression Algorithm

• More Than Two Classes: The Softmax

• Putting Everything Into Practice

To learn technical skills, you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

From the Step Function to the Sigmoid

The perceptron makes its prediction using a two-step process. First, it computes a review’s score as the weighted sum of the counts of words plus the bias. Then it applies the step function, which returns 1 if the value is 0 or higher, and 0 otherwise.

The score already contains the confidence information we want, but this information is lost in the step function, which maps every positive score to the same value of 1. So the aim is to keep the score and replace the step function with a function that preserves the information. This is how such a function should work:

• If the score is a large positive number, the output is close to 1.

• If the score is a large negative number, the output is close to 0.

• If the score is zero, the output is 0.5, meaning the model can’t decide.

Many functions behave this way, but the standard choice is the sigmoid function:

sigmoid(x) = 1 / (1 + e^(-x))

The formula matters less than what the function does: it takes the whole number line and squashes it into the interval between 0 and 1. A few values are enough to see the pattern:

• sigmoid(-5) = 0.007

• sigmoid(-1) = 0.269

• sigmoid(0) = 0.5

• sigmoid(1) = 0.731

• sigmoid(5) = 0.993

The prediction of a logistic regression model is the sigmoid applied to the score:

prediction = sigmoid(w1 × x1 + w2 × x2 + bias)

From a geometrical point of view, the boundary line is still there. Reviews on the line score 0, so they get a prediction of 0.5: the model can’t tell if they are happy or angry. The deeper a review goes into the happy zone, the closer its prediction gets to 1; the deeper it goes into the angry zone, the closer it gets to 0. No review ever gets exactly 1 or 0, so the model is never 100% certain about anything.

In the perceptron article, we said that the step function is an activation function. The sigmoid is the second activation function we meet, and by far the most important one historically: it powered neural networks for decades.

The sigmoid has one additional advantage over the step function, which is clear when we train the model. The step function is flat everywhere except at zero, so the error changes in large steps and provides no indication of which direction is better. The sigmoid is instead smooth and curved, so a minor change in weights always produces a small change in predictions. This is what makes gradient descent work.

A quick note about the name, since it is a bit peculiar. You might wonder why the model is called logistic regression even if it solves classification problems. There are two reasons for that. First, the sigmoid is also known as the logistic function. Second, the model technically gives a number, the probability, rather than a category.

If we want a category, and not a probability, the rule is simple: happy if the prediction is 0.5 or more, angry otherwise. This rule always agrees with the perceptron because scores of 0 or more lead to predictions of 0.5 or more. The line of the boundary doesn’t move; what changes is the confidence on either side.

Scoring Reviews with Probabilities

Let’s see the sigmoid in action on the 4 reviews from the first country. As a classifier, we use the one we built by hand in the perceptron article: a weight of +1 for taka and -1 for bruk, with a bias of 0. A review’s score is the number of takas minus the number of bruks, and the prediction is the sigmoid of that score:

• “Taka taka taka!” at (3, 0): score 3, prediction sigmoid(3) = 0.953

• “Bruk bruk bruk!” at (0, 3): score -3, prediction sigmoid(-3) = 0.047

• “Taka taka bruk!” at (2, 1): score 1, prediction sigmoid(1) = 0.731

• “Bruk taka bruk!” at (1, 2): score -1, prediction sigmoid(-1) = 0.269

The two happy reviews have predictions above 0.5, and the two angry reviews have predictions below 0.5. All 4 reviews end up in the correct class, exactly like with the perceptron. However, the numbers now tell us more. The model is more confident about "Taka taka taka!" than about "Taka taka bruk!", which is consistent with the intuition: three takas and not a single bruk make a stronger case than a mixed review.

There is another way of thinking about these numbers, which will be important in a moment. The 0.731 output suggests that the model believes the review is happy with 73% probability and angry with 27% probability. Every prediction has both pieces of information. The prediction is the probability of a happy review, while the probability of an angry review is 1 minus the prediction.

Last, the new review. "Taka bruk taka taka!" lands at (3, 1), with a score of 3 - 1 = 2 and a prediction of sigmoid(2) = 0.881. The model in the perceptron article simply returned “happy”. The model now answers that it’s happy with 88% probability. Same guess, but this time we know how much to trust it.

The Log Loss

We now have a model that provides probabilities, but we still need a way to assess its performance. In the perceptron article, we built an error function that was large for a bad classifier and small for a good one, then let the machine minimize it. We need the same thing here, and it takes a couple of tries to get right.

Read more

Welcome to the 186th issue of the Polymathic Engineer newsletter. This week we look at a data structure that most engineers underestimate the first time they encounter it: the disjoint set, also known as union-find.

The reason why this data structure is often overlooked is that the problem it wants to solve looks very simple. Just keep track of which group each item is in and merge the groups when necessary. However, going for the brute force solution without thinking twice is a trap, because you don’t notice how slow this is until your inputs get bigger.

Let’s make it concrete. Imagine you are building a social app. People send and accept friend requests, and you want to answer two questions quickly: are these two people part of the same circle of friends, and how do you merge the two circles into one when a new friendship forms? At the start, everyone stays in their own circle of size one. As connections form, circles join together, and they can only ever grow: once two circles merge, they stay merged. You never split them back apart.

This is the disjoint set problem in a nutshell. You have a lot of items that are split into separate groups that don’t have anything in common, and you need to join the groups on the go while asking yourself, “Are these two in the same group?”

This problem comes up a lot more often than you might think: connected components in a graph, image segmentation, minimum spanning trees, and clustering in machine learning all lean on it. In this article, we will cover all that you need to know:

• What a disjoint set is

• How to represent it in memory

• The naive implementations: quick find and quick union

• Union by size

• Path compression

• Applications

To learn technical skills you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

What a disjoint set is

Let’s start with the name. Disjoint sets are groups of items such that no item is shared between any two groups. Each item is in one group, and the groups never overlap. In the social app example, each group of friends makes up a circle, and nobody can be in more than one circle at the same time.

Each group needs a way to identify itself, so we pick one of its members to behave as the representative of the whole group. You can think of it like the group’s label: every item in the group points back, directly or indirectly, to the same representative. Two items are in the same group if and only if they share the same representative. This is the principle that makes everything work, and we’ll see why when we look at how the structure is stored.

The data structure goes by two names. Some people call it a disjoint set, others call it union-find. That second name gives away the two operations it’s built around:

• Find(x): return the representative of the group that x is in.

• Union(x, y): join the groups that x and y are in into a single one.

On top of these two, it is useful to have a third operation, which tells us whether two objects are already together:

• Same(x, y): check if x and y are in the same group.

Same is essentially just a small wrapper around Find. We run Find on both items and then compare the two representatives. If they match, the items are in the same group. Otherwise, they are in different groups. We’ll keep reusing this idea, because most of the time we don’t care who the representative actually is. We just worry whether two items land on the same one.

One last thing worth saying is that the “is connected to” relation must act sensibly if this grouping is to make any sense. Each item is connected to itself. If A is connected to B, then B is connected to A. And if A is linked to B and B is connected to C, then A is connected to C. That last property is the important one: it’s what lets us merge two circles and think of each member of both as connected, even the pairs that were never directly linked. The data structure doesn’t define this relation for you. It just assumes it behaves this way and takes care of the bookkeeping.

How to represent it in memory

Once we know which operations we need, the next question is how to store everything.

The first thought that comes to mind is to maintain an actual list for each group and a map saying, for each item, which list it is part of. This works, but it has two problems. The first is that you have two structures to keep in sync by hand every time something changes, which is error-prone. The second is related to performance. When you merge two groups, you have to walk through one of the lists and update the mapping of every item in it. So a single merging costs linear time.

We can do much better if we structure each group in the form of a tree, where the root is the representative of the group. However, instead of using classical trees with nodes and link to child nodes, it’s more convenient to use an array. The underlying idea is to give each item an index between 0 and n-1. If your items aren’t integers in that range, you can always map them to an index using a hash table.

So, we use a single array of size n, which we can call parent. The reason of the name is that for each item the array only stores the index of its parent in the group’s tree, and not the full group. The representative of a group is that special item which is its own parent (parent[i] == i). To get from any item to its representative, we follow the parent links upward until we reach an item that points to itself.

In the beginning, each item is in its own group of size one, and therefore each item is its own representative. That means we initialize the array so that every position points to itself: parent[i] = i for all i. In Python, the whole setup fits in the constructor:

class DisjointSet:
    def __init__(self, n):
        self.parent = list(range(n))

That one line sets parent[0] = 0, parent[1] = 1, and so on. We now have n groups, each containing a single item. Every operation we build from here is just based on reading and rewriting entries in this one array.

The naive implementations: quick find and quick union

With the array in place, we have two natural ways to use it. They are at opposite ends of the same trade-off, so it is worth seeing both.

The first method is quick find, which keeps things flat: instead of pointing to a parent somewhere in a tree, every item points directly to its representative. This means that parent[x] is always the representative of x’s group.

Find becomes trivial, since we just read the array and return what is there in constant time:

def find(self, x):
    return self.parent[x]

Same is just as cheap, because it only compares two Finds.

The cost shows up in Union. To merge x’s group into y’s, we keep one representative, say the one for x, and then go through the whole array to relabel every item that was part of y’s group:

Read more

Welcome to the 185th issue of the Polymathic Engineer newsletter. This week we conclude our series of articles on the Internet Protocol and addressing.

In Part I, we stayed inside a single network and learned how an address and a mask work together. In Part II, we extended our discussion to the whole Internet and saw how CIDR, NAT, and DHCP kept IPv4 alive. In both parts, we have considered the packet itself as a black box: something with a source address and a destination address that somehow finds its way.

This week, we open the box. A packet is an specific string of bits, and each bit is there for a reason. We will look at the IPv4 header field by field, and discuss what happens when a packet is too big for the phyisical link it has to go through. We will also find out why both TCP and IP have a checksum, which is something that a lot of engineers do not understand. At the end, we will look at how IPv6 redesigned the whole thing. This is the best way to get a sense of which parts of IPv4 were good ideas and which parts were not.

The outline is as follows:

• The IPv4 datagram format

• Fragmentation and MTU

• Why a checksum at two layers?

• IPv6: what they fixed

To learn technical skills you must work on real projects. CodeCrafters is a great platform for that. You can build your own Redis, Kafka, DNS server, SQLite, HTTP server, or Git from scratch using your chosen programming language.

The IPv4 Datagram Format

At the network layer, a packet is called a datagram. An IPv4 datagram has two parts: a header, which is usually 20 bytes, and a payload, which carries the actual data. We are interested in the header because it is the protocol. Every rule we have talked about so far in this series can be found in one of these fields.

Let’s go through the most important fields of the IP header:

• Version. The first 4 bits say which version of IP the datagram uses. A router reads this field first, because it determines how to interpret everything that follows. For IPv4, the value is 4.

• Header length. The header can contain optional fields, so its size is not fixed. These 4 bits tell the receiver where the header ends and the payload begins. The typical value corresponds to 20 bytes.

• Datagram length. The total size of the datagram, header plus payload, in bytes. The field is 16 bits, so a datagram can in theory be 65,535 bytes long. In practice, datagrams are rarely bigger than 1,500 bytes, for a reason we will see in the next section.

• Identifier, flags, fragmentation offset. These three fields have been introduced to handle fragmentation, when a packet must be split into smaller pieces along the way. They deserve their own section, and they will get one shortly.

• Time-to-live (TTL). A counter that prevents a datagram from remaining on the network forever, for example when routers end up in a routing loop. Every router that processes the datagram decreases the TTL by one, and when it reaches 0, the datagram is dropped.

• Protocol. This field is a code that indicates which transport protocol the payload belongs to: 6 means TCP, 17 means UDP. It is only used at the final destination, to hand the payload to the right protocol. It plays the same role between the network and transport layers that the port number plays between the transport layer and applications.

• Header checksum. A small value that lets a router detect bit errors in the header. Routers typically drop a datagram whose checksum doesn’t match. Notice that since the TTL changes at every hop, the checksum must be recalculated at every hop too.

• Source and destination addresses. The two fields we spent Parts I and II on. The source inserts its own address in the first and the address of the final destination, usually obtained through a DNS lookup, in the second.

• Options. Rarely used extensions to the header. They sound harmless, but they make the header variable in length, so a router can’t know in advance where the payload starts, and processing time becomes unpredictable. Keep them in mind, because IPv6 makes a pointed decision about them.

• Payload. The data itself, which is normally a TCP or UDP segment, but can carry other things as well, such as ICMP error messages.

One number worth remembering: with a 20-byte IP header and a 20-byte TCP header, every packet you send carries 40 bytes of headers before a single byte of your data.

Fragmentation and MTU

Not every link-layer technology can transport packets of the same size. The maximum transmission unit (MTU) is the largest amount of data that a frame can hold. Ethernet, the most common technology, can carry 1,500 bytes, while some older wide-area links stop at 576. Since every datagram has to travel inside a link-layer frame, the MTU is a hard upper bound on the size of the datagram. In the previous section we mentioned that datagrams are rarely larger than 1,500 bytes exactly because that is how much an Ethernet frame can carry.

The problem is that a packet may traverse many links, each with a possibly different MTU. Let’s imagine a router has to forward a 4,000-byte datagram on a link with an MTU of 1,500 bytes. The datagram simply doesn’t fit. The solution is to break the payload into pieces, then send each piece as a smaller datagram. Each of these bits is termed a fragment.

The fragments must be put back together before the data reaches the transport layer, since TCP and UDP want to receive complete segments. The designers of IPv4 had to decide where that reassembly should take place, and they chose the end host, not the routers. Reassembling fragments in the middle of the network would force routers to hold state and wait for pieces, slowing everything down. The principle of allowing the destination to do the work, and keeping the network core simple, appears throughout the whole design of the Internet.

This is where the identifier, flags, and offset fields from the previous section are used. Every datagram gets an identifier from its sender, and when a router fragments it, all the fragments keep the identifier of the original. The destination uses it to recognize which fragments belong together. The offset tells us where each fragment’s data is in the original payload so we can put the parts back together and spot any gaps. The flag bit distinguishes the last fragment, which carries 0, from all the others, which carry 1; without it, the destination could never be sure the sequence was complete.

Let’s take a numerical example. Suppose we have a 4,000-byte datagram with 20 bytes of header and 3,980 bytes of data. It must be transmitted over a link with an MTU of 1,500 bytes. Each fragment can carry up to 1,480 bytes of data, because its own 20-byte header takes up the rest. The router then breaks the datagram into three fragments: two fragments of 1480 bytes each and a third fragment comprising the remaining 1020 bytes.

Each fragment becomes a complete IP datagram, with its own header, and travels over the link independently. If one of them gets lost, the destination cannot reconstruct the original datagram. This fragility is one of the reasons why most people don’t consider fragmentation as a feature, and is one of the things IPv6 got rid of, as we will see.

Why a Checksum at Two Layers?

The header checksum raises a question that comes up often in networking interviews: since TCP already has a checksum, why does IP compute a checksum too? Isn’t that wasted work?

The first part of the answer is that the two checksums don’t cover the same bits. The IP checksum protects only the IP header, the 20 bytes we already discussed. The TCP or UDP checksum instead covers the whole segment, including the header and the data. The network layer only checks the information it needs to do its own job, such as the destination address and the TTL, and lets the layer above it ensure the data is correct. A router doesn’t care whether your payload is intact; it cares only about not sending a packet whose destination address got corrupted.

The second part of the answer is that layers needs to be kept separated from each other. TCP and IP are used together most of the time, but nothing requires it. In principle, TCP can run on top of a different network protocol, and IP can carry payloads that are not TCP or UDP at all, such as the ICMP messages we mentioned earlier. Each level must be able to protect its own data without guessing anything about the layers around it.

There is also a practical reason why the IP checksum stays small. The TTL goes down at every hop, so the checksum needs to be calculated again. While checking 20 bytes at every router is cheap, checking the whole packet at every router would not be so. This way, the routers check only the modifications on the way while the end hosts check the full data once.

IPv6: What They Fixed

IPv6 is a new format that was designed since there were not enough IP addresses on the internet. The main change is the address size. An IPv6 address is 128 bits instead of 32, which takes us from 4.29 billion addresses to a number with 38 digits. Address exhaustion, the problem that gave us NAT and private addresses, simply stops existing.

However, the interesting part for this article is not what IPv6 added. It is what IPv6 removed. The designers had 20 years of experience studying IPv4 in the wild, and the new header is a field-by-field verdict on the old.

No more fragmentation. When receiving a datagram too large to be forwarded over the next link, an IPv6 router does not fragment it, but drops it and returns an ICMP “packet too big” message. The sender then retries with smaller datagrams. All of that machinery we saw before, the identifier, the flags, the offset, the fragile reassembly at the destination, goes away, and routers get faster because they never have to fragment anything.

The header checksum is also gone. There was no need to calculate a checksum at each hop just for the TTL since the link layer finds errors on its own and the transport layer checks its data.

The options are no longer there either. An IPv6 header is always 40 bytes long, and any additional information is placed in separate extension headers that follow. A router always knows where the payload begins, and processing time is predictable, precisely what IPv4 headers that can have different lengths could not offer. Broadcast, as we saw in Part I, is gone as well. Multicast and a new type called anycast have taken its place.

With that, we have reached the end of our tour through IP. In Part I, we learned how an address and a mask tell a device what is local and what is not. In Part II, we talked about how the Internet stretched 4.29 billion addresses across a world that needed far more. And in this part, we opened the packet itself and found that every field earns its place, because the ones that didn’t are exactly the ones IPv6 threw away.

Welcome to the 184th issue of the Polymathic Engineer newsletter. This week, we continue our series of articles on the Internet Protocol and addressing.

In Part I, we stayed inside a single network. We had a look at how an IPv4 address is structured, how a subnet mask splits it into network and host parts, and how a device uses both to decide whether to deliver a packet directly or forward it to its default gateway.

This week, we zoom out to the whole Internet. IPv4 gives us only about 4.29 billion addresses, and that number turned out to be far too small for the world we ended up with. So a big part of this story is about the tricks that kept IPv4 alive: a more flexible way to carve up address space, and a way for many devices to share a single public address. Along the way, we will see who actually owns the address space and how a device gets an address in the first place.

The outline is as follows:

• Classful addressing and why it failed

• CIDR

• Private addresses and NAT

• Who assigns IP addresses

• DHCP

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up and become a better software engineer.

Classful Addressing and Why It Failed

In the early days of the Internet, IPv4 addresses were assigned using a system known as classful addressing (see RFC 791). The idea was to split the address space into fixed classes. The class of an address decided where the line between the network and host portions fell.

Three of these classes were meant for regular networks:

• Class A used a /8 mask: the first 8 bits were the network, leaving 24 bits for hosts. That is about 16 million hosts per network.

• Class B used a /16 mask: 16 bits of network and 16 bits of host, or roughly 65,000 hosts per network.

• Class C used a /24 mask: 24 bits of network, leaving 8 bits for hosts, so only 254 usable hosts per network.

The problem is that these three sizes are extremely far apart, and most organizations fit uncomfortably between them. Let’s suppose a company has 2,000 hosts. A Class C network tops out at 254 addresses, which is nowhere near enough. So the company would be given a Class B network instead, which holds more than 65,000 addresses. It uses 2,000 of them and wastes the other 63,000, and nobody else can touch that block.

Multiply that waste across thousands of organizations, and you can see the issue. The Class B space disappeared quickly, with large chunks of address ranges left unused in the given blocks, and the fixed /8, /16, and /24 limits left no way to assign anything in between. The method was simple, but far too rigid for how the Internet was actually growing.

CIDR

The fix for all this rigidity is the system we still use: Classless Inter-Domain Routing, or CIDR (pronounced “cider”), defined in RFC 4632. The name tells it all. CIDR does away with the fixed classes and allows the boundary between network and host to be anywhere.

This is exactly the slash notation from Part I. An address is written as a.b.c.d/x, where x is the number of bits in the network prefix, and x can be any value, not just 8, 16, or 24. The company with 2,000 hosts doesn’t have to take a /16 anymore. A /21 has about 2,000 addresses, which fits it almost perfectly, and the rest of the space stays free for someone else.

The x most significant bits of the address are known as the network prefix, and every device in the organization shares the same prefix. This is where CIDR does something clever for routing. To the rest of the Internet, the organization appears to be a single prefix. However, a router outside simply looks at those x bits to know where to send a packet. So one entry of the type a.b.c.d/x is sufficient to forward traffic to any device in an organization. This allows the forwarding tables in the Internet’s core routers to be far smaller than they would otherwise be.

Let’s make this concrete. Let’s suppose an ISP is allocated the block 200.23.16.0/20. It can subdivide that block into eight smaller /23 blocks and allocate one to each of eight customer organizations:

ISP block        200.23.16.0/20
Organization 0   200.23.16.0/23
Organization 1   200.23.18.0/23
Organization 2   200.23.20.0/23
   ...
Organization 7   200.23.30.0/23

To the outside world, the ISP advertises a single route: send me anything that matches 200.23.16.0/20. The rest of the Internet has no knowledge there are eight organizations hiding behind that prefix, and it doesn’t need to. This is called address aggregation, and it is what prevents the Internet’s routing tables from expanding as more networks come online.

Aggregation works fine as long as addresses are sent down in neat blocks, but the real world is not always that simple. Normally, when an organization leaves an ISP, it hands back the addresses it was lent and renumbers its devices into a block from the new ISP. Renumbering every host and router is painful, though, so an organization sometimes holds on to its old addresses even after switching providers. Let’s follow what happens when Organization 1 keeps its 200.23.18.0/23 block but moves to a new ISP.

Read more

Welcome to the 183rd issue of the Polymathic Engineer newsletter. This week, we are going to start a new series of articles on the Internet Protocol (IP) and addressing.

IP is the protocol that moves packets from a source to their final destination, whether the two devices are on the same network or on opposite sides of the world. To arrive there, every packet needs an address, and understanding how those addresses work is the same as understanding how the network layer works.

In this first part, we concentrate on the IPv4 address itself: how it is structured, what a subnet mask does, and how a device can use both to figure out which network it belongs to.

The outline is as follows:

• IPv4 address structure

• Subnet masks

• Finding the network address

• Valid and invalid subnet masks

• Network, broadcast, and host addresses

• Types of destination addresses

• What exactly is a subnet?

• How devices communicate

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up and become a better software engineer.

IPv4 Address Structure

We write an IPv4 address in dotted-decimal notation, such as 192.168.1.100. But that is just for our convenience. Under the hood, an IPv4 address is really a 32-bit number, which gives us about 4.29 billion possible addresses in total.

To make those 32 bits readable, we divide them into four 8-bit sections called octets. Then we convert each octet into its decimal value and separate them with dots. That is where the dotted decimal notation comes from. Since each octet is 8 bits, its value can go from 0 to 255, so an IPv4 address ranges from 0.0.0.0 all the way up to 255.255.255.255.

For example, the address 193.32.216.9 looks like this in binary:

11000001 00100000 11011000 00001001

Every IPv4 address has two parts: a network portion and a host portion. The network portion says which network the address belongs to, and the host portion identifies a specific device on that network. We’ll spend most of this article on the line separating these two parts, because that’s where most of the confusion lies.

One important detail before we move on. An IP address is not really tied to a device, but to an interface, the boundary between a device and the link that connects it to the network. A regular host normally has just one link, so it has a single interface and a single IP address. A router is different: its whole job is to receive a packet on one link and forward it on another, so it has several interfaces, one per link. That means that a router has several IP addresses, one for each interface.

Subnet Mask

We just said that an IPv4 address has a network portion and a host portion. However, the address by itself doesn’t tell us where one ends and the other begins. That is the job of the subnet mask, also known as the prefix length. It marks the line between the network and the host portions.

A subnet mask is also 32 bits long, and it follows a simple rule: a group of 1s followed by a group of 0s, with no mixing. The 1s line up with the network part of the address, and the 0s line up with the host part. So if a mask starts with 16 ones, the first 16 bits of the address are the network, and the remaining 16 bits are the host.

You can write a subnet mask in two ways. The first is dotted-decimal notation, exactly like an IPv4 address. For example, 255.255.0.0 is a mask with 16 ones followed by 16 zeros. The second is slash notation, which just gives the number of 1 bits in the mask. That same mask becomes /16, as in 172.18.0.0/16.

Slash notation is more widespread these days, largely because it is shorter and harder to get wrong. We’ll see where it actually comes from in Part II, when we look at how addresses are handed out across the Internet. For now, treat the slash as a shorthand for how many bits belong to the network.

Finding the Network Address

So we have an address and a mask. How does a device actually combine them to find the network it belongs to? The operation it performs is a bitwise AND between the host address and the subnet mask.

The rule for AND is simple: if both bits are 1, then the output is 1; in all other cases, the output is 0. We line up the two 32-bit values and use that rule on each bit.

Let’s work through an example. Suppose a host has the address 192.168.1.100 and the mask 255.255.255.0. In binary, they look like this:

192.168.1.100   11000000 10101000 00000001 01100100
255.255.255.0   11111111 11111111 11111111 00000000
AND result      11000000 10101000 00000001 00000000

Read more

Welcome to the 182nd issue of the Polymathic Engineer newsletter.

In a recent article, we covered linear regression: how a machine learns to draw a line through your data and guess a number, like the salary of an engineer. But many of the questions we worry about in practice don’t have a number as an answer. Is this email spam or not? Is this patient sick or healthy? Will this user click or not?

Guessing a category rather than a number is known as classification, and it is one of the most important areas of machine learning. In this issue, we look at the oldest and simplest classification model: the perceptron. Invented in 1957, the perceptron is the building block of every neural network in use today. Once you grasp how a perceptron works, you have done a good part of the work needed to understand deep learning.

The outline is as follows:

• Building an Intuition: Opinion Mining

• Why We Need a Bias

• Weights, the Score, and the Step Function

• More Than Two Words: Hyperplanes

• How Good Is a Classifier? The Perceptron Error

• The Perceptron Trick and the Perceptron Algorithm

• A Bit of History: Where Perceptrons Fall Short

• Putting Everything Into Practice

How Is AI Changing Software Architecture?

AI is introducing new architectural questions. Other than models and frameworks, architects are now considering governance, reliability, agentic systems, inference at scale, and how AI fits within existing software ecosystems.

ARC 2026: Software Architecture in the Age of AI is an event that brings together software architects and engineering leaders from Google, AWS, Salesforce, Netflix, and other leading technology organizations to share how they’re tackling these challenges.

If these are conversations you are having within your team, the agenda is worth exploring.

25-26 July (Virtual)

Register here: ARC 2026

Use the code POLYMATHIC50 to get 50% off (Valid for the first 10 users)

Building an Intuition: Opinion Mining

Let’s suppose that your company launches an app in a foreign market, and the reviews start coming in. Each review has a star rating, so you know which ones are happy and which ones are angry. However, the reviews are written in a language you don’t speak, and you would love to know what makes a review positive or negative.

You notice that two words keep showing up, taka and bruk, so you pick 4 reviews and count how many times each word appears:

• “Taka taka taka!” (5 stars): taka 3 times, bruk 0 times

• “Bruk bruk bruk!” (1 star): taka 0 times, bruk 3 times

• “Taka taka bruk!” (5 stars): taka 2 times, bruk 1 time

• “Bruk taka bruk!” (1 star): taka 1 time, bruk 2 times

Then a new review arrives: “Taka bruk taka taka!”. There is no star rating yet, so what do you think it says? Likely, something good. Although you don’t speak the language, you can observe that taka appears more in the positive reviews and bruk more in the angry reviews.

We can turn this intuition into a rule. Give each word a score: taka gets +1 point, and bruk gets -1 point. To classify a review, add up the scores of all its words. If the total is zero or more, predict the review is happy. If it is negative, predict it is angry. The new review scores 3 - 1 = 2, so we predict it is happy.

The rule we just made is a perceptron. There is also a nice way to see it geometrically. If we plot each review on a 2D graph, with the number of times taka appears on the horizontal axis and bruk on the vertical axis, the happy reviews end up on the bottom right and the angry ones on the top left. The two groups are divided by the line where both words appear the same number of times. Everything below the line means happy, and everything above means angry.

That is what a perceptron does: it draws a line that separates the two classes of points.

Why We Need a Bias

A few months later, your app takes off in another country. Again, the reviews arrive in a language you don’t speak, and two words keep popping up: miko and zumi. You pick 6 reviews and count how many times each word appears:

• “Miko!” (1 star): miko 1 time, zumi 0 times

• “Zumi zumi!” (1 star): miko 0 times, zumi 2 times

• “Miko zumi miko!” (1 star): miko 2 times, zumi 1 time

• “Zumi miko zumi zumi!” (5 stars): miko 1 time, zumi 3 times

• “Miko zumi miko zumi!” (5 stars): miko 2 times, zumi 2 times

• “Miko miko zumi zumi miko!” (5 stars): miko 3 times, zumi 2 times

Let’s try the same approach and give each word a score. However, this time something doesn’t add up. Both words appear in happy and angry reviews, so neither one seems positive or negative on its own. But if you look closer, you can see a different pattern: all the angry reviews are short, while all the happy reviews are long. When people are sad in this country, they drop a few words and move on. When people are happy, they write more. What decides the mood is not which words appear, but how many.

We can turn this into a rule, too. Give both words a score of +1, then add them up and guess “happy” if the sum is 4 or more. To make the math easier later, we use a cutoff of 3.5 instead: happy if the score is above 3.5, angry if it is below.

However, most of the time, perceptrons are not written by comparing the score to a cutoff. The standard way is to move the cutoff into the score itself. Instead of asking “is the score at least 3.5?”, we add -3.5 to the score and ask “is the result at least 0?”. The first question is the same as the second, but the second always compares to zero, regardless of the dataset. This extra number is known as the bias, and it’s the negative of the cutoff.

The bias has a nice interpretation: it is the score of an empty review. A review with no words is marked as “happy” if the bias is positive and “angry” if it is negative. In our case, the bias is -3.5, and that matches the data: users in this country who say nothing extra are likely upset. You can find both situations in the real world. In app stores, an empty review usually comes with 5 stars, since happy users just tap a rating and leave (positive bias). On the other hand, no one opens a support ticket to say that everything works fine (negative bias).

In terms of geometry, the bias moves the line away from the origin. The first dataset’s line went through the point (0, 0); this one crosses the axes at 3.5. The weights decide the direction of the line, and the bias decides where it sits.

Weights, the Score, and the Step Function

Now that we have built two classifiers by hand, it is a good moment to give the pieces their official names. The counts of each word are the features, which are what we use to guess. The scores we assigned to the words are known as the weights, and are what the model learns from the data together with the bias. If this sounds like something you have seen before, it’s because you have seen it in the linear regression article. The score of a review is calculated with the same formula we used to guess a salary:

score = w1 × x1 + w2 × x2 + bias

where x1 and x2 are the number of times each word appears, and w1 and w2 are their weights.

One more thing is needed: turning the labels into numbers. Machine learning models work with numbers, not with moods, so from now on “happy” becomes 1 and “angry” becomes 0. To make the prediction match, we use a function called the step function. The step function returns 1 if its input is 0 or more, and 0 if its input is negative:

prediction = step(w1 × x1 + w2 × x2 + bias)

That is the whole perceptron: weights, a bias, and a step function. The only difference from linear regression is in that last step. Linear regression returns the score directly since that is what we want, a number. The perceptron squashes the score into a 1 or a 0, since we want a category.

The step function is also what is called an activation function, a concept that plays a huge role in neural networks. For now, you can think of it as the function that turns a score into a “yes” or a “no”.

More Than Two Words: Hyperplanes

Our two examples used languages with only two words each, which is helpful because we can draw every review as a point on a 2D graph. However, real languages have a lot of words. What happens to the perceptron then?

In terms of the formula, there are no changes. Each word has a value assigned, and the score stays the same even if it has more terms:

score = w1 × x1 + w2 × x2 + ... + wn × xn + bias

In terms of geometry, the picture grows by one dimension for every word. With three words, the reviews turn into points in 3D space, and the perceptron splits them with a plane instead of a line. With n words, the points live in n-dimensional space, and the line between them is called a hyperplane. We can’t picture it anymore, but the idea is the same: there is a flat surface dividing the space into a happy and an angry side.

If the idea of hyperplanes seems too abstract, here is a mental picture that helps. Let’s imagine building the classifier for English. You have to go over the whole dictionary and assign a score to every single word: “amazing” gets +9, “refund” gets -6, “the” gets 0.1, and so on. When you want to classify a review, you add up the scores of its words and the bias, then check the sign. Words that carry strong opinions end up with large weights, and neutral words end up with weights close to zero.

One last thing to keep in mind is that this setup isn’t just for reviews. The features can be anything you can count or measure, like the symptoms of a patient or an email’s properties. As long as each feature has a weight, the perceptron works the same way.

How Good Is a Classifier? The Perceptron Error

So far we have figured out the weights and the bias by looking at the data. That works for 4 reviews and 2 words, but not for millions of reviews and an entire dictionary. We want the machine to find them, so we need to teach the machine how to distinguish a good classifier from a bad one. In the linear regression article, we did this with an error function: a value that is large when the model performs badly and small when it performs well. We need the same thing here, and it turns out that building it takes three attempts.

Read more

Welcome to the 181st issue of the Polymathic Engineer newsletter.

Take a moment and think about what your computer is doing right now. You have a browser open with a dozen tabs, a music player running in the background, a chat app, and the editor you are typing in. Everything is going on together, without any hitch. We take it for granted, but it is a remarkable thing.

Most of the time, all this is happening on a machine that can really only physically run a handful of things at a time, and just one thing at a time on a single core. So how can the computer keep these jobs going at once? And what is going on underneath when it seems as if everything is running in parallel?

In a recent article, we broke down concurrent programs into processes and threads, and started calling them tasks. Now we look at the layer that brings those tasks to life. The operating system takes many tasks that all want the CPU and juggles them so fast that it feels like everything is running at once. This trick is called multitasking, and it is the reason a single core can feel like many.

The outline is as follows:

• CPU-bound and I/O-bound tasks

• Why one core isn’t enough

• Preemptive multitasking

• Context switching

• The scheduler

• Multitasking beyond the operating system

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up and become a better software engineer.

CPU-bound and I/O-bound tasks

Before we get to how the operating system juggles tasks, we need to look at what kind of work those tasks actually do. Not all work is the same, and the difference matters a lot for how the OS treats it.

Every application does two broad kinds of work. The first is computation: arithmetic, logic, comparisons, all the number crunching the CPU is built to. The second is input and output (I/O): reading from the keyboard, the disk, or the network, drawing on the screen, writing files, or sending data over a connection. I/O is really just the process of sending and receiving signals to devices. And most of the time, it does not involve the CPU at all. There is nothing to compute. We are just waiting for the device to answer.

This provides us with a simple way to classify tasks. We say a task is bound by some resource when that resource is the bottleneck: the thing that, if it were faster, would make the whole task faster. There are two main kinds.

A task is CPU-bound if it would run faster with a faster CPU. It spends most of its time computing. Some typical examples are:

• Math like additions, divisions, and matrix multiplication

• Encryption and decryption, which lean on heavy operations like prime factorization

• Image and video processing

• Running algorithms like sorting and binary search

A task is I/O-bound if it would run faster with a quicker I/O subsystem, whether that means reading from disk, getting user input, or waiting on a network response. Here the CPU often does nothing but wait for data to move to or from some device, and the CPU time is expensive. Some typical examples are:

• Most apps with a graphical interface, even if they never touch the disk, since they spend their time waiting on the keyboard or the mouse

• Databases and web servers, which spend most of their time on disk or network I/O

Why this distinction matters? Because it tells us what resource to improve. Take two programs. The first multiplies two huge matrices and returns the answer. The second copies a huge amount of data from the network onto a disk. These two will not speed up in the same way. What good is a faster clock, or a thousand cores instead of one, if the computer spends most of its time waiting for the next batch of data to reach the disk? An I/O bound load will not get better with more cores. However a CPU bound load can really benefit from being spread out across many cores at once.

There is one more thing to know. Over the years, applications have become more and more I/O-bound. The CPU speed kept increasing, and let us run more instructions in the same amount of time, but the data transfer speed never caught up. So the limiting factor in a program is frequently some I/O operation that blocks the CPU. The good news is that these blocking tasks can be detected and forced to run in the background, and that is precisely what modern runtime systems do. That background juggling is where multitasking begins.

Why one core isn’t enough

Let’s make this more concrete. Suppose one of your friends pulls an old portable game console out of a drawer. The device has one slow core, a little pixelated screen, and a few buttons. Your friend knows you write code, so she asks you to build a simple maze game for it, where a hero runs around collecting coins while monsters chase it.

Think about all the things the game has to do. It has to read the buttons so the player can move the hero. At the same time, the world keeps changing on its own, with the monsters moving and the coins being tracked. And the player has to see all of this on the screen as it happens.

So you split the work into three jobs:

• Reading the player’s input from the buttons. This is I/O-bound, since most of the time it just waits for a press.

• Computing the next state of the world from the game rules and the player’s moves. This is CPU-bound.

• Drawing the updated world on the screen. This is I/O-bound again.

The problem is that all three jobs need to make progress together, or the game feels broken. If the console stops reading input while it redraws the screen, the controls freeze. But you have a single core, and it can only ever run one job at a time.

Your first instinct might be to pick up threads. You spawn a single process with three threads, one for each of the jobs, and let them share the game state. It is a reasonable idea. However, if you run it, then the software hangs in the first thread. Reading input is an unending cycle of its own, waiting for a button, never letting the other two run. The single core has room for one thread, and that thread is hogging it.

The lesson is that threads alone don’t buy you anything here. Running things truly at the same instant, which is what we call parallelism, needs hardware with more than one core. On a single core, no amount of threads will make two jobs run at the same moment.

So how do we get out of this? The way out comes from separating two ideas that are easy to mix up: concurrency and parallelism. Parallelism is when things literally run at the same instant on different cores. Concurrency is less powerful but more general. It means that numerous jobs have overlapping lifetimes: they are all in progress over the same span of time, even though at any given instant only one of them is running.

This distinction is the whole point. On one core, we can’t run the three gaming jobs in parallel. But we can run them concurrently, switching between them fast enough that they all make progress. To the player, it looks like everything is moving together. That act of switching is multitasking, and it is how a single core fakes doing multiple things at once.

There is a nice way to think about the relationship between the two. Parallelism is an implementation detail of how the work physically runs. Concurrency is part of how we design the program in the first place. With the right hardware, the OS can turn our concurrent design into real parallelism. Without it, the OS still honors the design by interleaving the jobs on one core. Either way, from our point of view as programmers, the jobs run concurrently.

Preemptive multitasking

Read more

Welcome to the 180th issue of the Polymathic Engineer newsletter.

In a recent article, we talked about the DRY principle and why removing duplication can lead to fewer bugs and more reusable code. As you know, DRY is one of the most well-known rules in software engineering, and most of us learn it early in our careers.

But here’s the thing: DRY…

Read more

Welcome to the 179th issue of the Polymathic Engineer newsletter.

Regardless of whether you are an experienced machine learning practitioner or just a data scientist who has started their journey, you have surely had the same frustrating experience. You built a model, trained it on your data, everything looked fine, but then it fell apart in production.

There are two issues to be aware of. The first one is called underfitting. A model that underfits is too simple for the data. It makes many mistakes, even on the training data itself, because it can't capture the patterns underneath.

The second problem is called overfitting. This is the opposite of the first problem. In this situation, the model memorizes the training set perfectly; therefore, it can’t make generalizations.

In this article, we explore both underfitting and overfitting, as well as the methods to detect and address them. The outline is as follows:

• What can go wrong when training a model?

• Underfitting and overfitting: two sides of the same coin

• Detecting the problem with testing

• The golden rule and the validation set

• The model complexity graph

• Solving the problem: regularization

• Measuring complexity: L1 and L2 norms

• The regularization parameter

• When to use L1 vs L2

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up, and become a better software engineer.

What can go wrong when training a model?

Let’s try to answer this question starting with a simple analogy. Imagine you have to study for an exam. There are many things that could go wrong as you prepare.

You might not have studied enough. There is no way to fix that, and you are likely to perform poorly. But what if you studied a lot, just in the wrong way? For example, you choose to memorize the entire textbook word for word instead of trying to understand the material. Will you do well? Likely not, because you memorized everything without actually learning anything. You won’t know how to answer questions on the exam that you haven’t seen before.

The best approach is to study in a way that lets you answer questions you haven’t seen during your preparation.

The same things can go wrong with machine learning models. Not studying enough is like underfitting: the model is too simplistic to learn anything helpful from the input data. Memorizing the textbook is like overfitting: the model becomes too complicated and remembers the training data rather than learning the underlying patterns. A good model is one that learns the data properly and can make good predictions on new data it has never seen.

There is a fundamental trade-off in machine learning between finding the best model for the training data and ensuring it performs well with unseen data. Optimization is the process of tuning a model to work as well as possible on the training data. On the other hand, generalization is how well the model performs with data it has never seen before. The goal is always to generalize well, but there is a catch: you can only control the optimization. You adjust the model to fit the training data, and you hope it works well on other data. If you optimize too much, you can overfit, and generalization gets worse.

In the next section, we will look at underfitting and overfitting in more detail and see what they look like in practice.

Underfitting and overfitting: two sides of the same coin

A different way to look at underfitting and overfitting is through the lens of problem- solving. There are two mistakes you can make when you have a problem to solve. You can make the problem too simple and come up with an overly basic solution. Or you can overcomplicate the problem and come up with a solution that is too elaborate.

Imagine your task is to fix a leaking faucet. If you just put a bucket under it, you have oversimplified the problem. The faucet keeps dripping, and you are only catching the water, not fixing anything. This is underfitting: you are trying to model your data with something that is too simple to work effectively.

On the other hand, if you knock out all the plumbing in your house to fix a small drip, you have overcomplicated the solution. You may have fixed the leak, but this approach wasted resources and made the work more difficult than necessary. This is overfitting: your data might be simple, but you try to fit it with a model that is too complex. The model will fit the training data, but it will memorize it rather than learn from it.

Let’s look at a concrete example with polynomial regression. Suppose that you have a dataset that looks like a parabola, which is a curve that goes up and then down. You want to fit a polynomial to this data, but you don’t know what degree. Should you use a line, a quadratic, or something more complex?

If you fit a line (a polynomial of degree 1), the model is too simple. It is impossible for a straight line to capture the shape of a parabola, and the model underfits. If you fit a quadratic (a degree-2 polynomial), the model captures the essence of the data well. It neither underfits nor overfits. However, if you fit a polynomial of degree 10, the model becomes too flexible. It can bend and twist to reach each point in the dataset, but that is not the goal. Instead of learning that the shape looks like a parabola, it memorizes the exact positions of each point. The model overfits.

The real problem with overfitting shows up when you try to make predictions on new data. A model that memorized the training set will perform very poorly on data it has never seen before. It learned the noise and quirks in the training data, rather than the underlying pattern. Here is a simple way to tell what is happening with your model:

• Underfitting: The model performs poorly on the training data and poorly on new data. It is too simple to learn anything useful.

• Good fit: The model performs well on the training data and on new data. It has learned the underlying patterns.

• Overfitting: The model performs very well on the training data but poorly on new data. It has memorized rather than learned.

But how can you actually measure this? How can you know if your model will perform well on new data before you deploy it? This is where testing and validation come in.

Read more

Welcome to the 178th issue of the Polymathic Engineer newsletter.

This week, we are wrapping up our series of articles on service mesh. In the first post, we talked about what a service mesh is and how its architecture works. In the second, we looked at what it actually gives you in practice and at the trade-offs that come with it.

There is one more topic that feels at home in this series, and often confuses folks: the difference between a service mesh and an API gateway. Both manage traffic, both can do things like authentication, routing, and observability, and sit somewhere between a client and a service. It’s easy to think of both as the same kind of tool, or to reduce the difference to a simple rule of thumb such as “API gateways are for north-south traffic, service meshes are for east-west,” and go on.

That shortcut is a one-liner, but it masks what each tool is actually good at. They were built to tackle different problems, and treating them as interchangeable usually leads to one of two architectural mistakes: forcing one tool to do the other’s job, or layering them together without a clear understanding of why.

In this issue, we look at API gateways and service meshes side by side. We will discuss what an API gateway does, how it differs from a service mesh, when to use which, and how they work together when used correctly.

The outline is as follows:

• What an API gateway does

• Side-by-side differences

• When to use which

• Using them together

• Wrapping up the series

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up, and become a better software engineer.

What an API gateway does

An API gateway lives at the edge of your system and is the front door for everything coming in from outside. Mobile apps, web browsers, partner systems, and any other external client send their requests to the gateway, and the gateway is responsible for handing them off to the right backend service. It is a centralized component, which means that a single instance, or a small cluster, takes care of all the traffic entering the system.

This central position is what makes the gateway so useful. Each external request flows through it, so you can put a lot of cross-cutting work in one place instead of spreading it across many services. The gateway works at the application layer of the OSI model. This means that it understands HTTP, HTTPS, gRPC, and WebSocket, and it can read URLs, headers, methods, and payloads to decide what to do with each request.

The most basic job of an API gateway is routing. It looks at the path or the method of an incoming request and forwards it to the right backend service. It can also translate between protocols: a common example is accepting an HTTP call from a mobile client and turning it into a gRPC call to an internal service.

Another big responsibility is authentication and authorization. The API gateway does validation once at the edge, instead of each backend service having to include its own logic. It can validate OAuth2 tokens, JWTs, API keys, and integrate with an identity provider. The requests that do not pass these checks are rejected before they reach the internal services.

The gateway also takes care of rate limiting and request shaping. It is able to throttle abusive clients, block requests that aren’t valid, and stop a single noisy customer from taking over the whole system. This keeps bad behavior on the outside from affecting services in the backend.

Some other common tasks are spreading the load across multiple backend instances, caching frequent responses to speed up the system, or gathering logs and metrics for the traffic that goes through. Since the gateway is the single point through which all external traffic passes, it is a natural place to gather observability data on how clients are using the APIs.

Many API gateways also handle API versioning. Backend services do change, but the contract that external clients rely on cannot break overnight. The gateway can give a stable interface to customers while sending requests to newer versions of the services behind the scenes. It can also change the response so that older clients get what they expect.

Besides traffic, API gateways today are typically the key to what is sometimes referred to as API productization. If you think about APIs as products which are consumed by internal teams, partners, or third-party developers, routing and auth are not enough. A lot of gateways are integrated with large API management platforms with developer portals, self-service onboarding, monetization features, sandbox environments, and access control dashboards. This transforms the gateway from a network infrastructure element into the front office of your APIs.

For a more detailed overview of API gateways, you can refer to our previous article.

Side-by-side differences

Now that we have looked at both an API gateway and a service mesh in isolation, let’s put them side by side. There are 5 main differences worth knowing about.

The first one is the deployment model. A gateway is a centralized component. You can have a single instance or a small cluster at the edge that handles all the traffic coming from the outside. That makes it easier to apply global policies and observe all external interactions from a single place.

On the other hand, a service mesh is decentralized. It uses a sidecar proxy adjacent to each service replica instead of a single component. This shifts traffic control to each service, giving you fine-grained control, but adding moving parts to the system.

Read more

Welcome to the 177th issue of the Polymathic Engineer newsletter.

This is the final article in our series on recursion. In the last article, we looked at the divide-and-conquer pattern, which breaks a problem into independent subproblems, solves each one, and combines the results. This time, we examine depth-first search, a pattern for traver…

Read more

Welcome to the 176th issue of the Polymathic Engineer newsletter.

This week, we continue our series of articles on service mesh. In the previous article, we discussed what a service mesh is and how it works. We’ve seen that a service mesh is a separate layer of infrastructure that handles all the traffic between services. The key idea is the sidecar pattern, where a small proxy runs next to each service instance and quietly takes over the networking job.

Now that we understand what a service mesh is, the natural next question is what you really get out of it. The short answer is that it gives you 4 things: traffic management, reliability, security, and observability. All microservice systems have to deal with these issues, and a service mesh solves them all in one place.

Another benefit that runs through all of these is polyglot support. As we have seen in the previous article, the library approach breaks down when teams use more than one programming language. A service mesh avoids this issue because it does everything in the proxy layer. Regardless of whether a service is written in Go, Python, Java, or C#, it gets the same retries, the same mTLS, the same metrics, and the same routing rules. There is no library for each language to build and maintain.

However, not all is for free. A service mesh has a cost in terms of latency, complexity, and resources. First, we will talk about the beneficial things, then about the trade-offs, and finally, we’ll take a quick look at the most typical service mesh implementations.

The outline is as follows:

• Traffic management and routing

• Reliability

• Security

• Observability

• Trade-offs: when a mesh isn’t worth it

• A quick tour of popular meshes

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up, and become a better software engineer.

Traffic management and routing

The first thing a service mesh gives you is control over how traffic flows between your services.

When a service wants to call another service, it needs to know its address. In the past, when data centers were on-premises, these addresses were usually hardcoded because machines rarely changed. This isn't true anymore in the cloud. Instances come and go, IP addresses change, and services often move to different nodes. Your code will break whenever the network layout shifts if you hardcode addresses.

A service mesh takes care of this for you through service discovery. The sidecar keeps track of where each service lives in real time. The proxy finds out the service that the application is calling by name and directs the request there. The application doesn’t need to know any IP addresses, and it doesn’t need to be redeployed when something moves.

On top of this, the mesh can normalize service names across environments. The code might, for example, call a service called sessions-service. Then, when the code is put into production, it talks to a production instance. When it is put into staging, it talks to a staging instance. The mesh knows which one to use based on where it is running, so the application doesn’t have to carry environment-specific configuration.

Another key feature is load balancing. When there are multiple instances of a service, the mesh spreads the traffic across them. Because the sidecar understands application-level protocols like HTTP/2 and gRPC, it can do request-level load balancing, not just connection-level. This is important since one gRPC connection can handle numerous requests. If there is no request-level balancing, one connection would pin all the work to a single instance.

Beyond simple balancing, a service mesh supports more advanced traffic control. We can split traffic between two versions of a service: for example, send 95% of the calls to v1 and 5% to v2 to roll out a new release gradually. This is how canary deployments work at the infrastructure layer, without having to touch the application. We can also mirror traffic, sending a copy of the requests to a test instance to see how it handles production load, without affecting the real responses.

Traffic shaping and traffic policing are other useful tools. Shaping is when you slow down some traffic to make it fit a certain pattern. For example, you might give paying customers priority over free-tier users. Policing is making sure a contract is followed. For example, you could block a service that makes too many requests to a downstream dependency. Both are ways to keep the system stable when some parts behave badly.

You can set all of this without changing the application code. The mesh configuration is stored in files that the platform or infrastructure teams can update, and the sidecars pick up the changes.

Reliability

Networks fail. Services can become slow or unresponsive. Instances can crash or get too loaded during traffic spikes. In microservice systems, any call between services is a potential point of failure. If you don’t manage these failures in a consistent way, the issues spread through the system very quickly.

A service mesh can apply the well-known reliability patterns to every call, without the applications having to know about them.

The first one is retries. If a call fails because of a temporary error, the sidecar retries it automatically, usually with exponential backoff. This means that the proxy waits a bit longer between each attempt to avoid making things worse when a service is already having trouble. The mesh can also distinguish between errors that are worth retrying, like a network timeout, and errors that are not, like a 404 response.

Timeouts work in the same way. The sidecar gives up instead of waiting forever when a call takes too long. This is important because slow calls can add up quickly.If there is no timeout, a slow downstream service can use up all the system’s resources while every call upstream waits for a response that never comes.

Read more

Welcome to the 175th issue of the Polymathic Engineer newsletter.

Building software has changed a lot in the last decade. We moved from big monolithic applications to systems made up of many smaller services that talk to each other over the network. This change helped teams in several ways, such as the possibility to pick the right language for each job, faster development, and independent deployments.

But there is something to consider. A call to a function inside a monolith is now a call between two services over the network. And network calls are a whole different story. They can be slow, break or become lost. We have to think about things like balancing the load across service instances, retrying failed requests, encrypting traffic between services, and deciding which service can call which.

These are the kinds of problems a service mesh is built to solve.

At a high level, a service mesh is a dedicated layer that sits between your services and handles all the network traffic between them. The mesh handles encryption, timeouts, retries, and service discovery, so the services don’t need to take care of them in their own code. The application code makes a standard HTTP or gRPC call, and the mesh does the heavy lifting behind the scenes.

In this issue, we break down the fundamentals of service mesh: why the pattern exists, what it actually is, and how its architecture works. This is the first of three articles in a series. In the next two, we will talk about what a service mesh gives you in terms of features and how it stacks up against an API gateway.

The outline is as follows:

• Why we need a service mesh

• What a service mesh is

• The sidecar pattern

• Data plane and control plane

• Where a service mesh is deployed

Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).

Sign up, and become a better software engineer.

Why we need a service mesh

As we discussed, switching to microservices made it harder for services to talk to each other. These problems are not new. Anybody who has worked on distributed systems knows that network calls are slow and unreliable, and getting services to trust each other is hard. The size is what changed. With dozens of services talking to each other, every engineering team has to deal with these difficulties daily.

The first attempt to solve this was the library approach. Teams would create a shared library that managed things like retries, circuit breaking, load balancing, and service discovery, and every service would use it as a dependency. The Netflix stack from the early 2010s is a well-known example. It had Eureka for service discovery, Hystrix for circuit breakers, and Ribbon for client-side load balance. For a while, it worked well, and it became the standard way to build reliable microservice architectures. But the approach started to show its flaws as systems got bigger and teams used more than one programming language.

The biggest issue was that every programming language needed its own version of the library. If the team used C#, Python, and Go, they needed three implementations that behaved the same way. This is tougher than it sounds because even slight changes in how timeouts or retries are handled might make the system behave differently.

Another problem was version skew. Different services would use different versions of the library at any given time. To roll out a bugfix in the retry logic, every team had to redeploy their services. This slowed down changes that should have been easy to make.

Last but not least, the library model put a lot of work on the application teams to keep things running. Every developer had to know how to set up circuit breakers, discovery clients, and retries, although these have nothing to do with the business problem they were trying to solve. The code was harder to read, test, and change since application logic and networking logic were mixed together.

These limitations made engineering teams look for another approach. If networking concerns were the same across services, and if the library version of it was painful to maintain across languages, maybe the best thing to do was to move them out of the application altogether.

This is the idea behind a service mesh.

What a service mesh is

A service mesh is a separate layer of infrastructure that handles all the traffic between the services in a distributed system. It takes care of routing, load balancing, timeouts, retries, encryption, and observability, without the services having to be aware of it.

The last sentence is the main point. In the library approach we discussed before, each service had to consume a dependency and call specific APIs to get circuit breaking or retries. With a service mesh, the service just makes a standard call to another service. The mesh sits in the network path and quietly adds the extra behavior before the call reaches its destination.

This kind of traffic, where one internal service talks to another, is usually called east-west traffic. Traffic that comes from outside the system, such as a mobile app or a web browser calling a backend, is called north-south traffic. These 2 kinds of traffic have different needs and patterns. A user may only send a request to the backend once in a while, but with a microservice design, one request can lead to dozens of calls between services. This is why service mesh tools focus on east-west traffic and look different from the tools that manage traffic at the edge of the system. We will come back to this comparison in the third part of the series.

So how does a service mesh sit in the middle of every service call without the services having to know? The answer is the sidecar pattern.

The sidecar pattern

The sidecar pattern is the trick that makes a service mesh work. The key idea is to run a small proxy process next to each service instance, on the same machine or inside the same pod if we are on Kubernetes. The proxy is always deployed with the service as a single unit. This is why it is termed a “sidecar,” like the sidecar of a motorbike.

Read more