Service Mesh: Why and How It Works (Part I)
How a dedicated infrastructure layer can take over the job of managing service-to-service communication.
Hi Friends,
Welcome to the 175th issue of the Polymathic Engineer newsletter.
Building software has changed a lot in the last decade. We moved from big monolithic applications to systems made up of many smaller services that talk to each other over the network. This change helped teams in several ways, such as the possibility to pick the right language for each job, faster development, and independent deployments.
But there is something to consider. A call to a function inside a monolith is now a call between two services over the network. And network calls are a whole different story. They can be slow, break or become lost. We have to think about things like balancing the load across service instances, retrying failed requests, encrypting traffic between services, and deciding which service can call which.
These are the kinds of problems a service mesh is built to solve.
At a high level, a service mesh is a dedicated layer that sits between your services and handles all the network traffic between them. The mesh handles encryption, timeouts, retries, and service discovery, so the services don’t need to take care of them in their own code. The application code makes a standard HTTP or gRPC call, and the mesh does the heavy lifting behind the scenes.
In this issue, we break down the fundamentals of service mesh: why the pattern exists, what it actually is, and how its architecture works. This is the first of three articles in a series. In the next two, we will talk about what a service mesh gives you in terms of features and how it stacks up against an API gateway.
The outline is as follows:
Why we need a service mesh
What a service mesh is
The sidecar pattern
Data plane and control plane
Where a service mesh is deployed
Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for tackling exciting projects, such as building your own Redis, Kafka, a DNS server, SQLite, an HTTP server, or Git from scratch using your favorite programming language. Now you can also try to build your own Claude Code (for free, still in beta).
Sign up, and become a better software engineer.
Why we need a service mesh
As we discussed, switching to microservices made it harder for services to talk to each other. These problems are not new. Anybody who has worked on distributed systems knows that network calls are slow and unreliable, and getting services to trust each other is hard. The size is what changed. With dozens of services talking to each other, every engineering team has to deal with these difficulties daily.
The first attempt to solve this was the library approach. Teams would create a shared library that managed things like retries, circuit breaking, load balancing, and service discovery, and every service would use it as a dependency. The Netflix stack from the early 2010s is a well-known example. It had Eureka for service discovery, Hystrix for circuit breakers, and Ribbon for client-side load balance. For a while, it worked well, and it became the standard way to build reliable microservice architectures. But the approach started to show its flaws as systems got bigger and teams used more than one programming language.
The biggest issue was that every programming language needed its own version of the library. If the team used C#, Python, and Go, they needed three implementations that behaved the same way. This is tougher than it sounds because even slight changes in how timeouts or retries are handled might make the system behave differently.
Another problem was version skew. Different services would use different versions of the library at any given time. To roll out a bugfix in the retry logic, every team had to redeploy their services. This slowed down changes that should have been easy to make.
Last but not least, the library model put a lot of work on the application teams to keep things running. Every developer had to know how to set up circuit breakers, discovery clients, and retries, although these have nothing to do with the business problem they were trying to solve. The code was harder to read, test, and change since application logic and networking logic were mixed together.
These limitations made engineering teams look for another approach. If networking concerns were the same across services, and if the library version of it was painful to maintain across languages, maybe the best thing to do was to move them out of the application altogether.
This is the idea behind a service mesh.
What a service mesh is
A service mesh is a separate layer of infrastructure that handles all the traffic between the services in a distributed system. It takes care of routing, load balancing, timeouts, retries, encryption, and observability, without the services having to be aware of it.
The last sentence is the main point. In the library approach we discussed before, each service had to consume a dependency and call specific APIs to get circuit breaking or retries. With a service mesh, the service just makes a standard call to another service. The mesh sits in the network path and quietly adds the extra behavior before the call reaches its destination.
This kind of traffic, where one internal service talks to another, is usually called east-west traffic. Traffic that comes from outside the system, such as a mobile app or a web browser calling a backend, is called north-south traffic. These 2 kinds of traffic have different needs and patterns. A user may only send a request to the backend once in a while, but with a microservice design, one request can lead to dozens of calls between services. This is why service mesh tools focus on east-west traffic and look different from the tools that manage traffic at the edge of the system. We will come back to this comparison in the third part of the series.
So how does a service mesh sit in the middle of every service call without the services having to know? The answer is the sidecar pattern.
The sidecar pattern
The sidecar pattern is the trick that makes a service mesh work. The key idea is to run a small proxy process next to each service instance, on the same machine or inside the same pod if we are on Kubernetes. The proxy is always deployed with the service as a single unit. This is why it is termed a “sidecar,” like the sidecar of a motorbike.