Shopify APIs

Case study: how Shopify scaled their new APIs. Plus Shopify architecture and tech stack.

Hi Friends,

Welcome to the 57th issue of the Polymathic Engineer newsletter.

Last week, I had the chance to chat with Jason Selby, a Principal Engineer at Shopify, about how they set up and scaled their new APIs.

It was an unexpected opportunity to dive into Shopify's architecture and gain insights into its workings. I am now happy to share this case study with you.

The outline will be as follows:

• The Shopify architecture

• How Shopify scaled their new APIs

• Posts that make you think

The Shopify Architecture

Shopify is a commerce platform that lets anyone start a business and sell products online and offline. Its initial version was developed in 2004 by Tobi Lütke as a SaaS platform using Ruby on Rails.

Unlike many other platforms, Shopify's core app has never been rewritten. It has evolved over the years but retains the original codebase, with Tobi's original contributions preserved in version control history.

The core Shopify application is still a Rails monolith, complemented by numerous domain-specific Rails apps across the organization. These apps are tailored to specific functions, such as shipping integration, single sign-on, and the App Store.

Shopify's tech stack has also been consistent over the years, relying on MySQL for relational databases, Memcached for key/value storage, and Redis for queues and background jobs.

In 2014, Shopify faced data storage challenges, leading to the adoption of sharding to distribute data across multiple MySQL instances. However, an unexpected Redis outage highlighted the risks of shared resources.

This experience pushed the transition to a pod-based architecture. Each pod is an isolated instance of Shopify with its data stores and can be deployed in various regions, reducing the impact of outages.

Today, Shopify operates more than a hundred pods, ensuring high availability. To manage deployments efficiently, Shopify utilizes Docker, Kubernetes, and Google Kubernetes Engine. Load balancing is handled by Nginx, Lua, and OpenResty, enabling scriptable load balancing.

Shopify's client-side stack has evolved from jQuery to an in-house SPA framework and, most recently, to React and TypeScript. With improved JavaScript execution, server-side rendering, and GraphQL integration, Shopify Admin offers a robust and efficient user experience.

How Shopify scaled their new APIs

Merchants sell on Shopify many products that may have a lot of variants depending on colors, sizes, and other features. The previous generation of APIs had a limit of only 100 variants per product. The new APIs extend the support to 2000 variants.

Scaling the Database

Shopify mainly uses MySQL as its relational database and Memcached and Redis as its caching layer. It was pretty simple to make the data model bigger to handle more versions. The data layer was ready for the new cardinality, even if it meant some tweaks had to be made.

For example, they optimized indexing strategies to speed up query performance by carefully selecting which columns to index and employing techniques like composite indexes. This allowed users to efficiently access specific product variants without scanning the entire database.

Pagination and APIs

The main changes to support the 2000 variants were in the API layer. The first step was leveraging pagination in their admin to make it easy for merchants to find groups of product variants quickly. This improves user experience and reduces server load.

The second step was to divide the APIs into two groups to satisfy different use cases.

The first group is for merchants using Shopify as a system of work. Such merchants use Shopify as their primary system to actively manage their product data. They need highly efficient, high-leverage APIs to create, update, and organize their products, including variants.

The second group is for merchants who use Shopify as a system of truth. They have a more complex ecosystem where Shopify is just one part of a larger whole. In these cases, Shopify reflects the accurate state of the products but isn't necessarily the central system for product management. So, they need more concise APIs to synchronize with Shopify.

Webhooks

Webhooks were also changed to handle the larger data set better. Instead of sending all variants every time there was a change, Webhooks were optimized to transmit only the necessary information. This cut down on the payload size and made communication more efficient.

Tracer bullet approach

Shopify's success relies on a large ecosystem of merchants and 3rd party applications. And most of them had assumptions based on the previous 100-variant limit.

So, a critical part of the development process was making the changes without breaking anything for them.

Two things have been critical for that. The first was to use a tracer bullets approach to create a simplified but functional version of the system handling 2000 variants. This allowed teams to run through a few use cases and see where the breakpoints and the bottlenecks were.

The second was to strictly collaborate with 3rd-party app developers to ensure a smooth transition. This allowed them to try the new APIs, provided feedback and pushed them to adopt the changes quickly. One way of doing this was by initiating closed beta programs. This allowed selected 3rd-party app developers to work closely with Shopify's team, experiment with the new APIs, and help identify and resolve issues before rolling out the changes to a broader audience.

Posts that make you think

Writing self-review can be uncomfortable, but it definitely helps for performance reviews.

I used SQLite a lot as a beginner. It helps you to focus on the fundamentals without getting lost in more advanced features.

I've stopped using Scrum for the last 3 years, and I've never been so productive. The benefit coming from all those meetings is too little to justify the effort.