Understanding Object Storage in Modern Cloud Systems
How object storage systems like S3 or Azure Storage work, and how you can make the best use of them.
Hi Friends,
Welcome to the 127th issue of the Polymathic Engineer newsletter.
This week, we talk about object storage systems. Modern applications deal with a huge amount of data that needs to be stored and retrieved again later.
However, databases are not always the best fit, especially with unstructured data and media content. Managed object storage in the cloud, like AWS S3 or Azure Storage, provides a flexible and robust solution to this problem.
In this article, we will have a deep look at what such object storage systems are, how they work internally, and how to leverage them effectively. The outline is as follows:
Storage Types
What is an Object Storage?
Object Storages Architecture
How Object Storage Systems Scale
Storage Tiers and Lifecycle Management
Integration with Other Services
Project-based learning is the best way to develop technical skills. CodeCrafters is an excellent platform for practicing exciting projects, such as building your version of Redis, Kafka, DNS server, SQLite, or Git from scratch.
Sign up, and become a better software engineer.
Storage Types
Traditional storage options often struggle to keep up with the needs of an application as it grows and needs to handle more data. Of course, using a CDN can cut down on the number of requests that go to the application server by a large amount.
On the other hand, the server's hard disks can only hold a limited number of images, videos, and other files before they run out of space. A common way to work around this limit is to use a managed file store to store large static files. Such stores are very useful because they are scalable, highly available, and provide strong durability.
Using a managed file store, developers can completely offload the storage and serving of static resources to more specialized services. However, there are different kinds of managed storage systems out there. One way to differentiate between them is based on how they represent the data.
File storage is a general-purpose solution based on block storage that abstracts files and directories at a higher level, making it easier to work with them. It’s accessible via file-level network protocols such as SMB and NFS, which makes it great for sharing large files and folders within an organization. File-level access control allows to set up permissions and access control lists (ACLs) allows to increase security.
Block storage divides data into fixed-sized blocks and saves each as a separate unit. Each block has a unique address assigned with a logical addressing scheme and is usually available as a volume. Block volumes can be set up as a file system and used that way, or applications can handle them directly. Block is the most flexible and performant form of storage, but it can only be used on a single server and can't be shared.
Object storage is the more recently introduced form of storage. It is a large-scale, low-cost, durable option that can store and back up unstructured data. It stores data as flat objects, which are usually accessible through RESTful APIs. AWS S3 and Azure blob storage are well-known object storage implementations.
In the rest of this article, we will focus on object storages looking more precisely at what they are and how they are implemented.
What is an Object Storage?
An object storage is a system which is made to handle large amounts of unstructured data. In contrast to traditional file systems, which are organized in a tree-like layout, object storage uses a flat structure and stores data as separate "objects."
Each object is made up of three main parts:
The data itself. This could be a picture, movie, document, backup file, or something else. Object storage can hold any kind of data.
A unique identifier. Everything has a globally unique number that lets the system find it without having to know where it is physically. Often, this is given as a URL or URI.
Metadata is information about the object, like when it was created, how big it is, what kind of data it contains, and custom attributes that were added. Metadata helps organize and find things by letting items describe themselves. It's also possible to give each file a version number, like v1, v2, v3, etc. Since files can't be changed, versioning can be helpful if you want to make changes to a file and then share the new version.
A critical difference between object storage and other types of storage systems is indeed that objects are immutable. It is not possible to change a file that has already been uploaded to object storage. If you want to change it, you have to make the changes locally and then send the new version as a separate item. Either the old object has to be deleted or a whole new version has to be added in its place. It's not possible to make incremental changes.
This immutability is helpful in many ways. First, since no one can change the file while you're getting it, there's no need for file locks or synchronization tools. Second, you can divide a file into pieces and put them in different items or regions without worrying that changing one of the pieces will break the whole file. Third, you can simplify the data lifecycle process by setting rules to automatically delete old files when a newer version comes out or move all new versions to a different bucket.
Objects are organized using logical containers called "buckets," so you must first create a bucket to upload data. Because each bucket has its own namespace, object names must be unique within a bucket but can be used more than once in different buckets. You might create separate buckets for various types of data, like backup files, multimedia, or application logs.
The main operations you can do with object storage are simple and the same across all cloud providers: