Google File System
A distributed storage case study: the Google File System.
Hi Friends,
Welcome to the 78th issue of the Polymathic Engineer newsletter.
This week, we will discuss an interesting case study for distributed storage: the Google File System.
The outline will be as follows:
Introduction
Design Assumptions
Architecture
Architectural Considerations
Data Integrity
Interface
Mutations
Introduction
The Google File System (GFS) was designed and introduced in the early 2000s to meet the unique needs of Google’s extensive data processing requirements.
During that period, Google rapidly expanded its infrastructure to support various web services, including search, indexing, and analytics.
The existing file systems could not handle the scale and performance demands, leading Google to create GFS to meet its specific needs.
GFS was designed to provide high scalability and fault tolerance, ensuring data is stored reliably and accessed quickly, even if hardware failures and network issues occur.
Even if Google replaced it with a more advanced file system called Colossus, GFS significantly impacted the development of other distributed file systems.
For example, the Hadoop Distributed File System (HDFS), a key component of the Apache Hadoop project, was directly inspired by GFS. HDFS adopted many core principles of GFS, such as its distributed architecture, fault tolerance, and scalability, and became widely used for big data processing in the open-source community.
In the following sections, we'll delve into the most critical aspects of GFS.
GFS Design Goals
The goal of supporting Google's application workload affected several crucial design decisions. Here are some key assumptions:
Failures are the Default, Not the Exception: GFS assumes that component failures (e.g., disk failures, network partitions, server crashes) are common and must be managed automatically without human intervention.
Handling Large Files: Google typically works with large files, often several gigabytes. This influenced GFS to optimize for large files rather than small ones.
Access and Update Patterns: GFS is optimized for sequential reads and appends rather than random reads and writes. Most updates are performed by appending data, which aligns with Google’s data processing patterns.
GFS Architecture
GFS employs a distributed architecture composed of multiple nodes organized into a cluster.
There are three primary types of nodes in a GFS cluster:
Master Node: The master node acts as the central control unit of the GFS cluster. It maintains metadata about the file system, including the namespace and access control information. Files in GFS are divided into fixed-size chunks, identified by a unique 64-bit handler. The master assigns the handler to each chunk and keeps the files mapped to chunks. All metadata are stored in memory, but namespaces and file-to-chunk mapping are also persisted by logging the mutations to an operation log. This log is stored on the master’s disk and replicated on remote machines, making it possible to restore the master state simply and reliably in case of crashes.
Chunk Servers: These are the storage nodes where the data is stored. Each chunk server stores data in fixed-size chunks (typically 64MB), and each chunk is replicated across multiple chunk servers to ensure fault tolerance. Chunk servers regularly communicate with the master to report their status and the chunks they hold. GFS can scale horizontally by adding more chunk servers to the cluster if necessary.