Delivering Mail

How the process of delivering emails works. Plus, a detailed explanation of the communication protocols involved.

Hi Friends,

Welcome to the 44th edition of the Polymathic Engineer newsletter. We all make extensive use of emails in our daily jobs. But as software engineers, it’s also interesting to dig deeper into how email delivery works under the hood.

The outline will be as follows:

• Physical mail vs. Emails

• Emails delivery pipeline

• Email protocols

• Email message format

• Interesting tweets

Physical mail vs. Emails

The process of delivering an email is similar to the one used to deliver physical mail.

In the real world, when we want to send mail to a friend, we put it in a nearby mailbox. The nearby post office gathers all the mail from the mailbox periodically and forwards it to the destination postal office. The destination postal office then puts the postcard in our friend's mailbox.

In the internet world, the process is very similar, but it takes only seconds instead of a few days. There are three main differences:

• we use a mail user agent like Gmail or Outlook instead of putting our mail in a mailbox

• an email is not delivered by postal offices but by mail transmission agents that communicate via the SMTP protocol

• our friend receive the mail from another mail user agent that uses the POP3/IMAP protocol

Email delivery pipeline

The whole workflow used to deliver emails can be summarized in these steps:

1. The sender writes an email using its local mail client and clicks send.

2. The email heads to the sender's mail remote server via SMTP

3. The sender's mail remote server gets the IP address of the receiver's mail remote server using the DNS MX record.

4. The sender's mail remote server sends the email

5. The receiver's mail remote server gets the email and stores it in the receiver's mailbox

6. The receiver logs into its local mail client.

7. The receiver's local mail client fetches unread emails using IMAP or POP.

Fundamental email protocols

There are many communication protocols involved in the email delivery process. Here is a more detailed explanation of the main ones:

• SMTP (Simple Mail Transfer Protocol) is a text-based protocol that uses a series of commands and responses to transfer emails between servers. As a result, SMTP has two sides: a client side that runs on the sender's mail server and a server side that runs on the receiver's mail server. It is good to note that SMTP does not typically require intermediary mail servers while sending email, even if the two mail servers are located on opposite sides of the world. It operates on port 25, but secure variants (e.g., SMTPS) use ports 465 or 587.  Suppose a receiver's email address is incorrect or no DNS MX record is found. In that case, SMTP is also responsible for notifying the sender with a bounce-back message generated by the receiver's mail server.

• POP (Post Office Protocol) is a standard email protocol that allows emails to be received and downloaded from a remote mail server to a local email client. Emails are often removed from the email server once downloaded to our computer or phone. This means that we can only access our emails on the device that downloaded them. RFC 1939 describes POP3, the most recent version of this protocol. POP has the significant limitation of requiring the entire email, including attachments, to be downloaded before it can be seen, which can be time-consuming for emails with large attachments.

• IMAP (Internet Message Access Protocol) is another popular email protocol for receiving emails. However, it works differently from POP. When we use IMAP, our emails stay on the server and are only downloaded when we click to see them. Email clients can sync with the server to keep the same mailbox state across numerous devices, thanks to this capability. IMAP is the most used protocol for individual email accounts, particularly among users who access their email on several devices. IMAP uses port 143, while IMAPS, the secure version, uses port 993.

• SPF (Sender Policy Framework): it's a protocol that helps in the prevention of email spoofing. It determines which email servers may send emails on behalf of a domain. Another helpful method against spoofing is DKIM (DomainKeys Identified Mail), which allows the receiver's server to verify that an email message was sent and authorized by the domain it claims to be from.

• SMTP Authentication: is an extension of SMTP that allows email clients to send outgoing email through an SMTP server securely. It typically requires a username and password to authenticate the sender.

• DNS (Domain Name System): DNS is essential for email because it converts domain names into IP addresses, allowing email servers to find each other online.

• TLS (Transport Layer Security): TLS encryption is used to safeguard the confidentiality and integrity of email data during transmission over SMTP (SMTPS), IMAP (IMAPS), and other email protocols.

Messages format

The format of emails is defined in RFC 5322 and is composed by a header and a body separated by a blank line. The body of the email contains the text, whereas the header contains additional information.

Each header line contains readable text, which consists of a keyword, a colon, and a value. Some keywords are mandatory, while others are optional.

For example, every header must contain a From: header line and a To: header line specifying the email addresses of the sender and the receiver. Instead the Subject: header line is optional.

Diagram

This week, I made the diagram of the email delivery process with a different tool called Eraser. It is an interesting mix between Draw.io, markdown notes, and diagram-as-code. I tried it and decided to collaborate with them for a few next issues of the newsletter.

Interesting Tweets

As Uncle Bob said, the code is the ultimate source of truth. But this truth is helpful only when the code is readable and self-documenting. It requires multiple iterations of refactoring to get there, plus being careful and intentional while naming things.

I'm used to call the approach that Vlad described Bugfix Driven Development since it is a variant of TDD. It always worked like a charm for me.

Reading is, for sure, not the highest priority for most developers. So, having this as a habit, you'll quickly gain more knowledge than other people.