Email Protocols: SMTP, IMAP and POP3

From Outbox to Inbox

Email still runs on plain-text commands over TCP, store-and-forward delivery that can take days, and security bolted on decades later. Three protocols do the work. SMTP moves messages from sender to recipient's server, and IMAP or POP3 let the recipient read them. Knowing which hop uses which protocol, on which port, is most of what you need to debug "my email didn't arrive".

The Journey of a Message

When alice@example.com mails bob@example.net:

1. Alice's MUA submits the message over SMTP to her provider's MSA, logging in first.
2. Her provider's MTA looks up the MX records of example.net in DNS and connects to the best one on port 25.
3. The receiving MTA runs spam and authentication checks, then hands the message to the MDA.
4. Bob's MUA fetches it with IMAP or POP3.

Only step 2 crosses between organisations, and it runs without user authentication: the receiving server accepts mail for its own domains from anyone, which is why spam and spoofing exist and why SPF, DKIM and DMARC were invented (see email security).

Finding the Receiving Server

An MTA picks the destination from MX records, trying lower preference values first and falling back to the rest:

import dns.name
import dns.resolver

def mail_servers(domain):
    '''Hosts an MTA would try for this domain, in order (RFC 5321 section 5).'''
    try:
        answer = dns.resolver.resolve(domain, "MX")
    except dns.resolver.NoAnswer:
        return [(0, domain)]                 # no MX at all: use the domain's own A/AAAA
    except dns.resolver.NXDOMAIN:
        raise ValueError("domain does not exist: bounce now")
    if any(r.exchange == dns.name.root for r in answer):
        raise ValueError("null MX: this domain accepts no mail")
    return sorted((r.preference, r.exchange.to_text(omit_final_dot=True)) for r in answer)

for domain in ["gmail.com", "outlook.com", "example.com", "no-such-domain-xyz.invalid"]:
    try:
        print(f"{domain:<27}", mail_servers(domain))
    except ValueError as e:
        print(f"{domain:<27} -> {e}")

gmail.com                   [(5, 'gmail-smtp-in.l.google.com'), (10, 'alt1.gmail-smtp-in.l.google.com'), (20, 'alt2.gmail-smtp-in.l.google.com'), (30, 'alt3.gmail-smtp-in.l.google.com'), (40, 'alt4.gmail-smtp-in.l.google.com')]
outlook.com                 [(5, 'outlook-com.olc.protection.outlook.com')]
example.com                 -> null MX: this domain accepts no mail
no-such-domain-xyz.invalid  -> domain does not exist: bounce now

Three details matter. With no MX record, SMTP falls back to the domain's A/AAAA address, so a web-only domain may silently receive delivery attempts; publish a null MX (MX 0 .) to refuse them cleanly. Hosts with equal preference are tried in random order to share load. And delivery is store and forward: if every MX is unreachable, the sending MTA queues the message and retries, typically for several days, before bouncing it.

The SMTP Dialogue

SMTP is a line-based conversation of commands and three-digit replies. You can watch it with Python's smtplib against a local test server (pip install aiosmtpd, then python -m aiosmtpd -n -l 127.0.0.1:8025 prints every message it receives instead of delivering it):

import smtplib
from email.message import EmailMessage

msg = EmailMessage()
msg["From"] = "Alerts <alerts@example.com>"
msg["To"] = "ops@example.net"
msg["Subject"] = "Disk 91% on web-1"
msg.set_content("Disk usage on web-1 is 91%.\nThreshold: 85%.")
msg.add_alternative("<p>Disk usage on <b>web-1</b> is 91%.</p>", subtype="html")

with smtplib.SMTP("127.0.0.1", 8025, timeout=10) as smtp:
    smtp.set_debuglevel(1)            # print the SMTP dialogue to stderr
    refused = smtp.send_message(msg)  # returns {} if every recipient was accepted
print("refused:", refused)

The dialogue, trimmed:

send: 'ehlo [192.168.56.1]\r\n'
reply: b'250-simpleprog\r\n'
reply: b'250 HELP\r\n'
send: 'mail FROM:<alerts@example.com>\r\n'
reply: b'250 OK\r\n'
send: 'rcpt TO:<ops@example.net>\r\n'
reply: b'250 OK\r\n'
send: 'data\r\n'
reply: b'354 End data with <CR><LF>.<CR><LF>\r\n'
send: b'From: Alerts <alerts@example.com>\r\nTo: ops@example.net\r\nSubject: Disk 91% on web-1\r\nMIME-Version: 1.0\r\nContent-Type: multipart/alternative; ...\r\n.\r\n'
reply: b'250 OK\r\n'
send: 'QUIT\r\n'
reply: b'221 Bye\r\n'
refused: {}

• EHLO names the client and the server replies with its extensions, one per line. A hyphen after the code (250-) means more lines follow.
• MAIL FROM and RCPT TO form the envelope. The From: and To: headers inside DATA are just part of the message, and nothing in SMTP forces them to match the envelope. That is how Bcc works (the recipient is in the envelope but not the headers; send_message strips the Bcc: header for you), and it is also how spoofed From: headers get through.
• The message ends with a line containing only .. A body line starting with a dot is escaped by doubling it, which smtplib does for you.

Reply codes follow a pattern worth memorising: 2xx success, 3xx send more, 4xx temporary failure (the sender should retry later), 5xx permanent failure (bounce). Modern servers add enhanced codes such as 5.1.1.

STARTTLS versus Implicit TLS

On port 587 the connection starts in plain text and upgrades with STARTTLS. Watching Gmail's submission server shows why the order matters:

import smtplib, ssl

with smtplib.SMTP("smtp.gmail.com", 587, timeout=10) as s:
    s.ehlo("laptop.example")
    print("plain:", "starttls" in s.esmtp_features, s.esmtp_features.get("auth"))
    s.starttls(context=ssl.create_default_context())
    s.ehlo("laptop.example")                   # capabilities must be re-read after TLS
    print("tls:  ", "starttls" in s.esmtp_features, s.esmtp_features.get("auth"))

plain: True None
tls:   False  LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH

AUTH is only offered after encryption starts, so passwords never cross the wire in clear. The weakness of STARTTLS is that the first exchange is unprotected: an attacker on the path can delete STARTTLS from the capability list (STARTTLS stripping), and a careless client then continues in plain text. smtplib.starttls() raises an error if the server does not offer it, which is the correct behaviour; code that catches that error and carries on has reintroduced the bug. Implicit TLS on port 465 (smtplib.SMTP_SSL) negotiates TLS before any SMTP byte, removing the downgrade window, which is why current guidance (RFC 8314) prefers it for submission.

Between MTAs on port 25, TLS is opportunistic: servers encrypt when both sides support it and otherwise fall back to plain text, and they usually accept any certificate. MTA-STS (a policy published over HTTPS) and DANE (TLSA records in DNSSEC-signed zones) let a receiving domain insist on authenticated TLS. To inspect a server's TLS setup, use openssl s_client -starttls smtp -connect smtp.example.com:587 -servername smtp.example.com, as in the TLS handshake lesson.

Sending Mail from Applications

For real submission, authenticate over TLS:

import smtplib, ssl

context = ssl.create_default_context()
with smtplib.SMTP_SSL("smtp.example.com", 465, context=context, timeout=20) as smtp:
    smtp.login("alerts@example.com", password_from_secret_store)
    smtp.send_message(msg)

Exceptions tell you which layer failed: SMTPAuthenticationError (535, wrong credentials or basic auth disabled), SMTPRecipientsRefused (every recipient rejected), SMTPServerDisconnected, or a socket timeout (usually a blocked port). Large providers increasingly disable password logins for SMTP and IMAP in favour of OAuth 2.0 (XOAUTH2 in the capability lists above) or per-app passwords. For volume sending, a transactional email service handles retries, bounces and reputation better than a script, and either way delivery depends on SPF, DKIM and DMARC for the sending domain.

IMAP versus POP3

IMAP commands carry a tag so replies can be matched to requests, and untagged * lines carry data. A typical exchange looks like this:

C: a1 SELECT INBOX
S: * 172 EXISTS
S: * OK [UIDVALIDITY 3857529045] UIDs valid
S: a1 OK [READ-WRITE] SELECT completed
C: a2 UID SEARCH UNSEEN
S: * SEARCH 4821 4822
S: a2 OK SEARCH completed

Use UIDs, not message sequence numbers: sequence numbers shift whenever a message is deleted, while a UID stays fixed as long as the folder's UIDVALIDITY value does not change. A sync tool that stores UIDs must discard them if UIDVALIDITY changes. In Python:

import email
import imaplib
from email.policy import default

with imaplib.IMAP4_SSL("imap.example.com", 993) as imap:
    imap.login("alice@example.com", password_from_secret_store)
    imap.select("INBOX", readonly=True)                # readonly: never change flags
    status, data = imap.uid("SEARCH", None, "UNSEEN")
    for uid in data[0].split()[-5:]:
        status, parts = imap.uid("FETCH", uid, "(BODY.PEEK[HEADER.FIELDS (FROM SUBJECT)])")
        headers = email.message_from_bytes(parts[0][1], policy=default)
        print(uid.decode(), headers["From"], "|", headers["Subject"])

BODY.PEEK fetches without setting the \Seen flag, so a monitoring script does not mark the user's mail as read.

Debugging Delivery

Every MTA adds a Received: header at the top, so a message's headers read bottom to top as its route. Gaps in timestamps show where it waited in a queue. Bounces and logs carry the reply codes:

Practice

Run the local aiosmtpd server, send a message with a Bcc: header, and compare the envelope recipients in the debug output with the headers the server printed. Then run the STARTTLS script against your own provider's submission host and note which AUTH mechanisms it offers.