HTTP Caching & Cache Headers

Making the Web Fast Without Touching the Server#

Caching is the single most effective performance technique on the web. A cached response never reaches your origin, never touches your database, and travels a fraction of the distance. HTTP caching is also the hidden machinery behind every CDN, and misconfigured caching is behind some of the most confusing bugs ("I deployed, why do users still see the old version?") and some serious security incidents (one user's account page served to another).

Where Caches Live#

A single response may be stored in several places:

• Browser cache: private to one user. DevTools marks hits as "(memory cache)" or "(disk cache)".
• Shared caches: CDNs, reverse proxies (NGINX, Varnish), corporate forward proxies. One stored copy serves many users.
• Application caches (Redis, in-process memoization): not governed by HTTP headers, but often confused with them.

All HTTP caches obey the same headers; the difference is who the stored copy may be served to.

Freshness vs. Validation#

Caching has two separate mechanisms:

1. Freshness: while a stored response is fresh, the cache reuses it without contacting the server at all. Controlled by Cache-Control: max-age=N (and the older Expires date header, which max-age overrides).
2. Validation: once stale, the cache asks the server "has this changed?" by sending a validator. The server replies 304 Not Modified with no body (reuse the stored copy, and refresh its freshness) or 200 with a new copy.

A cache decides freshness with simple arithmetic: a response is fresh while its current age is less than its freshness lifetime. Shared caches report the age they add in the Age header, so a browser receiving Cache-Control: max-age=300 and Age: 240 knows it has only 60 seconds of freshness left.

If a response has neither max-age nor Expires, caches may apply heuristic freshness, typically a fraction (often 10%) of the time since Last-Modified. A file last modified 10 days ago may silently be cached for a day. If you do not want surprises, always send explicit Cache-Control.

The Cache-Control Header#

The most misread pair: no-cache does not mean "do not cache". It means "always check with me first", which still saves bandwidth through 304s. If data must never be written to disk or a proxy, use no-store.

Validators: ETag and Last-Modified#

• ETag: an opaque version identifier (often a hash of the content). The client returns it in If-None-Match.
• Last-Modified: a timestamp with one-second resolution. The client returns it in If-Modified-Since.

ETag is more precise and is preferred when both are present. A weak ETag (W/"abc") means "semantically equivalent" rather than byte-identical; servers sometimes weaken ETags automatically when they compress responses on the fly.

Here is conditional GET implemented with the standard library, so you can watch validation happen:

import hashlib
from http.server import BaseHTTPRequestHandler, HTTPServer

BODY = b'{"price": 42, "currency": "EUR"}'
ETAG = '"' + hashlib.sha256(BODY).hexdigest()[:16] + '"'

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        # Validation: the client already holds this exact version
        if self.headers.get("If-None-Match") == ETAG:
            self.send_response(304)
            self.send_header("ETag", ETAG)
            self.send_header("Cache-Control", "max-age=60")
            self.end_headers()
            return
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(BODY)))
        self.send_header("ETag", ETAG)
        self.send_header("Cache-Control", "max-age=60")   # fresh for 60 s
        self.end_headers()
        self.wfile.write(BODY)

HTTPServer(("127.0.0.1", 8000), Handler).serve_forever()

curl -si http://127.0.0.1:8000/
curl -si -H 'If-None-Match: "0435d7e6b71c40a1"' http://127.0.0.1:8000/

HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 32
ETag: "0435d7e6b71c40a1"
Cache-Control: max-age=60

{"price": 42, "currency": "EUR"}

HTTP/1.0 304 Not Modified
ETag: "0435d7e6b71c40a1"
Cache-Control: max-age=60

The 304 carries no body; the client reuses its stored copy. A real implementation should also handle a comma-separated list of ETags in If-None-Match. A frequent production bug is ETags that differ between servers behind a load balancer (for example, derived from file inode numbers or build times), so clients rarely get a 304. Derive ETags from content or a version number.

The Vary Header and the Cache Key#

A cache stores responses under a cache key, by default the method plus the full URL. Vary adds request headers to that key:

• Vary: Accept-Encoding stores separate gzip, brotli and uncompressed variants. Without it, a client that cannot decompress may receive compressed bytes.
• Vary: Accept-Language keys localized pages separately. Forgetting it produces the classic "why is my CDN serving French to everyone?" bug.
• Vary: Cookie or Vary: User-Agent technically works but fragments the cache into near-uselessness.
• Vary: * makes the response effectively uncacheable.

CDNs let you customize the key further (ignore some query parameters, add a header). Anything that changes the response but is not in the key is a bug waiting to happen, and headers that change the response but are not keyed are exactly what web cache poisoning exploits.

Client Caches vs. Shared Caches#

Anything user-specific (account pages, dashboards, API responses containing personal data) must be private or no-store. Otherwise a shared cache can serve one user's data to the next. Two related traps:

• Authenticated responses: shared caches do not store responses to requests carrying Authorization unless you explicitly allow it (public, s-maxage). Cookie-authenticated pages get no such protection by default, so mark them yourself.
• Web cache deception: if /account/profile.css returns the profile page and the CDN caches by file extension, an attacker who tricks a victim into visiting that URL can then fetch the cached copy. Configure the CDN to respect origin headers, and make the origin return 404 for paths it does not recognize.

Some CDNs also support targeted headers such as CDN-Cache-Control or Surrogate-Control, letting you give the edge a long TTL while browsers get a short one. Check your provider's documentation for which it honours.

Recipes That Work#

Cache Busting#

You cannot reliably and instantly invalidate every cache in the world: browser caches are out of your reach and CDN purges take time. The robust pattern is cache busting: put a content hash in the asset filename, which build tools (webpack, Vite, Django's ManifestStaticFilesStorage, Rails' asset pipeline) do automatically. When content changes, the URL changes, so stale copies are simply never requested again. HTML pages get a short or zero freshness so they always point at the newest hashes.

Query-string versioning (app.js?v=2) also works, but some caches are configured to ignore query strings, so filename hashing is safer.

Debugging Caching Problems#

Find which cache is responsible:

curl -sI https://example.com/app.js | grep -iE 'cache-control|age|etag|last-modified|vary|x-cache|cf-cache-status|via'

• Age greater than zero means a shared cache served the response.
• Vendor headers such as X-Cache: HIT, CF-Cache-Status: HIT/MISS/EXPIRED/BYPASS or Via identify the layer.
• A hard reload in the browser bypasses the browser cache but not the CDN. If a hard reload fixes it, the browser copy was stale.
• A unique query string (?nocache=123) bypasses caches keyed on the full URL. If that fixes it, an intermediate cache is stale.
• In DevTools, tick "Disable cache" while the panel is open, and check the Size column for "(disk cache)" and the status for 304.

Common root causes: HTML cached for hours so users load old asset references; a missing Vary; ETags that differ per server; Set-Cookie on responses you intended to cache (many CDNs refuse to cache those, killing your hit rate); and a framework adding no-store to every response by default.

Never rely on manual purges as your deployment strategy. Version asset URLs, give HTML a short lifetime, and caching stops being a source of fear. For scrapers and API clients, the same headers let you avoid re-downloading unchanged pages; see Caching and Resume.

Practice#

Run the ETag server above, then request it with requests: store the ETag from the first response, send it back in If-None-Match on the second, and confirm you receive 304. Then change BODY and restart the server to see the validator fail and a fresh 200 returned.