Storage Probes, ETags and Supercookies

The Cookie Is Not the Identifier Any More

Ask a vendor how they recognise returning visitors and they will talk about cookies. Ask an anti-bot vendor and they will talk about everything the browser stores on disk: localStorage, sessionStorage, IndexedDB, the Cache Storage API, service worker registrations, and the HTTP cache's own validators. Each of those is origin-scoped, none of them is cleared by "clear browsing data" unless the user finds the right checkbox, and one of them does not require JavaScript at all.

That last one is the interesting one. An ETag derived from a device feature, cached by the browser and echoed back in If-None-Match, identifies a device with zero script execution, zero permissions and zero DOM. The response code alone discloses whether the client is the same machine that fetched the resource last week.

Why Vendors Moved Off Cookies

Cookies are the most aggressively defended storage on the web. Every major browser offers a third-party cookie block, SameSite=Lax by default since Chrome 80 killed a large class of cross-site request, and Safari's Intelligent Tracking Prevention partitions storage by first-party site after a 24-hour cap. A vendor that needs a durable identifier cannot rely on a mechanism the user can delete in one click from a menu that also says "cookies and site data".

What is left is the set of origin-scoped stores below. None of them is protected by the same user-facing controls, and several of them survive a cookie clear by design.

The last row is the outlier and the strongest: "clear cookies and site data" in Chrome does not flush the HTTP disk cache, so cached ETag validators, and the fact that a validator is present at all, survive the one action most users take when they want to be untracked.

ETag and If-None-Match Need No JavaScript

RFC 9110 defines a validator: an opaque token the server hands out with a representation and the client hands back to ask "is my copy still current?". A strong ETag is a byte-for-byte identity of the representation, written as a quoted string such as "54752130e6b83cd3". A weak validator carries a W/ prefix and means "semantically equivalent", which is all a fingerprint needs.

The mechanics of the cache-based supercookie, published in 2015 as a browser fingerprinting technique that works cross-browser and needs no script, are these four steps:

1. The server computes a token from a device feature it can already observe: client hints, Accept-Language, Accept-Encoding, or anything a prior JavaScript probe reported.
2. It serves a tiny cacheable body, typically a 1x1 GIF or a 43-byte script, with ETag: "<token>".
3. The browser caches the body and the validator. Nothing happens visibly.
4. On the next request to that URL the browser attaches If-None-Match: "<token>" automatically, because the resource is stale or the site asked for revalidation. The server compares the two strings and learns whether this is the same device.

The response code is the whole payload. A 304 Not Modified means the token matched, so this is a device that has been here before. A 200 OK means it did not, so this is a new device, or a device whose cache was cleared. A WAF does not even need the body; it needs one bit per request, delivered in a header the page never sees.

GET /px.gif HTTP/1.1
Host: shop.example
Sec-CH-UA-Platform: "Windows"
Sec-CH-UA-Mobile: "?0"
Accept-Language: en-GB,en;q=0.9

HTTP/1.1 304 Not Modified
ETag: "54752130e6b83cd3"
Vary: Sec-CH-UA-Platform, Sec-CH-UA-Mobile, Accept-Language
Cache-Control: no-cache

Vary Multiplies the Cache, and Therefore the Fingerprint

Vary tells the cache which request headers participate in the cache key. Vary: Accept-Encoding is routine because content codings differ. Vary: User-Agent and Vary: Accept-Language are where the multiplication starts, because each combination is a separate cache entry with its own ETag.

A single URL with Vary: Sec-CH-UA-Platform, Sec-CH-UA-Mobile, Accept-Language holds one entry per platform, per mobility flag, per language list. On a real browser that is a handful of entries after a week of use. The interesting property is not the count but the shape: a client that has entries for en-GB and de-DE has changed its own language, which is a different claim from the one in its current Accept-Language, and a client with a platform entry for both Windows and macOS has changed its own operating system.

Two techniques make the token survive a determined clear:

• Cache-Control: no-cache with a genuinely cacheable body. no-cache does not mean "do not store", it means "store it, but revalidate before every use". The entry stays in the disk cache, the browser still sends If-None-Match, and a naive reading of the header name ("this is not cached") leads defenders to leave it alone.
• A long max-age on a resource the page never needs. If the site ships its own pixel with Cache-Control: public, max-age=31536000, immutable, the browser holds the validator for a year and revalidates whenever the entry is re-requested, which is a low-priority request the page never blocks on.

The Storage Round Trip Is a Probe in Disguise

The JavaScript side of the same idea is older and simpler: write a value, navigate, read it back. The pattern every collector uses is a fixed key, a fixed value, and a fixed shape.

// probe.js, served from the collector's own origin
const KEY = '__sess_probe_9f2c';
try {
  localStorage.setItem(KEY, String(Date.now()));
  sessionStorage.setItem(KEY, '1');
  indexedDB.databases().then(dbs => {
    const names = dbs.map(d => d.name + '@' + d.version);
    document.title = JSON.stringify({
      ls: localStorage.getItem(KEY) !== null,
      ss: sessionStorage.getItem(KEY) !== null,
      idb: names,
      keys: Object.keys(localStorage).length,
    });
  });
} catch (e) { document.title = 'blocked:' + e.name; }

The interesting fields are not the booleans you just set. They are Object.keys(localStorage).length and the IndexedDB database list, because a real origin that has been visited by a real person accumulates state: consent records, cart identifiers, feature-flag buckets, cached lists. The value of the round trip is that it tells the collector how old the storage state is and how much of it there is, which a boolean set by the probe itself cannot.

For a scraper the practical consequence is the mirror image. A fresh automation context per job has empty localStorage, an empty sessionStorage, no IndexedDB, no service worker and no cache entries, so a write-read round trip returns nothing on the second page. If you reuse one context for a whole session, you must also reuse the same profile across sessions, or the emptiness itself becomes the signal. The store-level detail is covered in Cookie Jars, Storage State and Persistence.

Clearing Storage Is Itself a Signal

The obvious evasion is to wipe everything between requests, and the obvious flaw is that a machine with empty cookies and empty storage and an empty cache has a state no real user reaches. Privacy tooling produces a subset: Firefox privacy.resistFingerprinting and Safari's Intelligent Tracking Prevention keep partitioning state consistent, but a scraper that calls context.clearCookies() and context.clearPermissions() between every navigation produces a client whose storage is born empty on every single request.

The distinction that matters to a scorer is between a first visit and a cleared client. Both answer 200. A first visit is common. A cleared client is a client whose only stable state was storage, and a fleet of them, each with a plausible fingerprint and a permanently empty origin state, is one of the densest clusters a WAF can find. Browser Pool Architecture exists because a warm pool that ages origins is worth more than a fresh context per call.

Partition Keys Change What a Cross-Site Collector Sees

Since Chrome 115 the Partitioned attribute and the CHIPS mechanism scope a cookie to the top-level site rather than the registrable domain, so a cookie set by tracker.example while embedded in shop.example is only readable again while the top-level site is shop.example. Firefox Total Cookie Protection and Safari ITP partition the same way, using the top-level site plus, in some builds, a daily rotation of the partition key.

The effect on a collector is structural rather than a matter of tuning. A third-party storage probe served from an ad domain and read from a different top-level site sees an empty store every time, regardless of what it wrote. The collector that wants durable state must therefore run first-party, from the origin it is measuring, or must live inside a partition the user actually visits. This is a genuine improvement for users and a hard constraint for anyone relying on cross-site storage, and it is the reason Defending Against Scrapers can rely on partitioned cookies for anything that is not a first-party session.

A service worker registration is the strongest single marker in this family, because it survives every clearing action short of an explicit unregister, it persists across tabs and restarts, and navigator.serviceWorker.controller is null on a first visit and non-null on a returning one. It is also the most expensive to fake correctly, since a page that registers a worker must then serve that worker's script from a path the collector controls.

Running the ETag Round Trip Locally

The mechanism is easy to demonstrate without a browser, because the conditional request is an ordinary HTTP request. The script below starts a ThreadingHTTPServer on a fixed port, derives an ETag from the request headers a real browser would send anyway, and then replays the sequence: a cold fetch, two revalidations from the same "device", one from a different "device", and one with a changed language to show the Vary multiplication.

import hashlib
import threading
from http.client import HTTPConnection
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

PIXEL = bytes([
    0x47, 0x49, 0x46, 0x38, 0x39, 0x61,   # GIF89a
    0x01, 0x00, 0x01, 0x00, 0x80, 0x00, 0x00,   # 1x1 screen descriptor
    0x00, 0x00, 0x00, 0xFF, 0xFF, 0xFF,   # black / white colour table
    0x21, 0xF9, 0x04, 0x01, 0x00, 0x00, 0x00, 0x00,   # graphic control
    0x2C, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00,   # image desc
    0x02, 0x02, 0x44, 0x01, 0x00,         # LZW-compressed pixel
    0x3B,                                # trailer
])

FEATURE_HEADERS = ("Sec-CH-UA-Platform", "Sec-CH-UA-Mobile", "Accept-Language")


class PixelHandler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def log_message(self, *args):
        pass

    def do_GET(self):
        features = "|".join(self.headers.get(h, "") for h in FEATURE_HEADERS)
        etag = '"%s"' % hashlib.sha256(features.encode("utf-8")).hexdigest()[:16]
        vary = ", ".join(FEATURE_HEADERS)
        if self.headers.get("If-None-Match") == etag:
            self.send_response(304)
            self.send_header("ETag", etag)
            self.send_header("Vary", vary)
            self.send_header("Cache-Control", "no-cache")
            self.send_header("Content-Length", "0")
            self.end_headers()
            return
        self.send_response(200)
        self.send_header("Content-Type", "image/gif")
        self.send_header("Content-Length", str(len(PIXEL)))
        self.send_header("ETag", etag)
        self.send_header("Vary", vary)
        self.send_header("Cache-Control", "no-cache")
        self.end_headers()
        self.wfile.write(PIXEL)


def fetch(port, headers, if_none_match=None):
    sent = dict(headers)
    if if_none_match:
        sent["If-None-Match"] = if_none_match
    conn = HTTPConnection("127.0.0.1", port, timeout=5)
    conn.request("GET", "/px.gif", headers=sent)
    resp = conn.getresponse()
    body = resp.read()
    result = (resp.status, resp.getheader("ETag"), len(body))
    conn.close()
    return result


WINDOWS = {"Sec-CH-UA-Platform": "Windows", "Sec-CH-UA-Mobile": "?0",
           "Accept-Language": "en-GB,en;q=0.9"}
MACOS = {"Sec-CH-UA-Platform": "macOS", "Sec-CH-UA-Mobile": "?0",
         "Accept-Language": "en-US,en;q=0.9"}

server = ThreadingHTTPServer(("127.0.0.1", 8931), PixelHandler)
port = server.server_address[1]
threading.Thread(target=server.serve_forever, daemon=True).start()

print("serving /px.gif on 127.0.0.1:%d, ETag = sha256(platform|mobile|language)" % port)
print("Vary: %s   Cache-Control: no-cache on every response" % ", ".join(FEATURE_HEADERS))
print()
print("%-22s %-20s %-7s %s" % ("client", "If-None-Match sent", "status", "ETag returned"))
print("-" * 78)

status, etag, size = fetch(port, WINDOWS)
print("%-22s %-20s %-7d %s" % ("windows laptop, hit 1", "(none, cold cache)", status, etag))
for n in (2, 3):
    status, seen, size = fetch(port, WINDOWS, if_none_match=etag)
    print("%-22s %-20s %-7d %s" % ("windows laptop, hit %d" % n, etag, status, seen))
status, mac_tag, size = fetch(port, MACOS, if_none_match=etag)
print("%-22s %-20s %-7d %s" % ("macos laptop, hit 1", etag, status, mac_tag))
status, seen, size = fetch(port, MACOS, if_none_match=mac_tag)
print("%-22s %-20s %-7d %s" % ("macos laptop, hit 2", mac_tag, status, seen))
GERMAN = dict(WINDOWS, **{"Accept-Language": "de-DE,de;q=0.9"})
status, de_tag, size = fetch(port, GERMAN, if_none_match=etag)
print("%-22s %-20s %-7d %s" % ("windows, German UI", etag, status, de_tag))
print("-" * 78)
print("cache entries this one URL now holds, keyed by Vary: 3")
print("body bytes: cold hit %d, revalidation 0, cache miss %d" % (len(PIXEL), len(PIXEL)))
server.shutdown()

serving /px.gif on 127.0.0.1:8931, ETag = sha256(platform|mobile|language)
Vary: Sec-CH-UA-Platform, Sec-CH-UA-Mobile, Accept-Language   Cache-Control: no-cache on every response

client                 If-None-Match sent   status  ETag returned
------------------------------------------------------------------------------
windows laptop, hit 1  (none, cold cache)   200     "54752130e6b83cd3"
windows laptop, hit 2  "54752130e6b83cd3"   304     "54752130e6b83cd3"
windows laptop, hit 3  "54752130e6b83cd3"   304     "54752130e6b83cd3"
macos laptop, hit 1    "54752130e6b83cd3"   200     "c7c469d82b601f8a"
macos laptop, hit 2    "c7c469d82b601f8a"   304     "c7c469d82b601f8a"
windows, German UI     "54752130e6b83cd3"   200     "b480a337b4af4240"
------------------------------------------------------------------------------
cache entries this one URL now holds, keyed by Vary: 3
body bytes: cold hit 42, revalidation 0, cache miss 42

Rows 2 and 3 are the identifier: the client volunteered a token nobody asked it to send, and the token matched. Row 4 is a different device, and the only evidence is the status code. Row 6 shows the Vary multiplication: the same machine with a different Accept-Language holds a different cache entry, so the server learns a history of language settings rather than a current one. Note that Cache-Control: no-cache is present on every response and does nothing about any of this.

Failure Modes

• Reusing one If-None-Match token everywhere. If your client always sends the same validator, the server learns you have one device, which is a stronger statement than being one device.
• Treating a 200 as a block. A 200 on a cacheable URL means the entry was evicted or the token changed. It is not a rate-limit response, and retry logic built on that assumption will hammer the endpoint.
• Vary: * on anything. It disables caching entirely and quietly removes the mechanism you were relying on.
• Disabling the HTTP cache to "look fresh". Chromium's --disable-http-cache or a Cache-Control: no-store on your side means no validator is ever stored, so every request is a 200 and your client joins the cleared-storage cluster.
• Expecting localStorage to survive a fresh context. It does not. storage_state round-tripping is the mechanism that makes it survive, and it is a different file format from a cookie jar.
• Counting default entries as devices. Chrome synthesises one default and one audiooutput per kind, so a naive enumerateDevices().length overstates the hardware by two.

Defensive Checklist

• No ETag on any response is derived from a client-observable feature; validators describe the representation, nothing else.
• Static assets ship Cache-Control with a real max-age and a content hash in the URL, not a feature hash in the ETag.
• Vary names only headers that genuinely change the representation, so the cache key stays small.
• Client hints are not used as a cache key on a pixel that a third party can request.
• If you must support users who clear storage, the site does not treat an empty origin state as a new device: it re-establishes identity through an explicit sign-in, which is the only identifier a user can revoke.
• A first-party Clear my data control removes cookies, all origin storage, service workers and the HTTP cache entries for the site, and the behaviour is documented.

The Legitimate Route

This technique is a published research finding and it is in every commercial fingerprinting product, which is the argument for it existing at all. What is not acceptable is applying it to people who never consented, which in practice means turning off the controls the browser gives them, or tracking across sites through a partition the user chose to isolate. The legitimate uses are your own first-party session continuity on a site you operate, and the defensive measurement described above: run the round trip against your own domain, confirm that a cleared client and a returning client are indistinguishable from your validators, and fix the server if they are not.