If the Site Remembers You, So Should You

A browser is a state machine: every page visit can add a cookie, and the next request proves your history. Scrapers that treat each request as an isolated event miss half of what the site remembers and get flagged for it. This lesson is the mechanics of that state — what a cookie is, how requests manages them, and how to make a multi-day crawl act like one continuous visitor.

What a Cookie Actually Is

A cookie is a small key=value the server sends in a Set-Cookie response header, and the client repeats back in a Cookie request header on subsequent requests to that domain. Attribute fields decide its lifespan and reach: Max-Age/Expires control how long it lives (no expiry means it only survives to the end of the "session"), Domain and Path control which URLs it is sent to, Secure restricts it to HTTPS, HttpOnly hides it from JavaScript, and SameSite=Lax/Strict/None controls third-party sending. A generated session id or a tracking id is the payload.

import requests
r = requests.get("https://example.com/")
print(r.cookies)              # the cookies that just landed
r2 = requests.get("https://example.com/next")  # new session: cookies lost!

That second line is the trap: a bare requests.get creates a fresh client each call, so cookies from r never travel with r2. Every request looks like a brand-new visitor.

The Session Object Is a Cookie Jar

requests.Session() persists cookies (and connection pooling, and one headers dict) across calls, which is exactly the "one continuous visitor" behavior:

s = requests.Session()
s.headers.update({"User-Agent": "Mozilla/5.0 ..."})
s.get("https://example.com/login")      # server sets a session cookie
s.get("https://example.com/dashboard")  # cookie is sent automatically

Use one session per logical user per crawl — never one Session shared across parallel threads without thought, and never a fresh session per URL. You can also set cookies manually when you hold them from a prior run (for example, copying a sessionid out of the browser): s.cookies.set("sessionid", token, domain="example.com").

Survive Restarts: Persist the Jar

A session object only lives as long as your process. To log in on day one and keep working on day three, serialize and reload the cookie jar. pickle a requests.cookies.RequestsCookieJar or store the raw pairs in a JSON file:

import json, requests

# save
jar = [{"name": c.name, "value": c.value,
        "domain": c.domain, "path": c.path}
       for c in s.cookies]

# reload (or use pickle.dumps(s.cookies) / loads())
s2 = requests.Session()
for c in jar:
    s2.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])

Store the file outside the target's reach (cookies are credentials) and verify liveness before every run: if a reloaded session gets a 302 back to the login page, the token expired — re-auth and re-save. This is the authentication-sessions lesson, with the storage half made explicit.

For Browser-Relevant State: LocalStorage

When a site's fetch/XHR carries Authorization tokens in a header and keeps them in localStorage (invisible to plain requests), replicate the header directly — which is the header the API actually checks — instead of pretending to have logged in via the browser. Playwright can also seed state before a run:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    b = p.chromium.launch()
    page = b.new_page()
    page.goto("https://example.com/")
    page.evaluate("() => localStorage.setItem('token', 'abc123')")
    page.goto("https://example.com/app")   # app now sees you as authed
    b.close()

Cookie jars, session reuse and persisted state turn a flock of anonymous requests into one identifiable visitor — which is both more polite and more likely to be let in.