Cookies, Sessions & Persistent State

If the Site Remembers You, So Should You#

HTTP itself is stateless: every request stands alone. Websites build memory on top of it, mostly with cookies, and increasingly with tokens kept in browser storage. A scraper that ignores that memory looks like a brand-new visitor on every request, loses logins, gets different content than a browser would, and trips defences that expect a continuous visit. This lesson covers how cookies work in detail, why a cookie you expected is sometimes not sent, how to persist state across runs, and how to move state between a real browser and a plain HTTP client.

What a Cookie Actually Is#

The server sets a cookie with a Set-Cookie response header. The client stores it and sends it back in the Cookie request header on later requests that match its scope. One response can set several cookies:

HTTP/1.1 200 OK
Set-Cookie: sessionid=9f2c1ab7; Path=/; HttpOnly; Secure; SameSite=Lax
Set-Cookie: region=eu; Domain=example.com; Path=/; Max-Age=2592000
Set-Cookie: __Host-csrf=Zk31x; Path=/; Secure

On the next request to a matching URL, the client sends only names and values:

GET /account HTTP/1.1
Host: www.example.com
Cookie: sessionid=9f2c1ab7; region=eu; __Host-csrf=Zk31x

The attributes decide where and how long a cookie applies:

Names starting with __Host- must be Secure, have Path=/ and no Domain; names starting with __Secure- must be Secure. The browser-side rules are covered in HTTP cookies and sessions.

The Session Object Is a Cookie Jar#

The most common beginner bug: module-level calls like requests.get() start with an empty jar every time, so cookies from one response are never sent with the next.

import requests

s = requests.Session()
s.headers["User-Agent"] = "research-bot/1.0 (+contact@example.org)"
s.get("https://www.example.com/")                # server sets sessionid and region
r = s.get("https://www.example.com/products")    # both cookies sent automatically
print(r.request.headers.get("Cookie"))

sessionid=9f2c1ab7; region=eu

r.request.headers shows exactly what was sent, which is the first thing to check when debugging. The httpx.Client behaves the same way; see requests and sessions for the other things a session object keeps.

Use one session per identity. Sharing a single session between unrelated crawl workers mixes their state (one worker's region choice changes what another sees), and requests does not guarantee Session is thread-safe. Give each thread or worker its own session. If you route traffic through proxies, keep each session on the same exit IP for its lifetime: many sites bind a session to the IP it started from, as described in proxies and session stickiness.

Why Isn't My Cookie Being Sent?#

When a browser works and your scraper does not, trace the cookies hop by hop, including redirects, where login flows usually set their session cookie:

r = s.get("https://www.example.com/login-callback?code=abc")
for hop in r.history + [r]:
    print(hop.status_code, hop.url)
    print("   set-cookie:", hop.headers.get("set-cookie"))

for c in s.cookies:
    print(f"{c.domain:22} {c.path:8} {c.name:14} secure={c.secure} expires={c.expires}")

In requests, multiple Set-Cookie headers are joined with commas in headers.get(), which is fine for reading by eye; in httpx, use r.headers.get_list("set-cookie") to see them separately. Compare the jar against the browser's view in DevTools (Application tab, then Cookies). The usual culprits:

1. Scope mismatch. The cookie is host-only for www.example.com and you are calling api.example.com, or its Path does not cover the URL.
2. Set by JavaScript. The page runs document.cookie = "..." after computing a value. No Set-Cookie header exists, so an HTTP client never receives it. Search the page's scripts for document.cookie, then reproduce the value or obtain it from a browser.
3. Secure over HTTP. The request went to an http:// URL, often from a stale link or a redirect.
4. Expired or replaced. The server rotated the session ID on login, and you are still sending the old one you set manually.
5. Bound to other signals. The cookie is valid only with the same User-Agent, IP or TLS fingerprint that obtained it. Copying a cookie from your browser into a script with a different identity often fails for this reason.

Persist the Jar Across Runs#

A session lives only as long as your process. To log in once and keep working on later runs, save the jar to disk. The standard library's MozillaCookieJar reads and writes the Netscape cookies.txt format (also understood by curl and wget), and requests accepts it as a session jar:

import os
from http.cookiejar import MozillaCookieJar, LoadError
import requests

COOKIE_FILE = "state/example_cookies.txt"

s = requests.Session()
jar = MozillaCookieJar(COOKIE_FILE)
try:
    jar.load(ignore_discard=True)
except (FileNotFoundError, LoadError):
    pass
s.cookies = jar

# ... crawl ...

os.makedirs("state", exist_ok=True)
jar.save(ignore_discard=True)
os.chmod(COOKIE_FILE, 0o600)

ignore_discard=True is the line people miss. Session cookies (no Expires) are marked "discard", and by default save() skips them. Most login session cookies are exactly that kind, so without the flag your saved file is missing the one cookie that matters, and the reloaded session is logged out.

Cookies are credentials. Store the file outside the repository, restrict its permissions (0o600 means only your user can read it), and never paste it into tickets or logs.

Check the Session Before Trusting It#

A reloaded session might have expired server-side. Test it with one cheap request before starting a crawl, and re-authenticate if it fails:

def session_is_alive(s) -> bool:
    r = s.get("https://www.example.com/account", allow_redirects=False, timeout=15)
    if r.status_code in (301, 302, 303, 307) and "login" in r.headers.get("location", ""):
        return False
    return r.status_code == 200 and "Sign out" in r.text

if not session_is_alive(s):
    log_in(s)             # your login routine from the authentication lesson
    jar.save(ignore_discard=True)

allow_redirects=False makes an expired session visible as a redirect to the login page instead of a 200 response containing the login form. Checking for something only a logged-in page contains ("Sign out", the account name) catches sites that return 200 either way. Logging in itself, including CSRF tokens, is covered in authentication and sessions.

Importing Cookies From a Browser#

For a quick experiment on your own account, you can copy the Cookie request header from DevTools and load it:

from http.cookies import SimpleCookie

raw = "sessionid=9f2c1ab7; region=eu"
parsed = SimpleCookie()
parsed.load(raw)
for name, morsel in parsed.items():
    s.cookies.set(name, morsel.value, domain="www.example.com", path="/")

This loses the real domain, path and expiry of each cookie, so it is fine for testing and fragile for production.

Moving State Between a Browser and HTTP Client#

A common production pattern is to let a real browser handle the part that needs JavaScript (a login flow or a first-visit check), then continue with a fast HTTP client. Playwright can save and restore a whole browser context's state, including cookies and localStorage:

from playwright.sync_api import sync_playwright
import requests

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context()
    page = context.new_page()
    page.goto("https://www.example.com/login")
    # ... perform the login in the page ...
    context.storage_state(path="state/browser_state.json")   # reuse later: new_context(storage_state=...)
    cookies = context.cookies()
    ua = page.evaluate("navigator.userAgent")
    browser.close()

s = requests.Session()
s.headers["User-Agent"] = ua                   # keep the identity consistent
for c in cookies:
    s.cookies.set(
        c["name"], c["value"],
        domain=c["domain"], path=c["path"], secure=c["secure"],
        expires=int(c["expires"]) if c["expires"] > 0 else None,   # -1 means session cookie
    )

Carry the same User-Agent over, since some sites bind the session to it. The full pattern, including when to hand control back to the browser, is in browser session handoff.

State That Isn't a Cookie#

Single-page apps often keep an access token in localStorage and send it as an Authorization: Bearer ... header. A plain HTTP client never sees localStorage, so find the header in the Network tab and set it on your session directly. Such tokens usually expire within minutes to hours and come with a refresh token and a refresh endpoint; call it when you get a 401 rather than logging in again.

Other state lives in the page itself: CSRF tokens in hidden form fields or meta tags, and server-side form state in hidden inputs. Those must be read from the latest page and sent back with the next request, as covered in form submission and multi-step flows.

Practice#

Log in to a practice site built for scraping exercises using a requests.Session. Save the jar with and without ignore_discard=True, reload each in a new process, and use session_is_alive to show which one is still logged in. Then print the cookie table and identify which cookies are session cookies.