requests, Sessions & Headers

State Management in Web Scraping#

HTTP itself is stateless: each request stands alone. Web applications build state on top of it with cookies, which carry session IDs, login state, CSRF tokens, consent choices, A/B test buckets, and anti-bot clearance. Calling requests.get() over and over throws all of that away and opens a fresh TCP and TLS connection for every URL. To the server you look like thousands of unrelated first-time visitors, and to your own machine it is slow.

The fix in Python is a session object: requests.Session() or its equivalent httpx.Client().

What a Session Gives You#

1. Cookie persistence. Every Set-Cookie the server sends is stored in session.cookies and sent back on later requests that match its domain and path.
2. Connection pooling. The underlying TCP/TLS connection is kept alive and reused. For many small requests to one host, skipping the handshake often cuts per-request time noticeably.
3. Shared defaults. Headers, query parameters, auth, proxies, and certificate settings are configured once and applied everywhere.

You can see the pooling effect yourself:

import time
import requests

URL = "https://example.com/"

start = time.perf_counter()
for _ in range(10):
    requests.get(URL, timeout=10)
print(f"no session: {time.perf_counter() - start:.2f}s")

start = time.perf_counter()
with requests.Session() as s:
    for _ in range(10):
        s.get(URL, timeout=10)
print(f"session:    {time.perf_counter() - start:.2f}s")

On a typical connection the session loop finishes in a fraction of the time, because only the first request pays for DNS, TCP, and TLS. The exact numbers depend on your latency to the server.

Configure the Session in One Place#

The session is your scraper's identity and connection budget, so build it with a single factory function instead of sprinkling settings around:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def make_session() -> requests.Session:
    s = requests.Session()
    s.headers.update({
        "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
    })
    retry = Retry(
        total=4,
        backoff_factor=1.0,
        status_forcelist=[429, 500, 502, 503, 504],
        allowed_methods=["GET", "HEAD"],
        respect_retry_after_header=True,
    )
    adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=10)
    s.mount("https://", adapter)
    s.mount("http://", adapter)
    s.trust_env = False   # ignore HTTP_PROXY etc. from the environment
    return s

The mounted HTTPAdapter is where connection pool size and transport-level retries live. pool_maxsize matters once you use threads: if ten threads share a session with the default pool of ten, an eleventh concurrent request makes urllib3 open a throwaway connection and log "Connection pool is full, discarding connection".

trust_env = False stops requests from silently picking up proxy variables and .netrc credentials from the machine it runs on, which is a common source of "works on my laptop, fails on the server".

The Timeout Trap#

There is no working session-wide timeout in requests. Writing session.timeout = 10 sets an attribute nothing reads, and every request can still hang forever. Pass timeout= on each call, or wrap the session:

class TimeoutSession(requests.Session):
    def request(self, *args, **kwargs):
        kwargs.setdefault("timeout", (3.05, 20))
        return super().request(*args, **kwargs)

httpx.Client does accept a client-level timeout and defaults to five seconds, which is one reason people migrate.

Logging In and Proving It Worked#

Many scrapers need to log in first. With a session, the auth cookie from the login response is stored automatically. The part beginners skip is verifying the login, because failed logins usually return 200 with the login form again, not an error.

import os
from bs4 import BeautifulSoup

session = make_session()

# 1. GET the form page to receive the session cookie and CSRF token
form_page = session.get("https://example.com/login", timeout=15)
soup = BeautifulSoup(form_page.text, "html.parser")
token_input = soup.select_one('input[name="csrf_token"]')
if token_input is None:
    raise RuntimeError("login form changed: no csrf_token field")

# 2. POST credentials plus the token, as form data (data=, not json=)
resp = session.post(
    "https://example.com/login",
    data={
        "username": os.environ["SCRAPER_USER"],
        "password": os.environ["SCRAPER_PASS"],
        "csrf_token": token_input["value"],
    },
    headers={"Referer": "https://example.com/login"},
    timeout=15,
)

# 3. Verify with a signal that only exists when logged in
if "Log out" not in resp.text:
    raise RuntimeError(f"login failed, landed on {resp.url}")
print("cookies now held:", sorted(session.cookies.keys()))

Details that matter:

• Frameworks such as Django, Rails, and Laravel reject a POST whose CSRF token does not match the one tied to your session cookie. The GET and POST must go through the same session, or the token is useless.
• An HTML form sends application/x-www-form-urlencoded data, so use data=. A JavaScript login that calls a JSON API needs json=. Check the request in DevTools to see which one the site uses.
• Some frameworks also check Referer or Origin on POSTs.
• Credentials come from environment variables or a secrets manager, never from source code you might commit.

Only log in to accounts you own and are permitted to automate. Logging in usually means accepting terms of service, which the legal considerations lesson explains can change your position considerably.

Inspecting and Persisting Cookies#

When a session misbehaves, look at the jar:

for c in session.cookies:
    print(c.name, c.domain, c.path, "secure" if c.secure else "", c.expires)

A cookie scoped to .example.com is sent to every subdomain; one scoped to www.example.com is not sent to api.example.com. Many "logged in on the site but 401 on the API" problems are exactly this.

Sessions do not survive a process restart. To avoid logging in on every run, persist the jar and reload it, then check it is still valid before using it:

import pickle
from pathlib import Path

JAR = Path("session_cookies.pkl")

def save_cookies(s):
    JAR.write_bytes(pickle.dumps(s.cookies))

def load_cookies(s):
    if JAR.exists():
        s.cookies.update(pickle.loads(JAR.read_bytes()))

Treat that file like a password: it is one. Only unpickle files you wrote yourself, since unpickling untrusted data can execute code. The cookies and state and authentication sessions lessons go further into expiry and refresh.

Hooks for Cross-Cutting Checks#

Session hooks run on every response, which is a tidy place for logging and failing fast:

def check(resp, *args, **kwargs):
    print(resp.status_code, resp.request.method, resp.url, f"{resp.elapsed.total_seconds():.2f}s")
    if resp.status_code in (401, 403):
        raise RuntimeError(f"access denied at {resp.url}; stop and investigate")

session.hooks["response"].append(check)

When the Browser Works and the Session Does Not#

Walk through the differences in this order, because it matches how often each one is the cause:

1. A cookie set by JavaScript. Some cookies are created by scripts (document.cookie = ...), never by a Set-Cookie header, so your session never receives them. Compare session.cookies.keys() with the Application tab in DevTools.
2. A missing request step. The browser loaded a page, an API bootstrap call, or a consent endpoint before the request you copied. Record the full sequence in the Network tab with "Preserve log" enabled and replay it in order.
3. Cookie scope. The cookie exists but is scoped to another subdomain or path, as described above.
4. Headers the endpoint checks, such as Referer, Origin, X-Requested-With, or a CSRF header copied from a cookie value.

Change one thing at a time and re-test; changing four at once teaches you nothing about which one mattered.

requests.Session vs httpx.Client#

For a small synchronous scraper either is fine. Once you want asyncio, move to httpx.AsyncClient, covered in concurrency.

What a Session Does Not Give You#

A session does not check robots.txt, does not throttle, does not rotate IPs, does not validate that a 200 contains real data, and does not persist anything to disk. Each gap has its own lesson. Sessions make the request layer correct; the rest of the course makes the crawl dependable.