Logins, Tokens & Long-Lived Sessions

The Two Faces of Authentication#

Plenty of useful data sits behind a login: your own orders, dashboards you have access to, a partner portal your company pays for. Scraping it is legitimate only with an account you are allowed to use and within that site's terms; logged-in scraping is also where terms of service are most explicit, so read them first (see legal considerations). Never scrape with someone else's credentials or share session cookies.

Servers recognise a logged-in client in two ways:

Many real sites are hybrids: a cookie session for the web pages and a token for the API they call. The discipline is the same for all of them: know exactly what authenticates you, keep it valid for the whole run, and persist it so the next run starts authenticated. The underlying mechanics are covered in HTTP cookies and sessions.

Logging In the Cookie Way, With CSRF#

Most cookie logins include a CSRF token in the login form. The server rejects submissions without the token that belongs to the current pre-login session, so the flow is always three steps: load the login page, read the token, and submit it with the credentials from the same client.

import os
import httpx
from lxml import html as lh

client = httpx.Client(base_url="https://portal.example.com", timeout=15.0,
                      follow_redirects=True)

# 1. GET the login page: sets the pre-login cookie and renders the token
page = client.get("/login")
tree = lh.fromstring(page.text)
form = tree.xpath('//form[.//input[@type="password"]]')[0]
fields = {i.get("name"): i.get("value", "")
          for i in form.xpath(".//input[@type='hidden']") if i.get("name")}

# 2. POST credentials plus every hidden field, from the same client
fields.update({"email": os.environ["PORTAL_USER"],
               "password": os.environ["PORTAL_PASS"]})
resp = client.post(form.get("action") or "/login", data=fields)

# 3. Verify: status codes lie, content does not
if "/login" in str(resp.url) or 'type="password"' in resp.text:
    raise RuntimeError("login failed: still on the login form")
print("logged in, cookies:", [c.name for c in client.cookies.jar])

Notes on the lines that matter:

• Copy all hidden inputs, not just the one named csrf. Forms often carry several (a token, a redirect target, a form id), and the field name differs between frameworks. Parsing the form generically survives renames; see form submission for the full technique.
• Credentials come from environment variables or a secrets manager, never from the source file, which ends up in version control.
• A failed login usually returns 200. The server re-renders the form with an error message. Check where you landed and what the page contains.
• follow_redirects=True is not the default in httpx. Without it, the successful 302 after login is returned as-is and you would need to follow it yourself.

The Client is what makes it work: it stores every Set-Cookie and sends cookies back on later requests to matching domains and paths. Keep exactly one client per identity and never share it between accounts.

Persist the Session Across Runs#

Logging in on every start is slow, and repeated fresh logins from a server are exactly what "suspicious sign-in" detection looks for. Save the cookie jar with its domain and path, not a flat dict, because two cookies can share a name across subdomains:

import json

def save_cookies(client, path="session.json"):
    data = [{"name": c.name, "value": c.value, "domain": c.domain,
             "path": c.path, "expires": c.expires}
            for c in client.cookies.jar]
    with open(path, "w") as f:
        json.dump(data, f)

def load_cookies(client, path="session.json"):
    with open(path) as f:
        for c in json.load(f):
            client.cookies.set(c["name"], c["value"],
                               domain=c["domain"], path=c["path"])

Restoring is not proof of being logged in. Sessions expire, get revoked when you change your password, or are invalidated by the server after inactivity. Probe cheaply before the real work:

def is_logged_in(client):
    r = client.get("/account", follow_redirects=False)
    return r.status_code == 200 and 'type="password"' not in r.text

follow_redirects=False is deliberate here: an expired session typically answers with a 302 to /login, and you want to see that redirect instead of silently following it to a 200 login page. If the probe fails, run the full login once and save again. Store session.json with restrictive file permissions; it is equivalent to a password until it expires.

Access Tokens and Refreshing#

Token APIs normally give you two tokens. The access token is short-lived and goes in the Authorization header. The refresh token lives longer and is only ever sent to the token endpoint to obtain a new access token.

A JWT's payload is base64url-encoded JSON, readable without the signing key. You cannot forge or verify it without the key, but you can read the expiry and refresh before it lapses instead of waiting for a 401:

import base64
import time

def jwt_exp(token):
    payload = token.split(".")[1]
    payload += "=" * (-len(payload) % 4)          # restore stripped padding
    return json.loads(base64.urlsafe_b64decode(payload))["exp"]

print(time.strftime("%H:%M:%S", time.localtime(jwt_exp(tokens["access"]))))

Put all refresh logic in one wrapper so no caller ever handles tokens directly:

import threading

class TokenAuth:
    def __init__(self, client, token_url, access, refresh):
        self.client, self.token_url = client, token_url
        self.access, self.refresh = access, refresh
        self.lock = threading.Lock()

    def _refresh(self):
        r = self.client.post(self.token_url, data={
            "grant_type": "refresh_token", "refresh_token": self.refresh})
        if r.status_code in (400, 401):
            raise PermissionError("refresh token rejected: full login needed")
        r.raise_for_status()
        body = r.json()
        self.access = body["access_token"]
        # rotating servers issue a new refresh token and revoke the old one
        self.refresh = body.get("refresh_token", self.refresh)

    def get(self, url, **kw):
        with self.lock:
            if jwt_exp(self.access) - time.time() < 60:
                self._refresh()
            token = self.access
        r = self.client.get(url, headers={"Authorization": f"Bearer {token}"}, **kw)
        if r.status_code == 401:
            with self.lock:
                if token == self.access:        # nobody refreshed meanwhile
                    self._refresh()
            r = self.client.get(url, headers={"Authorization": f"Bearer {self.access}"}, **kw)
        return r

Three production lessons are encoded here:

• Refresh token rotation. Many servers return a new refresh token on every refresh and invalidate the old one. If you do not persist the new one, the next run fails with an invalid grant. Save tokens to disk after each refresh.
• The lock. With concurrent workers, several requests hit 401 at the same moment and all try to refresh. With rotation, the second refresh uses an already-revoked token and kills the session. The token == self.access check makes only the first caller refresh.
• The grant format shown is the standard OAuth 2.0 refresh request (form-encoded). Some private APIs use JSON bodies or different field names; copy exactly what the site's own app sends, as seen in DevTools.

Log In With a Browser When There Is No Clean Endpoint#

Some logins run through JavaScript, single sign-on redirects, or multi-factor prompts that are impractical to reproduce with raw HTTP. Run the login once in Playwright, then move the cookies into your HTTP client:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)     # watch it the first time
    context = browser.new_context()
    page = context.new_page()
    page.goto("https://app.example.com/login")
    page.get_by_label("Email").fill(os.environ["APP_USER"])
    page.get_by_label("Password").fill(os.environ["APP_PASS"])
    page.get_by_role("button", name="Sign in").click()
    page.wait_for_url("**/dashboard")
    cookies = context.cookies()
    user_agent = page.evaluate("navigator.userAgent")
    browser.close()

client = httpx.Client(headers={"User-Agent": user_agent}, timeout=15.0)
for c in cookies:
    client.cookies.set(c["name"], c["value"], domain=c["domain"], path=c["path"])

Setting cookies with their domains (rather than building one Cookie header string) lets the client send each cookie only where it belongs, and reusing the browser's user agent keeps the session looking like the same client that logged in. Some sites bind sessions to more than cookies, such as a token in localStorage that the app copies into a header. If requests fail after the export, check DevTools for an Authorization or custom header on the app's own API calls.

For multi-factor authentication on your own account, the pyotp library generates time-based codes from the setup secret: pyotp.TOTP(secret).now(). Keep that secret in the same vault as the password.

Debugging Authentication Failures#

Log the response URL and status for every auth step, and client.cookies.jar after each one, and most of these become obvious in minutes.