Debugging with HAR and Mitmproxy

See What Actually Traveled#

Browser DevTools is enough for the first hour of investigating a target. Then you hit the wall it cannot cross: you need to replay a captured request with surgical edits, you need the headers and timings of hundreds of requests at once, you need to compare what your scraper sends against what the browser sends, or the target lives inside an app whose traffic DevTools cannot see at all. The professional toolkit for that wall is a HAR export for offline analysis and mitmproxy for live interception. Both record the requests that actually went over the wire, not the ones you think your code is sending.

HAR: The Page's HTTP Logbook#

A HAR (HTTP Archive) file is a JSON log of every request a page made and every response it received: URLs, methods, headers, cookies, query strings, POST bodies, timings and status codes. In Chrome or Firefox DevTools, open the Network tab, reproduce the action you care about, and use the export button (or right-click, "Save all as HAR"). Tick "Preserve log" first, otherwise a navigation or redirect wipes the requests you wanted.

The structure you will touch most often looks like this (trimmed):

{
  "log": {
    "entries": [
      {
        "startedDateTime": "2026-03-02T10:15:04.120Z",
        "time": 182.4,
        "request": {
          "method": "POST",
          "url": "https://api.example.com/v2/search?page=1",
          "headers": [{"name": "content-type", "value": "application/json"}],
          "queryString": [{"name": "page", "value": "1"}],
          "postData": {"mimeType": "application/x-www-form-urlencoded", "text": "q=lamps&sort=price"}
        },
        "response": {
          "status": 200,
          "content": {"size": 5120, "mimeType": "application/json", "text": "..."}
        },
        "timings": {"dns": 0, "connect": 0, "ssl": 0, "send": 0.2, "wait": 170.1, "receive": 12.1}
      }
    ]
  }
}

Because it is JSON, it is scriptable. You can filter to "only API calls" or "only 4xx and 5xx" in one pass, something the DevTools UI makes you click through by hand.

import json
from collections import Counter
from urllib.parse import urlsplit

with open("capture.har", encoding="utf-8") as f:
    har = json.load(f)

entries = har["log"]["entries"]
hosts = Counter(urlsplit(e["request"]["url"]).hostname for e in entries)
print(hosts.most_common(5))

for e in entries:
    req, resp = e["request"], e["response"]
    mime = resp.get("content", {}).get("mimeType", "")
    if "json" in mime or resp["status"] >= 400:
        print(resp["status"], req["method"], req["url"][:90], f"{e['time']:.0f} ms")

Typical output tells you in seconds where the data lives and which calls failed:

[('www.example.com', 41), ('cdn.example.com', 38), ('api.example.com', 9), ...]
200 POST https://api.example.com/v2/search?page=1 182 ms
200 GET https://api.example.com/v2/facets?cat=12 95 ms
429 GET https://api.example.com/v2/search?page=2 12 ms

The power of a HAR is asking questions over the whole session instead of the one request you happened to click on. How many distinct API hosts did this page hit? Which request set the cookie that every later call depends on? Which POST body did the form submit right before the server rejected it? The timings block also helps diagnose slowness: a large wait means the server is slow to generate the response, while large dns, connect or ssl values point at connection setup, which a reused session removes.

Replay From a HAR#

Once you have filtered the API entries, replay them with httpx to test the hidden-API strategy from XHR and API reverse engineering before writing any scraper. The goal is to learn which headers matter, so start with everything the browser sent and remove things until the request breaks.

import httpx

SKIP = {"host", "content-length", "connection", "accept-encoding", "cookie"}

def to_httpx_args(entry):
    req = entry["request"]
    headers = {
        h["name"]: h["value"]
        for h in req["headers"]
        if not h["name"].startswith(":") and h["name"].lower() not in SKIP
    }
    body = req.get("postData", {}).get("text")
    return req["method"], req["url"], headers, body

client = httpx.Client(timeout=15.0, follow_redirects=True)
target = next(e for e in entries if "/v2/search" in e["request"]["url"])
method, url, headers, body = to_httpx_args(target)

resp = client.request(method, url, headers=headers, content=body)
print(resp.status_code, resp.headers.get("content-type"), len(resp.content))

The skipped headers are deliberate. HTTP/2 captures include pseudo-headers such as :authority and :path, which are not real header names and must be dropped. host and content-length are computed by httpx from the URL and body, and sending stale copies can make the request malformed. accept-encoding is left to the client so it only advertises encodings it can decode. Cookies are excluded because you want your own session's cookies, set by a real login or landing request, not a copy of the browser's.

When the replay returns 200 with the same JSON, you have proven the endpoint. Then remove headers one at a time: most APIs need only a handful (often an auth token, an x- custom header, and a content type). If a header you never noticed turns out to be required, that is the thing your scraper must reproduce or generate.

Two warnings about HARs. First, they contain live session cookies and bearer tokens, so treat them like passwords: do not commit them or paste them into tickets. Second, some browser versions offer a sanitised export that strips cookies and authorization headers. If a replay fails with 401 while the browser succeeded, check whether the auth headers are even present in the file.

Mitmproxy: Sit in the Middle#

Where a HAR reads after the fact, mitmproxy watches and changes traffic live. It runs as an HTTP(S) proxy on your machine in three flavours: mitmproxy (interactive terminal UI), mitmweb (browser UI) and mitmdump (headless, for scripts). To read HTTPS it generates its own certificate authority on first run, stored under ~/.mitmproxy/. The client must trust that CA, which you only ever do on your own devices and test profiles.

pip install mitmproxy
mitmweb --listen-port 8080          # UI at http://127.0.0.1:8081
mitmdump -w flows.mitm              # record everything to a file
mitmdump -nr flows.mitm             # read a recording back, no proxy

The feature that makes it a scraping tool is the add-on: a plain Python file whose functions are called on events such as request and response.

# log_json.py  -- run with: mitmdump -s log_json.py
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    # experiment: does the API still answer without this header?
    if flow.request.host == "api.example.com":
        flow.request.headers.pop("x-client-version", None)

def response(flow: http.HTTPFlow) -> None:
    ctype = flow.response.headers.get("content-type", "")
    if "application/json" in ctype:
        path = flow.request.path.split("?")[0]
        print(flow.response.status_code, flow.request.host, path, len(flow.response.content))

Add-ons can also modify traffic. A request hook can strip a header to test whether the server needs it; a response hook can save every JSON body to disk as a fixture for your parser tests. That second use pairs well with caching and resume: recorded responses become a regression suite you can run without touching the site.

Diff Your Scraper Against the Browser#

The most valuable debugging session with mitmproxy is running the browser and your scraper through the same proxy and comparing the two requests side by side. Point your client at the proxy and tell it to trust the mitmproxy CA:

import ssl
import httpx
from pathlib import Path

ca = Path.home() / ".mitmproxy" / "mitmproxy-ca-cert.pem"
ctx = ssl.create_default_context(cafile=str(ca))

with httpx.Client(proxy="http://127.0.0.1:8080", verify=ctx, timeout=20) as client:
    client.get("https://api.example.com/v2/search", params={"page": 1})

Now both requests appear in mitmweb. Differences you will typically find:

The last row matters: mitmproxy terminates TLS itself and opens its own upstream connection, so the server sees mitmproxy's TLS handshake, not your client's. A proxy is the right tool for comparing headers and bodies, but it hides fingerprint differences. Those belong to TLS fingerprinting. If a site behaves differently only when traffic goes through the proxy, that is itself a clue.

When the Browser Can't Be Your Eyes#

DevTools cannot inspect a desktop app, an embedded webview, or a phone app. mitmproxy can, as long as the device routes traffic through it and trusts its CA. Set the device's Wi-Fi proxy to your machine's IP and port 8080, then visit http://mitm.it from the device to install the certificate. Two limits to expect: modern Android apps do not trust user-installed CAs by default, and apps that use certificate pinning refuse the proxy entirely, so you will see connection errors instead of traffic. Handling those is covered in mobile API reverse engineering. Stay within the app's terms and the course's legal guidance while you do it.

Once you can see the requests, the workflow is identical to the browser case: capture, replay, trim to the minimum, then write a polite client for it.

Practice#

1. Export a HAR from a site you own or a public demo page, and list every JSON response with its size and wait time.
2. Pick one API call, replay it with the to_httpx_args helper, then remove headers one at a time until it breaks. Write down the minimal set.
3. Run your scraper and the browser through mitmweb, and use the table above to explain any difference between the two requests.