When the API Is the Front Door

Not every target needs reverse engineering — many sites hand you a documented REST API, or expose the same endpoints their web app uses with nothing but a header missing. This lesson is about consuming those APIs properly: correct parameters, proper pagination, honest rate limiting, and making the fewest requests that still get every record.

Endpoint Anatomy: Path vs Query

REST endpoints mix two kinds of parameters. Path parameters identify a resource (/users/42), query parameters shape the response (?fields=name,email&page=2). Pass path params in the URL and query params as the params dict so the client URL-encodes them for you:

import httpx

with httpx.Client() as client:
    r = client.get(
        "https://api.example.com/v2/users/{id}",
        params={"fields": "name,email", "locale": "en"},
    )
    r.raise_for_status()
    user = r.json()

Never f-string parameters straight into a URL — reserved characters like &, =, ? and # in a search term will break the request or change its meaning. params forces encoding and is the only safe habit.

Pagination: Learn the Three Dialects

JSON APIs paginate in three dialects, and you must detect each by reading the payload once. Offset/limit: ?offset=0&limit=50, advanced until you have more than total. Page number: ?page=1&per_page=50, stepped while page * per_page < total. Cursor/continuation: the response carries an opaque next token (often a full URL) and you stop when it is null — cursors are what GraphQL and most modern APIs use for exactly the reason the graphql lesson covers (stable slices under writes).

def fetch_all(client, url, per_page=50):
    out, offset = [], 0
    while True:
        r = client.get(url, params={"offset": offset, "limit": per_page})
        data = r.json()
        out += data["items"]
        if offset + per_page >= data["total"]:
            break
        offset += per_page
    return out

Guard every pagination loop with a hard cap on pages — a server-side bug that keeps returning next tokens can otherwise run forever and hammer the host.

Honest Rate Limits and the Headers That Describe Them

Good APIs tell you your budget: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and Retry-After (seconds) when you overrun. Read them and obey them, because a 429 with a Retry-After is the server naming the exact second you may resume:

r = client.get("https://api.example.com/v2/search", params={"q": "term"})
if r.status_code == 429:
    sleep(float(r.headers["Retry-After"]))
elif r.headers.get("X-RateLimit-Remaining") == "0":
    sleep(float(r.headers["X-RateLimit-Reset"]))

If the API gives you a key, authenticate with it in the requested header (Authorization, X-API-Key or a query param) and keep it out of source control — an environment variable or a .env file, never a hardcoded string.

Conditional Requests: Skip the Unchanged

APIs frequently send ETag (a content hash) or Last-Modified. Cache both, then send If-None-Match: <etag> on the next poll; a 304 Not Modified answer means "nothing changed, and you did not pay for a body":

etag = None
while True:
    headers = {"If-None-Match": etag} if etag else {}
    r = client.get(url, headers=headers)
    if r.status_code == 304:
        sleep(interval); continue
    etag = r.headers.get("ETag")
    process(r.json())
    sleep(interval)

That is the incremental-scraping lesson applied at the API layer: polling a documented endpoint with conditional headers is cheaper, more honest, and far less likely to get you throttled than re-downloading the world on a timer. Prefer a real API over any HTML page-walking any day — every scraping lesson still applies (paginate, rate-limit, validate), but the parser never touches markup.