Scraping GraphQL APIs

One Endpoint, Every Query#

GraphQL is a query language for APIs rather than a new transport. Instead of many REST resources, an app exposes one endpoint (usually /graphql) that accepts a POST whose JSON body describes exactly the data wanted. The server resolves that query against its schema and returns a JSON document with the same shape. For a scraper this is good news: the responses are structured, contain only the fields you asked for, and every screen of the app is served through the same door.

The skills are finding the endpoint, learning the schema, writing queries that match what the app already sends, and paginating with cursors, all while staying within the load the server expects.

Anatomy of a Request#

Every GraphQL call over HTTP carries up to three fields:

{
  "operationName": "SearchProducts",
  "query": "query SearchProducts($term: String!, $first: Int, $after: String) { search(term: $term, first: $first, after: $after) { pageInfo { hasNextPage endCursor } edges { node { id name price { amount currency } } } } }",
  "variables": {"term": "keyboard", "first": 24, "after": null}
}

• query is the operation text. Values are passed as $variables, never pasted into the string.
• variables is a separate JSON object. Changing a search term or a page cursor means changing variables only.
• operationName selects one operation when the document contains several, and servers often use it for logging and rate limits.

The response always has the same envelope: a data key with results and, when something went wrong, an errors list.

Finding the Endpoint#

Open DevTools, go to the Network tab, filter by Fetch/XHR, and use the app normally. Look for POST requests to a path containing graphql, or any request whose payload has query and variables keys. Some apps put the operation name in the URL (/graphql?op=SearchProducts), which makes them easy to spot.

Right-click the request and choose Copy as cURL. Note three things: the full body, any non-standard headers (for example an app version header, an API key, or a CSRF token), and the operation names used on each screen. The captured request is your first working query; start by replaying it exactly before changing anything.

import httpx

client = httpx.Client(
    base_url="https://shop.example.com",
    headers={"Content-Type": "application/json", "Accept": "application/json"},
    timeout=20.0,
)

def gql(query, variables=None, operation=None):
    resp = client.post("/graphql", json={
        "query": query, "variables": variables or {}, "operationName": operation,
    })
    resp.raise_for_status()
    body = resp.json()
    if body.get("errors"):
        raise RuntimeError(body["errors"])
    return body["data"]

The errors check is essential. GraphQL servers typically return HTTP 200 even when a query fails, and they can return partial data together with errors when one field fails while others succeed. A scraper that only checks the status code will silently store nulls.

Learn the Schema With Introspection#

GraphQL can describe itself. If introspection is enabled, you can ask for every type and field:

schema = gql('''
  query {
    __schema {
      queryType { name }
      types { name kind }
    }
  }
''')
print([t["name"] for t in schema["__schema"]["types"]
       if not t["name"].startswith("__")][:20])

Then drill into a type you care about:

product = gql('''
  query($name: String!) {
    __type(name: $name) {
      fields {
        name
        args { name }
        type { name kind ofType { name kind } }
      }
    }
  }
''', {"name": "Product"})
for f in product["__type"]["fields"]:
    t = f["type"]
    print(f["name"], t["name"] or f'{t["kind"]} of {t["ofType"]["name"]}')

The kind/ofType nesting is how GraphQL expresses wrappers: a field of type [Review!]! appears as NON_NULL of LIST of NON_NULL of Review. The snippet shows one level; walk ofType recursively to unwrap fully. Tools such as GraphiQL or GraphQL Voyager can render the full introspection result as browsable documentation.

Many production servers disable introspection. Then the captured queries are your schema: every field the app requests is known to exist and known to be allowed. Build your queries by trimming and combining the app's own operations rather than guessing field names. Guessing is noisy and shows up clearly in the server's error logs.

Persisted Queries#

Some clients do not send the query text at all. With persisted queries, the request carries a hash and the server looks up the query it registered at build time:

{
  "operationName": "SearchProducts",
  "variables": {"term": "keyboard", "first": 24},
  "extensions": {"persistedQuery": {"version": 1, "sha256Hash": "3f2a...c9"}}
}

There are two flavours. With automatic persisted queries (the Apollo convention), the hash is the SHA-256 of the query text; if the server does not recognise it, it replies with a PersistedQueryNotFound error and the client resends with the full query. With a strict allowlist, the server only accepts hashes registered in advance and rejects arbitrary query text entirely.

For scraping, the rule is simple: replay the app's own extensions object with your own variables. You can change the search term and the cursor, but not the selected fields, because the hash pins the query. Hashes change when the app is redeployed, so treat a sudden PersistedQueryNotFound as a signal to re-capture from DevTools. Some persisted-query setups also accept GET requests with variables and extensions as URL-encoded JSON parameters; copy whatever method the app uses.

Cursor Pagination#

Collections in GraphQL usually follow the connection pattern: you ask for first: N items after a cursor, and each page returns pageInfo { hasNextPage endCursor }.

import time

SEARCH = '''
query SearchProducts($term: String!, $first: Int!, $after: String) {
  search(term: $term, first: $first, after: $after) {
    pageInfo { hasNextPage endCursor }
    edges { node { id name price { amount currency } } }
  }
}'''

def all_products(term, page_size=50, max_pages=200):
    after, seen = None, set()
    for _ in range(max_pages):
        data = gql(SEARCH, {"term": term, "first": page_size, "after": after},
                   "SearchProducts")
        conn = data["search"]
        for edge in conn["edges"]:
            node = edge["node"]
            if node["id"] not in seen:
                seen.add(node["id"])
                yield node
        if not conn["pageInfo"]["hasNextPage"]:
            return
        if conn["pageInfo"]["endCursor"] == after:
            raise RuntimeError("cursor did not advance")
        after = conn["pageInfo"]["endCursor"]
        time.sleep(1.0)

Details that save you from real bugs:

• Cursors are opaque. Many are base64-encoded offsets or ids, and decoding one is a useful clue, but always pass back exactly the string you received. Constructing your own cursors breaks when the encoding changes.
• Guard against loops. A cursor that does not advance or a hasNextPage that never becomes false would otherwise run forever. The max_pages cap and the "did not advance" check stop that.
• Respect the maximum page size. Servers often cap first (commonly around 50 or 100) and return an error above it. Use the value the app uses, or slightly larger if the server accepts it.
• Dedupe on id. Items inserted while you paginate can shift pages and produce duplicates.

Aliases, Batching, and Query Cost#

GraphQL lets one request do a lot. Aliases fetch the same field several times with different arguments:

query {
  kb: search(term: "keyboard", first: 5) { edges { node { id name } } }
  ms: search(term: "mouse", first: 5) { edges { node { id name } } }
}

Some servers also accept a JSON array of operations in one POST (query batching). These features are exactly what servers defend against, because one request can cost as much as hundreds. Most production GraphQL APIs enforce limits on query depth, complexity or cost, and rate limits that count cost rather than requests. Some report the cost in the response extensions; if yours does, log it and pace requests against it.

For scraping, keep one operation per request, request only the fields you need, and avoid deeply nested selections (for example products, then reviews for each product, then the author of each review) unless the app itself makes that call. A large nested query that takes the server seconds to resolve is far more disruptive than several small ones spaced out, and it is also far more likely to be rejected. The rate limiting and politeness rules apply per operation cost, not just per HTTP request.

Debugging#

When a replayed query fails, diff your request against the captured one header by header and field by field; the difference is almost always a header. For endpoints that behave like conventional REST instead, see REST API scraping, and for the general method of discovering them, XHR and API reverse engineering.