Search Engines, SERPs & Directories

When the Frontier Is a Query#

Sometimes the data you need is not on one known site. It is the list of pages a search engine returns for a query, the entries in a directory, or every page in a public index that mentions a domain. This lesson covers the legitimate ways to harvest search results and open indexes: official search APIs first, lightweight HTML endpoints at low volume, directories and crawl archives as seed sources, and finally building your own index so you depend on none of them.

Why You Rarely Scrape a Major SERP Directly#

Search engine result pages (SERPs) are among the most heavily defended pages on the web, and the major engines' terms of service prohibit automated querying outside their APIs. Scraping them head-on means fighting constant blocks for data you have no licence to use (see legal considerations and robots and ethics). The professional default is an official or commercial search API: it costs money, but it returns clean JSON, documented quotas and a legal basis. Search APIs are also discontinued, repriced and restricted more often than most APIs, so check the provider's current terms and availability before building on one, and keep your code behind a small interface so you can switch providers.

Tier 1: A Search API Behind Your Own Interface#

Wrap whichever provider you use in a function that returns a uniform result shape. Here is the pattern with Google's Programmable Search JSON API as the example provider (it needs an API key and a search engine ID, and returns at most 10 results per request):

import time
import httpx

def search_api(query, *, key, cx, pages=2, client=None):
    client = client or httpx.Client(timeout=20)
    results = []
    for page in range(pages):
        r = client.get(
            "https://www.googleapis.com/customsearch/v1",
            params={"key": key, "cx": cx, "q": query, "start": 1 + page * 10},
        )
        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", "30")))
            continue
        r.raise_for_status()
        items = r.json().get("items", [])
        for rank, item in enumerate(items, start=1 + page * 10):
            results.append({"rank": rank, "title": item.get("title"),
                            "url": item.get("link"),
                            "snippet": item.get("snippet")})
        if len(items) < 10:
            break  # no further pages
    return results

The uniform shape (rank, title, url, snippet) is what the rest of your pipeline sees. Swapping in a different provider then means rewriting one function, not the crawler.

Tier 2: Lightweight HTML Endpoints, Slowly#

Some engines expose a deliberately minimal HTML interface with plain links and no JavaScript. DuckDuckGo's HTML endpoint is the well-known example. It is suitable for occasional, low-volume lookups such as disambiguating a company name, not for bulk harvesting. Read its current robots.txt and terms first, identify yourself honestly, and keep a long delay between queries:

from urllib.parse import urlparse, parse_qs
import requests
from bs4 import BeautifulSoup

def unwrap(href):
    '''Many engines wrap result links in a redirect URL; recover the target.'''
    if href.startswith("//"):
        href = "https:" + href
    qs = parse_qs(urlparse(href).query)
    for key in ("uddg", "url", "q", "u"):
        if key in qs and qs[key][0].startswith("http"):
            return qs[key][0]
    return href

r = requests.get("https://html.duckduckgo.com/html/",
                 params={"q": "acme widgets annual report"},
                 headers={"User-Agent": "research-bot/1.0 (contact@example.org)"},
                 timeout=20)
soup = BeautifulSoup(r.text, "lxml")
results = [{"title": a.get_text(strip=True), "url": unwrap(a["href"])}
           for a in soup.select("a.result__a")]

The unwrap step matters more than it looks. Result links are often redirect or click-tracking URLs; storing them unmodified gives you thousands of "different" URLs on the search engine's domain instead of the real targets. After unwrapping, run the URL through the normalisation rules from URL handling and normalization so the same page found by two queries deduplicates.

Minimal endpoints still rate-limit. Honour Retry-After on a 429, cache every query result so you never ask the same question twice, and stop entirely if you start receiving challenge pages; that is the operator telling you the volume is unwelcome, not a problem to engineer around. Their markup also changes without notice, so treat the selector as fragile and assert that results were found.

What Makes SERP Data Tricky#

Even through an API, search results are not a stable fact about the web:

• They are localised and sometimes personalised. Record the query, language, country parameters and timestamp with every result set; without them the ranks are uninterpretable.
• Ranks shift day to day. If you track rankings, store snapshots (query, date, rank, url) and compare over time rather than overwriting.
• Result counts are estimates. "About 1,200,000 results" is not a count you can page through; engines typically serve only a limited number of pages per query.
• Deep coverage needs query variation, not deeper paging. To enumerate many pages on a topic, split the query (by site, year, subtopic or file type) rather than requesting page 50.

Tier 3: Open Indexes and Directories as Seeds#

Often you do not need a search engine at all; you need a list of URLs to start crawling. Two sources are underused.

Crawl archives. Common Crawl publishes regular web crawls with a URL index you can query per domain. The list of crawls is at https://index.commoncrawl.org/collinfo.json, and each crawl has a CDX API that returns one JSON line per captured URL:

import json
import requests

crawls = requests.get("https://index.commoncrawl.org/collinfo.json", timeout=30).json()
api = crawls[0]["cdx-api"]  # most recent crawl first
r = requests.get(api, params={"url": "example.com/*", "output": "json"}, timeout=60)
urls = {json.loads(line)["url"] for line in r.text.splitlines() if line.strip()}
print(len(urls), sorted(urls)[:5])

This enumerates pages the crawl found on a domain, including pages missing from its sitemap, without sending a single request to the site itself. The index is large and shared, so query politely and cache the answers.

Directories and curated lists. Niche directories, association member lists and curated link hubs hand you URLs with titles and categories already attached. Scrape the listing pages, then feed the links into your frontier, where the dedupe and depth rules from crawl strategy and frontier take over:

from urllib.parse import urljoin

for a in soup.select("ul.listing li a[href]"):
    frontier.enqueue(normalize(urljoin(page_url, a["href"])),
                     meta={"title": a.get_text(strip=True), "source": page_url})

Combine these seeds with sitemaps (see sitemaps and feeds) and you often cover a domain more completely than any search engine query would.

Tier 4: Build Your Own Index#

The durable win is owning the index. Once you have crawled the pages, "which of my documents match this query?" is a local query, with no quota and no terms to break. SQLite's FTS5 extension, included in most Python builds, gives you ranked full-text search in a few lines:

import sqlite3

con = sqlite3.connect("index.db")
con.execute("CREATE VIRTUAL TABLE IF NOT EXISTS docs USING fts5(url UNINDEXED, title, body)")
con.executemany("INSERT INTO docs (url, title, body) VALUES (?, ?, ?)", [
    ("https://a.example/widgets", "Acme widgets", "Industrial widgets and gears"),
    ("https://b.example/gears", "Gear guide", "Choosing gears for widgets"),
    ("https://c.example/about", "About us", "A bakery in Leeds"),
    ("https://d.example/news", "Company news", "Quarterly results and hiring"),
    ("https://e.example/faq", "FAQ", "Shipping, returns and widgets warranty"),
])
rows = con.execute(
    "SELECT url, title, round(bm25(docs), 3) FROM docs "
    "WHERE docs MATCH ? ORDER BY bm25(docs) LIMIT 10", ("widgets gears",)).fetchall()
for row in rows:
    print(row)

The output is two rows, the Acme and gear-guide pages, each with a score of about -0.336. Space-separated terms are combined with AND, so the FAQ page that mentions only "widgets" is left out, as are the pages matching neither term. bm25() scores relevance where lower is better (the values are negative), which is why the sort is ascending; on a real corpus of thousands of pages the scores spread out and the order becomes meaningful. FTS5 query syntax also supports phrases ("industrial widgets"), prefixes (widg*) and OR. For larger corpora the same idea scales to a dedicated search engine such as OpenSearch or Elasticsearch, but the principle holds: canonicalise URLs, deduplicate, record where each document came from and when, and search your own repository.

Choosing a Route#

The official APIs and alternative data lesson in the anti-bot course expands on why paying for or using an official channel usually beats an arms race.