When the Frontier Is a Query

Sometimes the data you want is not on one known site — it is the list a search engine returns for a query, or the products inside an open directory. This lesson covers the honest ways to harvest search results: provider APIs, HTML-light endpoints, and web directories, with the ethical fence drawn firmly.

Why You Rarely Scrape Google the Hard Way

Search engine result pages (SERPs) are the most heavily anti-bot-protected pages on the internet, and scraping them head-on violates the provider's Terms of Service in almost every jurisdiction that treats ToS as binding (the legal lesson). The standard professional route is the advertised API or a licensing agreement, which costs money but keeps you legal, stable, and out of a spam folder. Treat walking a raw SERP as the non-default fallback, gated behind the same ethics floor as everything else in this course: check robots.txt, check ToS, rate-limit like a human on a coffee break.

Lightweight HTML Endpoints That Stay Polite

Some engines expose a deliberately minimal HTML surface (text links, no JS): DuckDuckGo's HTML lite endpoint answers plain queries with a simple list of result links — perfect for a low-volume disambiguation crawl and far more crawl-friendly than the rich equivalent. Parse with the now-familiar pipeline and keep it slow:

import requests
from bs4 import BeautifulSoup

r = requests.get(
    "https://html.duckduckgo.com/html/",
    params={"q": "monica bertagnolli vinegar"},
    headers={"User-Agent": "Mozilla/5.0 ..."},
)
results = [
    {"title": a.get("title"), "url": a.get("href")}
    for a in BeautifulSoup(r.text, "lxml").select("a.result__a")
]

These surfaces still rate-limit and can 429; honor Retry-After, keep per-host delays honest, and cache results aggressively so you never re-fetch a query you already answered. The lesson's shape matters more than the engine: a real API call gives you clean JSON and documented limits, which is strictly better than re-parsing HTML — so prefer the API tier first, and treat the HTML-lite endpoint as the polite second option.

Directories and Open Indexes

Web directories (dmoz-style, niche listings, curated link hubs) and "open indexes" of publicly linked pages are underused as crawl frontiers: they hand you curated starting URLs with titles and descriptions already attached, which is a sitemap-and-feeds quality entry point with zero JavaScript. Scrape the directory pages for the href + title pairs, feed them to the URL frontier, and let the crawl-strategy lesson's dedupe and depth rules take over:

links = [
    (a.get_text(strip=True), urljoin(page_url, a.get("href")))
    for a in soup.select("li a")
]
for title, href in links:
    frontier.enqueue(normalize(href))

Build Your Own Index to Stay Independent

The durable win is owning the frontier you control: a private sitemap-derived index plus directory lists plus your known URL seeds, stored and ranked the way your pipeline wants. Search-scraping done right is mostly search infrastructure — canonicalize URLs, dedupe, track domain credit limits (the rate-limiting lesson) and record sources. Then "what results match this query in my dataset" is a query over your repository, not a fragile dance with an engine that would rather you not be there at all.