Respecting the Target Server#

Every page you fetch is served by infrastructure someone else pays for. A careless scraper can degrade a small site for its real users, run up its hosting bill, or trigger an incident response at 3 AM. The practices in this lesson are the difference between a crawler that operators tolerate for years and one that gets blocked, reported, or sued. They also happen to produce better data, because a polite crawl is a crawl that keeps working.

What robots.txt Is, and What It Is Not#

robots.txt lives at the root of a host (https://example.com/robots.txt) and tells automated clients which paths the operator does not want crawled. The Robots Exclusion Protocol was used informally for decades and was standardized as RFC 9309 in 2022.

It is a published request, not access control. Nothing stops a client from ignoring it, which is exactly why ignoring it carries weight: it shows you knew what the operator wanted and did it anyway. Courts, hosting providers, and anti-bot vendors all treat robots.txt compliance as evidence of good or bad faith. It applies per host, so shop.example.com and www.example.com each have their own file.

Reading the File Correctly#

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/about
Disallow: /*.pdf$
Crawl-delay: 5

User-agent: MyDataBot
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

The rules that trip people up:

  • Groups, not a single list. Each User-agent line starts a group. A crawler uses the group whose name matches its product token (case-insensitive) and only falls back to * when nothing matches. In the file above, MyDataBot is allowed into /cart/ because its own group replaces the * group entirely. Name your bot, then read its group first.
  • Path prefixes. Disallow: /search blocks /search, /search?q=x, and /searchable. Add a trailing slash when you mean a directory.
  • Wildcards. * matches any sequence of characters and $ anchors the end, so /*.pdf$ blocks every URL ending in .pdf.
  • Longest match wins. When an Allow and a Disallow both match, the more specific (longer) rule applies, and a tie goes to Allow. That is why /search/about stays crawlable.
  • Empty values. Disallow: with nothing after it allows everything; Disallow: / blocks everything.
  • Crawl-delay is not in the standard. Many crawlers honor it and some major search engines ignore it. Treat it as a floor for your own delay, never as permission to go faster than your own limits.
  • Sitemap lines are independent of groups and point at URL lists that are often a better crawl source than link-following; see sitemaps and feeds.

Missing, Broken, or Blocked robots.txt#

What you do when the file cannot be fetched matters as much as parsing it. RFC 9309 says a 4xx response means there are no restrictions, while a 5xx or network failure means the crawler should assume everything is disallowed until it can read the file. A server error is not a green light. Cache the parsed file (the standard suggests not relying on a cached copy for more than 24 hours) instead of refetching it before every request.

Python's Parser Has Gaps#

urllib.robotparser ships with Python, but it predates the standard. It applies rules in file order (first match wins) instead of longest match, and it does not understand * or $ inside paths. Run both parsers on the * group above:

import urllib.robotparser
from protego import Protego

rules = '''User-agent: *
Disallow: /shop/
Allow: /shop/public/
Disallow: /*.pdf$
'''

rp = urllib.robotparser.RobotFileParser()
rp.parse(rules.splitlines())
pg = Protego.parse(rules)

for url in ["https://e.com/shop/public/a", "https://e.com/shop/x", "https://e.com/a.pdf"]:
    print(f"{url:32} stdlib={rp.can_fetch('MyDataBot', url)!s:5} protego={pg.can_fetch(url, 'MyDataBot')}")

Output:

https://e.com/shop/public/a      stdlib=False protego=True
https://e.com/shop/x             stdlib=False protego=False
https://e.com/a.pdf              stdlib=True  protego=False

The standard library blocks a page the site explicitly allows and allows PDFs the site explicitly blocks. Protego, the parser Scrapy uses, implements wildcards and longest-match. Note its argument order: can_fetch(url, user_agent), the reverse of the stdlib.

RobotFileParser.read() also fetches the file with Python's default User-Agent, which some servers reject, and it treats 401 and 403 as "disallow everything". Fetching the file yourself and passing the text to a parser gives you control over headers, timeouts, and caching:

import httpx
from protego import Protego
from functools import lru_cache
from urllib.parse import urlsplit

UA = "MyDataBot/1.0 (+mailto:data-team@mycompany.example)"
client = httpx.Client(headers={"User-Agent": UA}, timeout=10.0, follow_redirects=True)

@lru_cache(maxsize=1024)
def robots_for(origin: str):
    try:
        resp = client.get(origin + "/robots.txt")
    except httpx.HTTPError:
        return None                      # unreachable: treat as disallow-all
    if resp.status_code >= 500:
        return None                      # server error: disallow-all
    if resp.status_code >= 400:
        return Protego.parse("")         # no file: no restrictions
    return Protego.parse(resp.text)

def allowed(url: str) -> bool:
    parts = urlsplit(url)
    rules = robots_for(f"{parts.scheme}://{parts.netloc}")
    return rules is not None and rules.can_fetch(url, "MyDataBot")

In a long-running crawler, replace lru_cache with a cache that expires entries after a day, so rule changes are picked up.

Crawling for AI Training Data#

Many sites now publish separate groups for AI-related crawlers, using product tokens such as GPTBot, CCBot, or Google-Extended, often with stricter rules than for search engines. If you are collecting data to train or fine-tune models, the operator's intent is clear from those groups even though your bot has a different name: honor the most restrictive group that describes your purpose. In the EU, machine-readable opt-outs like these also have legal significance for text and data mining, which the legal considerations lesson covers.

Core Ethical Guidelines#

  1. Identify yourself. Use a descriptive User-Agent with a contact address, such as MyDataBot/1.0 (+mailto:data-team@mycompany.example). Operators who can reach you send an email; operators who cannot, block your subnet. Where a site only serves browsers and you choose a browser-like identity, keep the other practices here even more strictly.
  2. Throttle. One to two requests per second per host, one connection at a time, is a sensible default for sites you have not measured. Back off immediately on 429 and 503. The rate limiting and politeness lesson implements this.
  3. Cache during development. You will run your parser hundreds of times while building it. Fetch each page once, save it, and iterate on the local copy.
  4. Take only what you need. Skip images, video, and large files when you only need metadata. Use conditional requests and sitemaps' lastmod to avoid refetching unchanged pages.
  5. Schedule heavy jobs off-peak for the site's main audience, and spread large backfills over days instead of one burst.
  6. Minimize personal data. If you do not need names, emails, or profile details, do not collect them. If you do, you take on legal duties for them.
  7. Do not defeat access controls. Logins you are not authorized to use, paywalls, and technical blocks put in place specifically against you are the operator saying no.
  8. Answer complaints and stop when asked. Keep a per-host kill switch in your config so you can comply within minutes.

Meta Robots and X-Robots-Tag#

Pages can carry <meta name="robots" content="noindex, nofollow"> or an X-Robots-Tag header. These govern search engine indexing and link-following, not fetching, so they are not a crawl ban. They do tell you the operator does not want the content surfaced in search, which matters if you plan to republish it.

Practice#

Write a can_crawl(url) function that fetches robots.txt with your own User-Agent, applies the RFC's status-code rules, caches for 24 hours, and returns both the decision and the Crawl-delay for your bot. Test it against a local server that serves a file with a named group, a wildcard rule, and an Allow inside a disallowed directory, then against 404 and 503 responses.