Sitemaps and Feeds

The Structured Doors Into a Site#

Before writing a single selector, check whether the site publishes a machine-readable map of itself. Most content sites do, because search engines use these files. Two formats dominate: XML sitemaps, which list URLs with optional metadata, and RSS/Atom feeds, which list the newest content. Both turn "crawl the whole site to discover pages" into "read the index", which is faster for you and far lighter for the server. Use them first and crawl second.

Finding Them#

Check in this order:

1. Sitemap: lines in robots.txt. There can be several, and they may point to other hosts (for example a CDN). Python's standard library parses them: urllib.robotparser.RobotFileParser has a site_maps() method.
2. Conventional paths: /sitemap.xml, /sitemap_index.xml, /sitemap.xml.gz. Content management systems have their own conventions, so a 404 on one does not mean the others are missing.
3. Feed links in the HTML head: <link rel="alternate" type="application/rss+xml" href="..."> or type="application/atom+xml". Section and category pages often advertise their own narrower feeds.
4. Conventional feed paths: /feed, /rss, /atom.xml, /index.xml.

import httpx
from lxml import html as lh
from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser

client = httpx.Client(timeout=20.0, follow_redirects=True,
                      headers={"User-Agent": "example-research-bot/1.0 (contact@example.org)"})

def discover(base):
    rp = RobotFileParser()
    rp.parse(client.get(urljoin(base, "/robots.txt")).text.splitlines())
    sitemaps = rp.site_maps() or [urljoin(base, "/sitemap.xml")]

    head = lh.fromstring(client.get(base).content)
    feeds = [urljoin(base, h) for h in head.xpath(
        '//link[@rel="alternate"][contains(@type,"rss") or contains(@type,"atom")]/@href')]
    return rp, sitemaps, feeds

Keep the RobotFileParser object. A sitemap lists URLs the site wants search engines to find, but robots.txt rules still apply to you, and occasionally a sitemap contains URLs that robots.txt disallows. Check rp.can_fetch(user_agent, url) before fetching each one (see robots and ethics).

Parsing Sitemaps Correctly#

A sitemap is either a <urlset> of <url> entries or a <sitemapindex> of <sitemap> entries pointing to more sitemaps. Both use the namespace http://www.sitemaps.org/schemas/sitemap/0.9, and forgetting the namespace is the most common reason a sitemap parser returns nothing.

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemaps/products-1.xml.gz</loc>
    <lastmod>2026-09-28T04:10:00+00:00</lastmod>
  </sitemap>
</sitemapindex>

A parser that handles both types, gzip compression, and nesting:

import gzip
from lxml import etree

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
SAFE_PARSER = etree.XMLParser(resolve_entities=False, no_network=True, huge_tree=False)

def fetch_xml(url):
    body = client.get(url).content
    if body[:2] == b"\x1f\x8b":                  # gzip magic bytes
        body = gzip.decompress(body)
    return etree.fromstring(body, parser=SAFE_PARSER)

def walk_sitemap(url, seen=None, depth=0):
    seen = seen if seen is not None else set()
    if url in seen or depth > 5:
        return
    seen.add(url)
    root = fetch_xml(url)
    tag = etree.QName(root).localname
    if tag == "sitemapindex":
        for loc in root.xpath("sm:sitemap/sm:loc/text()", namespaces=NS):
            yield from walk_sitemap(loc.strip(), seen, depth + 1)
    elif tag == "urlset":
        for u in root.xpath("sm:url", namespaces=NS):
            yield {
                "loc": u.findtext("sm:loc", namespaces=NS).strip(),
                "lastmod": u.findtext("sm:lastmod", namespaces=NS),
            }

for entry in walk_sitemap("https://example.com/sitemap_index.xml"):
    print(entry)

Why the non-obvious lines are there:

• Checking gzip magic bytes instead of the file extension. Some servers send .xml.gz files with Content-Encoding: gzip, in which case httpx has already decompressed them; others send raw gzip bytes. Checking the first two bytes handles both.
• seen and depth. Misconfigured sites have indexes that reference themselves. Without the guard, the walker recurses forever.
• QName(root).localname strips the namespace, so the type check works regardless of prefix.
• The safe parser. Sitemaps are untrusted XML from the internet. Disabling entity resolution and network access protects against XML entity attacks such as "billion laughs" and external entity reads. The defusedxml package offers the same protection for the standard library's ElementTree.

The protocol limits each sitemap file to 50,000 URLs and 50 MB uncompressed, so large sites always use indexes. For very large files, etree.iterparse streams entries instead of loading the whole tree, which keeps memory flat.

Treat lastmod as a Hint#

lastmod is the site telling you when a URL last changed, and it powers incremental scraping: fetch only URLs whose lastmod is newer than your last run.

from datetime import datetime, timezone

def parse_lastmod(value):
    if not value:
        return None
    value = value.strip().replace("Z", "+00:00")
    dt = datetime.fromisoformat(value)       # handles 2026-09-28 and full timestamps
    return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)

last_run = datetime(2026, 9, 20, tzinfo=timezone.utc)
changed = [e for e in walk_sitemap(index_url)
           if (lm := parse_lastmod(e["lastmod"])) is None or lm > last_run]

Entries without lastmod are included, because unknown means "might have changed". Be sceptical of the values themselves: some sites set every lastmod to the time the sitemap was generated, which makes every URL look new on every run. If more than, say, 90 percent of entries changed since yesterday, the field is probably meaningless for that site. Fall back to comparing content hashes, or to conditional requests with ETag and If-Modified-Since. The <changefreq> and <priority> fields are optional hints that search engines themselves largely ignore; do not schedule around them.

Sitemap extensions add useful data for free. Image, video, and news sitemaps use additional namespaces (for example <image:loc> or <news:publication_date>), giving you media URLs and publication dates without fetching the page.

Feeds: The Change Stream#

RSS and Atom exist so readers can follow a site without re-scanning it. The feedparser library handles both formats, broken XML, and the many date formats in the wild:

import feedparser

resp = client.get("https://example.com/feed.xml")
feed = feedparser.parse(resp.content)

if feed.bozo:
    print("feed is malformed:", feed.bozo_exception)   # often still usable
for entry in feed.entries:
    print(entry.get("id") or entry.link, entry.title, entry.get("published", ""))

Notes from practice:

• Fetch with your own client, then parse the bytes. feedparser.parse(url) can fetch by itself, but using your HTTP client keeps timeouts, headers, proxies, and rate limits consistent with the rest of the scraper.
• bozo is set for feeds with any XML error. feedparser still returns what it could parse, so log it rather than discarding the feed.
• Deduplicate on the entry id (Atom) or guid (RSS) where present, not on the title, which gets edited after publication. feedparser exposes both as entry.id.
• Poll with conditional requests. Send If-None-Match with the feed's last ETag. A 304 means nothing is new and costs the server almost nothing, which makes frequent polling acceptable.

A feed only shows the latest N items. If you poll less often than the site publishes N items, you will miss entries. Measure the publishing rate from the timestamps in two consecutive fetches and set the polling interval well below the time it takes to fill the feed.

Combining Sources#

A robust design uses each source for what it does best: the feed for fast discovery of new items, the sitemap for periodic completeness checks and change detection, and a link crawl only for sections neither covers. Reconcile the three: URLs that appear in your crawl but not in the sitemap are worth logging, as they often reveal sections the site forgot to include. A site without a sitemap or feed is not a red flag by itself, but it does mean discovery falls back to the crawl frontier, which costs both you and the server more.

Practice#

For a site you are allowed to scrape, list every sitemap reachable from its robots.txt, count URLs per child sitemap, and compute what percentage have a lastmod within the last 24 hours. Then fetch its main feed twice, an hour apart, with a conditional request the second time, and record whether you got a 304.