The Scrapy Framework

Trade the Glue for a Framework#

By this point in the course you have hand-built every piece of a crawler: sessions, retries, deduplication, throttling, caching, storage. That is the right way to learn and a tiring way to operate. Scrapy is the long-established Python framework that packages those pieces into one engine. It schedules requests asynchronously, filters duplicate URLs, obeys robots.txt when configured to, retries failures, adapts its speed to the server, runs items through a pipeline, collects statistics, and can pause and resume a crawl. Your job shrinks to the site-specific parts: which links to follow and what to extract.

When Scrapy Is the Right Tool#

The trade-off is learning the framework's structure. Once you do, a new site becomes one spider file rather than a new copy of all your plumbing.

The Shape of a Project#

pip install scrapy
scrapy startproject shop
cd shop
scrapy genspider products example.com

shop/
    scrapy.cfg
    shop/
        items.py
        middlewares.py
        pipelines.py
        settings.py
        spiders/
            products.py

The data flow is always the same. A spider yields either Request objects (more work) or items (results). Requests go to the scheduler, which is the frontier from crawl strategy and the frontier, then through downloader middlewares (headers, proxies, retries) to the network. Responses come back to the spider's callback. Items go through item pipelines in order (validate, clean, deduplicate, store).

A Spider That Follows Listings to Detail Pages#

import scrapy

class ProductsSpider(scrapy.Spider):
    name = "products"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/products"]

    def parse(self, response):
        for card in response.css(".product"):
            yield response.follow(
                card.css("a::attr(href)").get(),
                callback=self.parse_product,
                cb_kwargs={"listing_price": card.css(".price::text").get()},
            )
        yield from response.follow_all(css=".pagination a.next", callback=self.parse)

    def parse_product(self, response, listing_price):
        yield {
            "url": response.url,
            "sku": response.css("[data-sku]::attr(data-sku)").get(),
            "title": response.css("h1::text").get(default="").strip(),
            "price": listing_price,
            "description": " ".join(response.css(".description ::text").getall()).strip(),
        }

Run it and write results as JSON Lines:

scrapy crawl products -O products.jsonl

What the framework is doing for you:

• response.follow resolves relative URLs against the page, sets the Referer, and hands the request to the scheduler. You never call the network yourself.
• cb_kwargs passes data from the listing page to the detail callback. Use it instead of global state, because requests complete in any order.
• follow_all(css=...) extracts the href of every matched link. The duplicate filter drops URLs already seen in this crawl, so paging links that appear on every page do not cause loops.
• allowed_domains makes the offsite filter drop links to other hosts, which keeps a crawl from wandering onto the whole web.
• -O overwrites the output file; -o appends. JSON Lines is the safer format for large crawls because each line is complete even if the run is interrupted.

Develop Selectors in the Shell#

scrapy shell URL fetches a page with your project's settings and opens a Python prompt with response loaded. Try selectors there before putting them in a spider:

$ scrapy shell "https://example.com/products"
>>> response.status
200
>>> response.css(".product h3::text").getall()[:3]
['Desk Lamp', 'Floor Lamp', 'Reading Light']
>>> response.css(".price::text").get()
'$24.99'
>>> view(response)

view(response) opens the downloaded HTML in your browser. It shows what Scrapy received, not what a browser renders after JavaScript runs. If the data is missing there but present in the live page, the content is loaded by JavaScript, and you should look for the API behind it or use a rendering plugin. The CSS and XPath syntax is the same as in the CSS selectors and XPath lessons.

Pipelines: Validate, Clean, Deduplicate, Store#

Pipelines are classes with a process_item method. Each returns the item (possibly modified) or raises DropItem. They run in ascending order of the number you register them with.

from decimal import Decimal, InvalidOperation
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem
import sqlite3

class ValidateAndClean:
    def process_item(self, item, spider):
        a = ItemAdapter(item)
        if not a.get("sku"):
            raise DropItem(f"missing sku: {a.get('url')}")
        raw = (a.get("price") or "").replace("$", "").replace(",", "").strip()
        try:
            a["price"] = Decimal(raw)
        except InvalidOperation:
            raise DropItem(f"bad price {raw!r}: {a.get('url')}")
        return item

class Deduplicate:
    def open_spider(self, spider):
        self.seen = set()

    def process_item(self, item, spider):
        sku = ItemAdapter(item)["sku"]
        if sku in self.seen:
            raise DropItem(f"duplicate sku {sku}")
        self.seen.add(sku)
        return item

class SQLiteStore:
    def open_spider(self, spider):
        self.db = sqlite3.connect("products.db")
        self.db.execute("CREATE TABLE IF NOT EXISTS products "
                        "(sku TEXT PRIMARY KEY, title TEXT, price TEXT, url TEXT)")

    def process_item(self, item, spider):
        a = ItemAdapter(item)
        self.db.execute("INSERT OR REPLACE INTO products VALUES (?, ?, ?, ?)",
                        (a["sku"], a["title"], str(a["price"]), a["url"]))
        return item

    def close_spider(self, spider):
        self.db.commit()
        self.db.close()

ItemAdapter lets the same pipeline handle plain dicts, scrapy.Item classes and dataclasses. open_spider and close_spider are the right places for connections and file handles. Dropped items appear in the final stats with the reason in the log, which makes data quality problems visible instead of silent. The cleaning rules themselves are the ones from data cleaning and validation.

Settings That Encode Politeness#

# settings.py
ROBOTSTXT_OBEY = True
USER_AGENT = "shop-research-bot/1.0 (+contact@example.org)"

CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 1.0
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 30.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 500, 502, 503, 504, 522, 524, 408]

HTTPCACHE_ENABLED = True           # development only: replay responses from disk
ITEM_PIPELINES = {
    "shop.pipelines.ValidateAndClean": 100,
    "shop.pipelines.Deduplicate": 200,
    "shop.pipelines.SQLiteStore": 300,
}

The project template generated by startproject sets ROBOTSTXT_OBEY = True; keep it. AutoThrottle measures response latency and adjusts the delay so that roughly AUTOTHROTTLE_TARGET_CONCURRENCY requests are in flight per server, backing off when the server slows down. That is the adaptive behaviour from rate limiting and politeness without writing the controller. HTTPCACHE_ENABLED stores responses on disk so you can re-run parsing code dozens of times without re-requesting the site; turn it off for production runs, or you will scrape stale copies.

Per-spider overrides go in a custom_settings class attribute, which is useful when one site needs a slower pace than the rest.

Middlewares for the Sharp Edges#

Downloader middlewares see every request before it is sent and every response before the spider. They are the place for cross-cutting behaviour: adding headers, choosing a proxy, or treating a "blocked" page with status 200 as a retryable failure.

class ProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies
        self.i = 0

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))

    def process_request(self, request, spider):
        if self.proxies:
            request.meta["proxy"] = self.proxies[self.i % len(self.proxies)]
            self.i += 1
        return None          # continue normal processing

Enable it with DOWNLOADER_MIDDLEWARES = {"shop.middlewares.ProxyMiddleware": 350}. The number places it relative to built-in middlewares, which also have numbers; lower runs earlier for requests. from_crawler is how a component reads settings and connects to signals. For choosing proxies well, see proxies and rotation.

Reading the Stats#

At the end of every run Scrapy logs a stats dictionary. Read it every time; it is the fastest health check you have.

{'downloader/request_count': 1843,
 'downloader/response_status_count/200': 1790,
 'downloader/response_status_count/429': 41,
 'retry/count': 41,
 'item_scraped_count': 1712,
 'item_dropped_count': 36,
 'finish_reason': 'finished',
 'log_count/ERROR': 3}

Here, 41 responses were rate limited and all retried, 36 items failed validation (check the log for reasons), and three callbacks raised exceptions. A finish_reason other than finished (for example shutdown or closespider_errorcount) means the run did not complete. Settings like CLOSESPIDER_ERRORCOUNT = 50 stop a crawl that is clearly broken instead of letting it burn through the site producing nothing. Feed these numbers into the alerts from scheduling and monitoring.

Jobs That Pause and Resume#

Run with a job directory and Scrapy persists the scheduler queue and the duplicate filter to disk:

scrapy crawl products -s JOBDIR=crawls/products-001

Stop it once with Ctrl-C (a single press lets it shut down cleanly), run the same command again, and it continues where it stopped. Requests must be serialisable for this to work, so callbacks must be spider methods, not lambdas. Use a new JOBDIR for each fresh crawl; reusing an old one means the duplicate filter skips every URL it has already seen.

Practice#

Build the products spider against a practice site such as a demo bookstore built for scraping exercises. Add the three pipelines, run with HTTPCACHE_ENABLED = True, then deliberately break the price selector and confirm that the dropped-item count in the stats tells you something went wrong.