The Document Object Model (DOM)#

An HTTP response gives you one long string of HTML. To pull values out of it reliably, you need to see that string the way a parser does: as a tree of nested elements, the Document Object Model (DOM). Every element has a tag name, zero or more attributes, and children that are either other elements or text nodes. Extraction is the act of describing a path through that tree.

<div class="product-card" data-sku="98765">
    <h2 class="product-title">Mechanical Keyboard</h2>
    <span class="product-price">$99.99</span>
    <a href="/products/mech-keyboard" class="btn-buy">Buy Now</a>
</div>

The mental model for the price is: find the div with class product-card, then its child span with class product-price, then take its text. The SKU is not text at all; it is an attribute on the container. Getting used to asking "is my value in text, in an attribute, or in a script?" saves most beginner confusion.

View Source Is Not the Elements Panel#

Browser DevTools shows you two different documents, and scrapers get burned by mixing them up:

  • View Source (Ctrl+U) shows the raw bytes the server sent. This is what requests and httpx receive.
  • The Elements panel shows the live DOM after the browser parsed the HTML, repaired it, and ran JavaScript that may have added, removed, or rewritten elements.

If a value appears in Elements but not in View Source, it was rendered by JavaScript, and the JavaScript-rendered pages lesson applies. A classic smaller trap: browsers insert a <tbody> into every table even when the server did not send one. An XPath or selector copied from DevTools such as table > tbody > tr then matches nothing in the raw HTML. When a copied selector fails, search View Source for the exact text you want and build the path from there.

Identifying Data Targets#

Before writing parsing code, inspect the page:

  1. Open DevTools (F12, or right-click and Inspect) and hover over the value you want.
  2. Walk up the tree to the smallest element that repeats once per record (a card, a row, a list item). That is your container.
  3. Note what makes the container unique: a class, an id, a data-* attribute, or its position under a named parent.
  4. Confirm the value is present in View Source, not just in Elements.

The Hooks: IDs, Classes, and Data Attributes#

  • IDs (id="product-price") should be unique per page, which makes them the most precise hook. In practice some sites reuse IDs, so verify with a count.
  • Classes (class="price-tag text-bold") are space-separated lists, often shared by many elements. You will iterate over elements sharing a class. Be wary of machine-generated class names like css-1q2w3e or ProductCard_price__x7Yz; they come from CSS-in-JS and CSS modules and change on every deploy.
  • Data attributes (data-sku="12345", data-price="99.99") exist so the site's own JavaScript can read clean values. They are usually more stable than visible text and already normalized.
  • Semantic attributes such as itemprop="price" (microdata), aria-label, datetime on <time>, and content on <meta> also carry machine-readable values.

The Containers Data Actually Lives In#

Most of the web's data sits in a handful of structural patterns:

  • Tables (table, tr, td, th): specifications, price grids, financial data. See HTML tables.
  • Definition lists (dl, dt, dd): the favorite of product pages, where dt is a label and dd its value.
  • Lists (ul, ol, li): navigation, tag clouds, search results.
  • Repeating wrapper divs (div.card, article.result, li.item): the unit of catalogs and feeds.

Every one of these is "repeat a container, map through its children", and spotting the pattern in seconds is what makes extraction fast.

Structured Data Hiding in the Head#

Many pages publish their key facts twice: once for humans in the body, and once for search engines as JSON-LD in a <script type="application/ld+json"> block, following schema.org vocabularies (Product, Article, Event, JobPosting). Open Graph <meta property="og:title"> tags are another copy. When they exist, they are the cleanest data on the page.

import json
from bs4 import BeautifulSoup

html = '''
<html><head>
<meta property="og:title" content="Mechanical Keyboard">
<script type="application/ld+json">
{"@context": "https://schema.org", "@type": "Product", "name": "Mechanical Keyboard",
 "sku": "98765", "offers": {"@type": "Offer", "price": "99.99", "priceCurrency": "USD"}}
</script>
</head><body>...</body></html>
'''

soup = BeautifulSoup(html, "html.parser")
for tag in soup.select('script[type="application/ld+json"]'):
    try:
        data = json.loads(tag.string or "")
    except json.JSONDecodeError:
        continue
    items = data if isinstance(data, list) else [data]
    for item in items:
        if item.get("@type") == "Product":
            print(item["name"], item["sku"], item["offers"]["price"])

og = soup.select_one('meta[property="og:title"]')
print("og:title =", og["content"] if og else None)

Expected output:

Mechanical Keyboard 98765 99.99
og:title = Mechanical Keyboard

Real-world JSON-LD is messier: some sites wrap items in an @graph array, emit several blocks per page, or ship invalid JSON with trailing commas. The try/except and the list normalization above handle the common cases.

Text Nodes vs. Elements#

An element's visible text is rarely one clean string. Price: $99.99 is often several adjacent text nodes split by inline tags, icons, and whitespace:

from bs4 import BeautifulSoup

soup = BeautifulSoup('<p class="price"> Price: <b>$99</b>.99 <small>incl. VAT</small></p>', "html.parser")
p = soup.select_one("p.price")
print(repr(p.string))                    # None: more than one child
print(repr(p.get_text()))                # ' Price: $99.99 incl. VAT'
print(repr(p.get_text(" ", strip=True))) # 'Price: $99 .99 incl. VAT'
print(repr(p.find(string=True, recursive=False).strip()))  # 'Price:'

Three lessons are packed in there. .string returns None as soon as an element has more than one child, which surprises everyone once. get_text() joins everything, including text you did not want (incl. VAT). And get_text(" ", strip=True) inserts a separator between every node, which can split a number in half. Pick the method per field, and when a value is split across tags, extract the parent's full text and clean it with a targeted regex.

How Parsers Disagree on Broken HTML#

Real HTML is frequently invalid: unclosed <p> tags, stray </div>, a <div> inside a <span>. Every parser repairs it, and they repair it differently. BeautifulSoup can use html.parser (built in, lenient), lxml (fast, C-based), or html5lib (slow, but repairs exactly like a browser). If a selector works in the browser but returns nothing in Python, try parsing the same document with another backend and compare len(soup.select(...)). The parsing engines lesson benchmarks them.

Recon Your Page Before Writing Selectors#

A short script can do the discovery: count repeating containers, list data-* attributes, and check for JSON-LD and inline state.

import re
from collections import Counter
from bs4 import BeautifulSoup

def recon(html: str) -> None:
    soup = BeautifulSoup(html, "html.parser")
    classes = Counter(c for el in soup.find_all(class_=True) for c in el["class"])
    print("most repeated classes:", classes.most_common(8))
    print("tables:", [len(t.select("tr")) for t in soup.select("table")])
    print("data-* attrs:", sorted(set(re.findall(r"data-([a-z0-9-]+)=", html)))[:20])
    print("json-ld blocks:", len(soup.select('script[type="application/ld+json"]')))
    print("itemprops:", sorted({el["itemprop"] for el in soup.select("[itemprop]")})[:15])

Classes that repeat exactly as often as the number of items on screen (say, 24 times on a page showing 24 products) are almost always your containers. If the page exposes data-sku or itemprop="price", you have found a machine-readable contract that tends to outlive visual redesigns.

Practice#

Save three different product or article pages from a site you are allowed to crawl. Run recon() on each, then write down, for five fields, whether the value lives in text, an attribute, JSON-LD, or a script, and which hook you would target. Check that each hook exists in all three saved pages, not just the first.