Parsing Engines and Text Encoding

The Depth Behind the Selectors#

Earlier lessons used BeautifulSoup and lxml as if they were interchangeable. They are not. The parser decides how fast you can process pages, what tree you get from broken markup, and, together with the decoding step before it, whether the text in your dataset is correct at all. This lesson covers the parser options, how they differ on real-world HTML, and the encoding layer that silently corrupts datasets.

The Parser Menu#

Practical defaults:

• Readable code, good speed: BeautifulSoup(html, "lxml"). You keep BeautifulSoup's friendly API with lxml doing the parsing.
• Maximum control and XPath: lxml directly, or parsel if you like Scrapy's .css()/.xpath()/.get() style.
• Maximum throughput, CSS only: selectolax.
• Must match what the browser sees on badly broken HTML: html5lib, accepting the speed cost.

Never leave html.parser as the default in a crawler that will process millions of pages without measuring first.

Measure on Your Own Pages#

Relative speed depends heavily on page size and structure, so benchmark with real pages from your target, not with numbers from someone's blog post:

from timeit import timeit
from bs4 import BeautifulSoup
from lxml import html as lh
from selectolax.lexbor import LexborHTMLParser

html_text = open("sample_page.html", encoding="utf-8").read()
N = 50

tests = {
    "html.parser": lambda: BeautifulSoup(html_text, "html.parser").select("div.product-card h3"),
    "bs4 + lxml":  lambda: BeautifulSoup(html_text, "lxml").select("div.product-card h3"),
    "lxml xpath":  lambda: lh.fromstring(html_text).xpath('//div[contains(@class,"product-card")]//h3'),
    "selectolax":  lambda: LexborHTMLParser(html_text).css("div.product-card h3"),
}
for name, fn in tests.items():
    print(f"{name:12} {timeit(fn, number=N) / N * 1000:7.2f} ms per page")

Measure parse plus selection together, as above, because that is what your crawler does. On typical pages, html.parser is usually several times slower than lxml, and selectolax is usually faster still. Also check the results are equal across engines: a faster parser that returns 23 products where the others find 24 is not faster.

Parsing is rarely the bottleneck in a polite crawler (network waits dominate), but it becomes one when you re-parse a cache of millions of stored pages after a selector fix. That is when the engine choice pays off.

Broken Markup: Same Input, Different Trees#

Real HTML is full of unclosed tags, misnested elements, and stray closing tags. Each engine repairs it differently, and the repaired tree is what your selectors run against. The BeautifulSoup documentation uses this tiny input to show it:

from bs4 import BeautifulSoup

broken = "<a></p>"
for parser in ("html.parser", "lxml", "html5lib"):
    print(f"{parser:12}", BeautifulSoup(broken, parser))

html.parser  <a></a>
lxml         <html><body><a></a></body></html>
html5lib     <html><head></head><body><a><p></p></a></body></html>

html5lib implements the HTML5 parsing algorithm, the same one browsers use, so its tree matches what you see in DevTools. lxml's libxml2 predates that standard. The practical consequence: a selector that works in DevTools can fail with lxml on badly broken pages, typically around tables (browsers insert tbody; libxml2 does not) and around elements that browsers move out of invalid positions. When a selector returns nothing on one page but works on others, parse that page with html5lib and compare the trees before rewriting the selector.

Selectolax in Practice#

selectolax's API is small and different from BeautifulSoup's:

from selectolax.lexbor import LexborHTMLParser

tree = LexborHTMLParser(html_text)
for card in tree.css("div.product-card"):
    title = card.css_first("h3")
    link = card.css_first("a")
    print(
        title.text(strip=True) if title else None,
        link.attributes.get("href") if link else None,
    )

css_first returns None when nothing matches, so guard every call. There is no XPath, so label/value and sibling-based extraction needs more code. selectolax fits best in high-volume pipelines with simple, stable CSS targets.

Parse Less#

Two techniques cut parsing cost without changing engine:

• Parse only what you need. BeautifulSoup's SoupStrainer builds only matching elements: BeautifulSoup(html, "lxml", parse_only=SoupStrainer("div", class_="product-card")). It does not work with html5lib.
• Skip parsing entirely when the data is JSON. Many pages embed their data in a <script type="application/ld+json"> or <script id="__NEXT_DATA__"> block. Extract that one element and json.loads it; it is faster and more stable than walking the rendered markup.

Why Encoding Breaks Datasets#

Parsers work on text, but the network delivers bytes. Something has to decode those bytes, and if it picks the wrong encoding you get mojibake: café stored as café, quotes turning into ’, or replacement characters �. Nothing raises an error. The corrupted text flows straight into your database.

A page can declare its encoding in several places, in this order of precedence in browsers: a byte order mark at the start of the body, the charset parameter of the Content-Type header, and a <meta charset> tag near the top of the document. Libraries use different subsets of these:

• requests sets resp.encoding from the header. If a text/* response has no charset, it falls back to ISO-8859-1, as old HTTP specifications required, and ignores the <meta> tag. resp.apparent_encoding offers a statistical guess from the bytes.
• httpx uses the header charset and otherwise falls back to UTF-8 by default. You can pass default_encoding to the client, including a function that detects the encoding from the bytes.
• BeautifulSoup given bytes (not a string) looks for a BOM and the <meta> declaration, then guesses. soup.original_encoding tells you what it decided.
• lxml given bytes also honours <meta charset>. Given a str, it trusts that you already decoded correctly.

The Correct Order of Operations#

Fetch bytes, resolve the encoding once, decode once, then parse. The simplest robust version is to hand bytes to a parser that reads <meta> itself, and only override when the header is explicit:

import re
from bs4 import BeautifulSoup

META_CHARSET = re.compile(rb'<meta[^>]+charset=["\']?([\w-]+)', re.I)

def decode_html(resp):
    raw = resp.content
    header = resp.headers.get("content-type", "")
    m = re.search(r"charset=([\w-]+)", header, re.I)
    if m:
        enc = m.group(1)
    else:
        m = META_CHARSET.search(raw[:2048])
        enc = m.group(1).decode() if m else "utf-8"
    if enc.lower() in ("iso-8859-1", "latin-1", "us-ascii"):
        enc = "windows-1252"          # what browsers actually do with these labels
    return raw.decode(enc, errors="replace"), enc

text, enc = decode_html(resp)
soup = BeautifulSoup(text, "lxml")

The windows-1252 substitution is not a hack. The WHATWG Encoding Standard, which browsers follow, maps the labels iso-8859-1 and us-ascii to windows-1252. The two differ in the byte range 0x80 to 0x9F, where windows-1252 has curly quotes, dashes, and the euro sign. Decoding such a page as true ISO-8859-1 turns them into invisible control characters.

errors="replace" keeps one bad byte from crashing a whole page. Count the � replacement characters in the output; more than a handful means the encoding is wrong and you should log the URL.

Diagnosing and Repairing Mojibake#

The pattern of garbage tells you what happened:

The first case is reversible if nothing else has touched the text:

bad = "café"
print(bad.encode("windows-1252").decode("utf-8"))   # café

The ftfy library automates this detection and repair (ftfy.fix_text(bad)) and is useful for cleaning data you already stored. The better fix is at decode time: if you kept the raw bytes (see caching and resume), re-decode them correctly instead of repairing strings after the fact.

Keep a sample of raw bytes and the declared charset for each site you scrape, the same habit the data cleaning and validation lesson recommends for layouts. When encoding problems appear months later, the sample tells you whether the site changed or your code did.

Practice#

Collect 20 pages from three different sites, including at least one that is not in English. Parse each with html.parser, lxml, and selectolax, time them, and compare the number of items each extracts. Then decode every page both with resp.text and with decode_html, and count replacement characters and Ã sequences in each version.