Extracting with Regular Expressions

Surgical String Extraction#

HTML parsers are the right tool for navigating the DOM, but a lot of valuable data is not shaped like a DOM. It sits inside an inline <script> as a JavaScript assignment, inside a messy text string like Price: $1,299.99 (In Stock), in a URL (/product/48213-mechanical-keyboard), or in a plain-text file. Regular expressions are a compact pattern language for exactly these jobs: finding, validating, and pulling apart strings.

Regex or Parser?#

The last row is the classic rule. HTML allows attributes in any order, optional quotes, nested elements, and comments, and a regex that handles all of those is unreadable and still wrong. Use a parser to get down to the element, then regex on its text.

The Core Functions#

import re

text = "SKU: AB-1234 | SKU: CD-99"

re.search(r"SKU: [A-Z]{2}-\d+", text)          # first match anywhere, or None
re.match(r"SKU", text)                          # match only at the start of the string
re.fullmatch(r"[A-Z]{2}-\d+", "AB-1234")        # whole string must match: use for validation
print(re.findall(r"SKU: [A-Z]{2}-\d+", text))   # ['SKU: AB-1234', 'SKU: CD-99']
print(re.findall(r"SKU: ([A-Z]{2})-(\d+)", text))  # [('AB', '1234'), ('CD', '99')]
for m in re.finditer(r"SKU: (?P<prefix>[A-Z]{2})-(?P<num>\d+)", text):
    print(m.group("prefix"), m.group("num"), m.span())

The findall behavior surprises everyone once: with no groups it returns full matches, with one group it returns that group's strings, and with several groups it returns tuples. finditer is usually clearer, because each item is a match object with named groups and positions.

re.match only checks the beginning of the string, which is a common source of "my pattern never matches". For validation, prefer fullmatch, so that AB-1234xyz does not slip through a pattern that only checks the prefix.

Greedy vs. Lazy#

Quantifiers like * and + are greedy: they take as much as possible and give back only when forced. Across a line of HTML, that eats far more than you meant:

html = '<a href="/a">One</a> <a href="/b">Two</a>'

print(re.findall(r'<a href="(.*)">', html))     # ['/a">One</a> <a href="/b']
print(re.findall(r'<a href="(.*?)">', html))    # ['/a', '/b']
print(re.findall(r'<a href="([^"]*)">', html))  # ['/a', '/b']

The lazy .*? fixes it, but the third form is better: [^"]* says precisely "anything except a closing quote". Negated character classes are faster, because the engine never has to backtrack, and they cannot run past the boundary you care about.

Named Groups and a Real Price Parser#

Prices are the most common regex job in scraping, and they come in many formats. A re.VERBOSE pattern lets you lay the pattern out with comments:

PRICE = re.compile(r'''
    (?P<currency>[$€£]|USD|EUR|GBP)?      # optional symbol or code before the number
    \s*
    (?P<amount>
        \d{1,3}(?:[,.\s]\d{3})*(?:[.,]\d{2})?   # 1,299.99  1.299,00  1 299,00
      | \d+(?:[.,]\d{2})?                       # 45  45.00  45,00
    )
''', re.VERBOSE)

for s in ["Price: $1,299.99 (In Stock)", "Total USD 422.50", "Now only £45", "Prix : 1.299,00 €"]:
    print(s, "->", PRICE.search(s).groupdict())

Output:

Price: $1,299.99 (In Stock) -> {'currency': '$', 'amount': '1,299.99'}
Total USD 422.50 -> {'currency': 'USD', 'amount': '422.50'}
Now only £45 -> {'currency': '£', 'amount': '45'}
Prix : 1.299,00 € -> {'currency': None, 'amount': '1.299,00'}

The last line shows a real limitation: the euro sign comes after the number, so the pattern misses it. Rather than making one regex handle every locale, detect the currency separately, or better, take it from a structured source such as JSON-LD's priceCurrency. Converting the amount to integer cents needs one more decision, which separator is the decimal:

def to_cents(amount: str) -> int:
    digits = re.sub(r"[^\d.,]", "", amount)          # drop spaces and symbols
    if re.search(r"[.,]\d{2}$", digits):             # ends with a 2-digit decimal part
        whole, frac = digits[:-3], digits[-2:]
    else:
        whole, frac = digits, "00"
    return int(re.sub(r"\D", "", whole) + frac)

print([to_cents(a) for a in ["1,299.99", "1.299,00", "1 299,00", "45"]])
# [129999, 129900, 129900, 4500]

This heuristic works for two-decimal currencies. Currencies with zero or three minor digits need a locale-aware library or explicit per-site rules, which the data cleaning and validation lesson discusses.

Extracting Inline JSON Safely#

Client-rendered pages often embed their state as window.__INITIAL_STATE__ = {...};. The common recipe re.search(r"= (\{.*?\});", html) breaks the moment a string inside the JSON contains };:

import json

page = '''<script>window.__INITIAL_STATE__ = {"items": [{"name": "A; b};"}], "n": 1};
var other = 2;</script>'''

print(re.search(r"window\.__INITIAL_STATE__\s*=\s*(\{.*?\});", page).group(1))
# {"items": [{"name": "A; b}      <- cut off inside a string

m = re.search(r"window\.__INITIAL_STATE__\s*=\s*", page)
start = page.index("{", m.end())
data, end = json.JSONDecoder().raw_decode(page, start)
print(data)
# {'items': [{'name': 'A; b};'}], 'n': 1}

Use the regex only to find where the value begins, then let raw_decode parse exactly one JSON value from that position. It understands strings and nesting, and returns where the value ended. If the blob is a JavaScript object literal rather than strict JSON (unquoted keys, single quotes, undefined), raw_decode raises, and you need a JavaScript-aware parser or the underlying API instead. The JavaScript-rendered pages lesson covers where these blobs come from.

Note the escaped dot in window\.__INITIAL_STATE__. An unescaped . matches any character; it usually still works, which is why the bug survives until it matches something unexpected.

Flags You Will Use#

• re.IGNORECASE (re.I): case-insensitive matching.
• re.DOTALL (re.S): lets . match newlines, needed when a target spans lines.
• re.MULTILINE (re.M): makes ^ and $ match at every line, useful for text files and robots.txt-style formats.
• re.VERBOSE (re.X): ignores whitespace and allows # comments in the pattern.
• re.ASCII (re.A): restricts \d, \w, and \s to ASCII.

The last one matters more than it looks. In Python 3, \d on a str matches any Unicode decimal digit, including Arabic-Indic and Devanagari digits. re.findall(r"\d+", ...) can therefore return strings that int() accepts but that you did not expect. When you mean ASCII digits, write [0-9] or pass re.ASCII.

Catastrophic Backtracking#

Some patterns take exponential time on inputs that almost match. Nested quantifiers are the usual cause:

import time

for n in (16, 18, 20, 22):
    s = "a" * n + "!"
    t = time.perf_counter()
    re.match(r"^(a+)+$", s)
    print(n, f"{time.perf_counter() - t:.3f}s")

On a typical machine the time roughly doubles with each extra character; a few more characters and one regex call hangs a worker for minutes. On scraped input, which is long and unpredictable, this is a real outage risk. Avoid nested quantifiers over overlapping classes ((\w+\s?)*, (a|aa)+), prefer specific negated classes, and on Python 3.11 or later use atomic groups (?>...) or possessive quantifiers like a++, which never give back what they matched: ^(?>a+)+$ fails instantly.

Performance and Testing Habits#

• Raw strings always. r"\d+" passes the backslash to the regex engine; "\d+" relies on Python leaving unknown escapes alone and raises a warning in recent versions.
• Compile once. re.compile at module level documents the pattern and avoids repeated lookups in hot loops. Python caches recently used patterns, so the speed gain is modest; the readability gain is not.
• Test against real samples. Keep a list of strings you have actually seen, including the ugly ones, and assert the results:

CASES = {"$1,299.99": 129999, "1.299,00 €": 129900, "USD 45": 4500}
for raw, cents in CASES.items():
    assert to_cents(PRICE.search(raw).group("amount")) == cents, raw

Regex failures are silent: an empty list looks exactly like "no data on this page". Every new format you meet in production goes into the test list before you change the pattern.

Practice#

Collect twenty real price strings from pages you are allowed to scrape, in at least two locales. Write a pattern and a to_cents function that handle all of them, a test list asserting each result, and one deliberately malformed string that must return None rather than a wrong number.