Regex Cheat Sheet for Web Scraping

Syntax

Python re functions

import re

re.findall(r"\d+-\d+", "1-2 3-4")        # ['1-2', '3-4']
re.findall(r"(\d+)-(\d+)", "1-2 3-4")    # [('1', '2'), ('3', '4')]  groups change the result
re.match(r"\d+", "abc 123")              # None: match() anchors at the start
re.search(r"\d+", "abc 123").group()     # '123'

m = re.search(r"(?P<year>\d{4})-(?P<month>\d{2})", "2024-05")
m.groupdict()                            # {'year': '2024', 'month': '05'}

re.sub(r"(?P<y>\d{4})-(?P<m>\d{2})-(?P<d>\d{2})", r"\g<d>/\g<m>/\g<y>", "2024-05-06")
# '06/05/2024'
re.sub(r"\d+", lambda m: str(int(m[0]) * 2), "a1 b20")   # 'a2 b40'

Always write patterns as raw strings (r"\d+") so Python does not eat the backslashes.

Flags

• Combine with |: re.compile(p, re.I | re.S).
• Scoped inline flags: (?i:h)ello matches Hello and hello, but not HELLO.
• Since Python 3.11, a global inline flag such as (?i) must be at the very start of the pattern, otherwise re.error.

Greedy vs non-greedy

s = "<b>one</b> and <b>two</b>"
re.findall(r"<b>(.*)</b>", s)     # ['one</b> and <b>two']   greedy: runs to the last </b>
re.findall(r"<b>(.*?)</b>", s)    # ['one', 'two']           lazy: stops at the first

A negated class is often clearer and faster than lazy dots: "([^"]*)" grabs a quoted string.

Lookarounds

The third-party regex module allows variable-width lookbehind and \p{L} style Unicode classes; it is a drop-in replacement for most of re.

Ready-made patterns

Each was tested; read the caveat before trusting it on real data.

Helpers for the messy cases

import re
from decimal import Decimal

def to_decimal(text):
    # '1,299.99' / '1.299,99' / '1 299' -> Decimal.
    # The last separator followed by 1-2 digits is the decimal mark.
    s = re.sub(r"[^\d.,]", "", text)
    m = re.search(r"[.,](\d{1,2})$", s)
    whole, frac = (s[:m.start()], m[1]) if m else (s, "0")
    return Decimal(re.sub(r"[.,]", "", whole) + "." + frac)

to_decimal("1,299.99")   # Decimal('1299.99')
to_decimal("1.299,99")   # Decimal('1299.99')
to_decimal("10,5")       # Decimal('10.5')
# Caveat: "1,234" -> 1234 (3 digits are read as thousands, never decimals)

def clean_url(u):
    # Strip trailing punctuation, but keep a ')' that closes a '(' in the URL.
    u = u.rstrip(".,;:!?")
    if u.endswith(")") and u.count("(") < u.count(")"):
        u = u[:-1]
    return u

[clean_url(u) for u in re.findall(r"https?://[^\s\"'<>]+", "(see https://ex.com/a).")]
# ['https://ex.com/a']

Money: parse into Decimal, never float.

JSON inside script tags

Use regex only to find where the JSON starts, then let the JSON decoder find where it ends. A lazy ({.*?}); stops at the first };, even one inside a string.

import json, re

m = re.search(r"window\.__INITIAL_STATE__\s*=\s*", html)
data, end = json.JSONDecoder().raw_decode(html, m.end())

Python gotchas

• \d matches any Unicode digit: re.findall(r"\d+", "٣٤") finds Arabic-Indic digits. Use [0-9] or re.ASCII when you mean 0-9 (int() accepts those digits, other code may not).
• \w matches accented letters (café); with re.A it stops at caf.
• $ also matches before a trailing newline: re.findall(r"\d+$", "123\n") finds 123. Use \Z or fullmatch() for strict checks.
• re.findall with groups returns only the groups. Use (?:...) for grouping you do not want back, or finditer.
• re.match is not "match anywhere": that is re.search.
• Compile patterns used in a loop (PRICE = re.compile(...)); re caches some, but explicit is clearer.

Catastrophic backtracking

Nested quantifiers such as (a+)+$ on a string that almost matches take exponential time: on "a" * 24 + "b" it took about 0.8 s, and each extra character roughly doubles it. Fixes:

• remove the nesting: a+$
• make the inner part atomic or possessive (Python 3.11+): (?>a+)+$, a++$
• anchor and bound repeats: \d{1,10} instead of \d+ when you know the limits
• never run user-supplied patterns over large pages without a timeout

When not to use regex on HTML

Regex sees characters, not elements. This "simple" link extractor misses two of three links and returns an escaped URL:

html = '''<a class="x" href='/one'>1</a><a href="/two?a=1&amp;b=2">2</a><a
href="/three">3</a>'''

re.findall(r'<a href="(.*?)"', html)   # ['/two?a=1&amp;b=2']

from lxml import html as lh
lh.fromstring(f"<div>{html}</div>").xpath("//a/@href")
# ['/one', '/two?a=1&b=2', '/three']

What breaks regex on HTML: attribute order, single vs double quotes, newlines inside tags, entities (&), comments and scripts that contain markup, and nesting.

Best of both: select the element with a parser, then run the regex on element.get_text().