XPath Selectors

The Other Selection Language#

CSS selectors find elements by tag, class, id, and attributes. XPath finds them by their position in the tree and by their content. When you need "the row that contains the word Total", "the value next to the label Price", or "the product card that contains this button", CSS has no answer and XPath has a short one. Real scrapers use both: CSS for simple, stable targets, XPath for anything relational or text-based. XPath is supported natively by lxml, Scrapy and parsel, Selenium, Playwright, and browser DevTools, so the skill transfers everywhere.

One version note: lxml and browsers implement XPath 1.0. Functions from XPath 2.0 and later, such as lower-case(), matches(), or ends-with(), raise errors. Everything in this lesson is XPath 1.0.

The Syntax in Fifteen Minutes#

An XPath expression is a path of steps. // means "any descendant", / means "direct child", . is the current node, .. is the parent, and @ selects attributes. Square brackets hold predicates that filter each step.

The difference between //li[1] and (//li)[1] catches everyone once: a predicate applies to the step it is attached to, so //li[1] returns the first li of every list, not the first on the page.

Selection in lxml#

from lxml import html as lh

page = '''
<div class="grid">
  <div class="product-card featured">
    <h3><a href="/p/kb-104">Mechanical Keyboard</a></h3>
    <span class="price">  $89.00 </span>
  </div>
  <div class="product-card">
    <h3><a href="/p/ms-220">Wireless <b>Pro</b> Mouse</a></h3>
    <span class="price sale">$39.00</span>
  </div>
</div>'''

tree = lh.fromstring(page)
for card in tree.xpath('//div[contains(concat(" ", normalize-space(@class), " "), " product-card ")]'):
    name = card.xpath("normalize-space(.//h3/a)")
    href = card.xpath("string(.//h3/a/@href)")
    price = card.xpath('normalize-space(.//span[contains(@class, "price")])')
    print(name, href, price)

Mechanical Keyboard /p/kb-104 $89.00
Wireless Pro Mouse /p/ms-220 $39.00

Three habits are packed into this example:

• The leading . in .//h3 keeps the search inside the current card. Without it, //h3 searches the whole document from every card, and each card reports the first product's title. This is the most common XPath bug.
• normalize-space(...) and string(...) return a single string instead of a list. The expression evaluates to "" when nothing matches, so there is no IndexError on [0]. That is convenient, but it also hides missing fields, so validate downstream.
• normalize-space(.//h3/a) returns all nested text. The mouse's name spans a <b> tag; .//h3/a/text() would return two separate fragments, "Wireless " and " Mouse", and lose "Pro".

Class Matching Done Properly#

contains(@class, "price") is the popular shortcut, but it is a substring test: it also matches price-old, priceless, and noprice. The exact-token test is verbose but correct:

//span[contains(concat(" ", normalize-space(@class), " "), " price ")]

Padding both the attribute and the class name with spaces means only a whole class token can match. Use the shortcut when you have checked that no similar class names exist on the page, and the exact form in anything long-lived. In parsel or Scrapy, you can also write the CSS span.price and let the library translate it to this XPath for you.

Axes: Walking Sideways and Upward#

Axes are what make XPath uniquely useful. The ones you will use most:

Label/value layouts are where axes pay off. Specification tables and definition lists rarely have a stable class per field, but their labels are stable text:

specs_html = '''
<dl class="specs">
  <dt>Weight</dt><dd>850 g</dd>
  <dt>Switches</dt><dd>Tactile</dd>
  <dt>Price</dt><dd>$89.00</dd>
</dl>'''
tree = lh.fromstring(specs_html)

def spec(label):
    return tree.xpath(
        "normalize-space(//dt[normalize-space()=$label]/following-sibling::dd[1])",
        label=label,
    )

print(spec("Weight"), "|", spec("Price"))   # 850 g | $89.00

Two details: following-sibling::dd[1] takes the nearest following dd, not all of them; without [1] you would get every later value. And $label is an XPath variable, passed as a keyword argument to xpath(). Always pass values this way instead of formatting them into the expression string, because a label containing a quote (Manufacturer's code) breaks a hand-built expression.

ancestor::div[@data-sku][1] is the reverse trick: find a unique child (a button, an icon), then climb to the container that holds the data. On reverse axes such as ancestor:: and preceding-sibling::, position [1] means the nearest one, not the first in document order.

Tables#

Tables are XPath's natural habitat. To extract a column by header name rather than by a hard-coded index, count the header's position first:

def column(tree, header):
    idx = tree.xpath(
        "count(//table//th[normalize-space()=$h]/preceding-sibling::th) + 1",
        h=header,
    )
    return [td.text_content().strip()
            for td in tree.xpath(f"//table//tr[td]/td[{int(idx)}]")]

If the header moves from column 3 to column 4, this keeps working. It assumes no colspan in the header row; tables with merged cells need the fuller approach in the HTML tables lesson.

Text Matching and Its Limits#

Useful text predicates:

• //a[normalize-space()="Next"] exact text, whitespace-insensitive.
• //a[contains(., "Next")] substring anywhere in the element's text, including children.
• //p[starts-with(normalize-space(), "Price:")] prefix match.

XPath 1.0 has no lower-case(), so case-insensitive matching uses translate():

//a[contains(translate(., "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"), "next")]

For regular expressions, lxml supports the EXSLT extension:

NS = {"re": "http://exslt.org/regular-expressions"}
skus = tree.xpath(r'//span[re:test(., "^[A-Z]{2}-\d{3}$")]/text()', namespaces=NS)

Keep text-based selectors for labels and navigation, which change rarely. Matching on content that is translated or A/B-tested makes a scraper that works in one locale and fails in another.

Testing and Debugging XPath#

• In the browser: the DevTools Console accepts $x('//dt[.="Price"]/following-sibling::dd[1]') and returns the matched elements. In the Elements panel, Ctrl+F also accepts XPath.
• Test against the HTML your scraper receives, not the live DOM. Browsers insert tbody into tables and JavaScript modifies the page, so an XPath copied from DevTools such as /html/body/div[2]/table/tbody/tr[3]/td[2] often fails against the raw response. Save the response and test there.
• Rewrite "Copy XPath" output. Absolute, index-based paths break on the smallest layout change. Replace them with an anchor on something meaningful: an id, a stable attribute, or a label's text.
• Compile hot expressions. lxml.etree.XPath("normalize-space(.//h3)") returns a reusable callable, which avoids re-parsing the expression for every element in a large crawl.

Choosing Between XPath and CSS#

Use CSS when the target has a stable id, class, or attribute: it is shorter, and most people on your team read it faster. Use XPath when you select by visible text, need a sibling or ancestor, work with tables or label/value pairs, or need string functions. Parsel (used by Scrapy) lets you mix both on the same selector, for example sel.css(".specs").xpath('./dt[.="Price"]/following-sibling::dd[1]/text()').get(), which is often the cleanest option of all.