Convert the Grid Back into Data

A table is one of the few HTML elements that encodes structure the browser can already understand. Every <table>, <tr>, <th> and <td> is a row-and-column grid waiting to become a list of Python dicts. When a target renders statistics, prices, schedules or comparison matrices, it almost always uses a table — and you can beat it with a parser instead of a browser.

The Anatomy of a Table

A well-formed table has three zones. The <thead> holds header cells (<th>), the <tbody> holds the data rows, and each row is a <tr> whose children are either <th> or <td> cells. That structure is your contract: iterate rows, read cells in order, pair them with header names, emit a dict per row.

import requests
from bs4 import BeautifulSoup

html = requests.get("https://example.com/stats").text
soup = BeautifulSoup(html, "lxml")
table = soup.select_one("table#stats")

headers = [th.get_text(strip=True) for th in table.select("thead th")]
rows = []
for tr in table.select("tbody tr"):
    cells = [td.get_text(strip=True) for td in tr.find_all(["td", "th"])]
    rows.append(dict(zip(headers, cells)))

That is the entire trick in five lines: find_all(["td", "th"]) handles cells no matter which tag holds them, and zip pairs them to the header row from thead. If the page has no thead, take the first row as headers and slice it off before emitting records.

Surviving Rowspan and Colspan

Real tables violate the tidy contract. A cell with rowspan="2" claims two rows; colspan="3" claims three columns. If you ignore the attributes, columns silently misalign and every row after the span is garbage. The robust fix is to reconstruct the grid manually with a column bookkeeper: track a per-row column cursor, fill spanned cells with their value, and shift the cursor by the span for others.

def table_to_rows(table):
    rows = []
    for tr in table.find_all("tr"):
        row, col = [], 0
        for cell in tr.find_all(["td", "th"], recursive=False):
            while col < len(row) and row[col] != "__skip__":
                col += 1
            text = cell.get_text(strip=True)
            colspan = int(cell.get("colspan", 1))
            for _ in range(colspan):
                row.append(text)
            col += colspan
        for cell in row:
            if cell == " ":
                cell = ""
        rows.append(row)
    return rows

The bookkeeper works because a spanned cell reserves colspan slots in the flattened row, and the same width is assumed for every row — validate your assumption by checking max(len(r) for r in rows) on a sample before trusting the dataset.

A cleaner common-case shortcut is pandas.read_html, which handles spans, mixed headers and missing cells for you and returns DataFrame objects directly. It is not magic — it is the same column-reconstruction logic, already written. Use it the moment your table has a single rowspan, because hand-rolled span handling is where table scrapers die.

import pandas as pd

frames = pd.read_html("https://example.com/stats")
df = frames[0].dropna(how="all").astype(object)
records = df.reset_index(drop=True).to_dict("records")

When the "Table" Is Not a Table

Many modern dashboards render the same grid as a pile of <div>s with utility classes, because the design team "does not use tables". The data is still there, just spread across repeated card markup. Recover it by treating the repeated element as the row and the labeled spans inside it as the cells.

cards = soup.select("div[data-row]")
for card in cards:
    fields = {el.get("data-key"): el.get_text(strip=True)
              for el in card.select("[data-key]")}
    rows.append(fields)

Test your assumption with len(cards) before committing to it: if zero cards match, re-OpenDevTools and check the real class names instead of guessing.

Normalize Into Records

Tables come with formatting that must die before storage: "$1,234.56", "12%", trailing units, empty cells rendered as " " (a non-breaking space that strip() will not remove — replace " " explicitly). Keep the header-to-dict mapping and apply the same hygiene function from the cleaning lesson to every cell so a reported number is mutable before it reaches the database.

Tables are the one place a site hands you structure for free. Read them with a parser, break the rowspan rule only through pandas, and normalize before you store; the grid becomes a dataset with no JavaScript involved.