PDF and Document Extraction

The Data That Hides Inside Files#

Listings, search results and catalogues live in HTML, but the detail often lives in a document. Regulatory filings, annual reports, tenders, court records, price lists and statistical releases are published as PDFs, Word files and spreadsheets because that is what people print and email. A pipeline that stops at the download link has not finished the job. This lesson takes the file you saved in downloading files and streaming and turns it back into rows.

Detect What You Actually Got#

A URL ending in .pdf can serve an HTML error page, and a Content-Type header can be wrong. The reliable check is the first bytes of the file, its magic number: %PDF- for PDFs, PK for the zip-based Office formats (.docx, .xlsx, .pptx) and plain zips, and PNG after one non-printable byte for PNG images.

def sniff(path: str) -> str:
    with open(path, "rb") as f:
        head = f.read(1024)
    if b"%PDF-" in head:          # some generators put junk before the header
        return "pdf"
    if head[:2] == b"PK":
        return "office-zip"
    if head[1:4] == b"PNG":
        return "png"
    if head.lstrip().lower().startswith((b"<!doctype", b"<html")):
        return "html"             # an error or login page, not a document
    return "unknown"

Searching the first kilobyte for %PDF- rather than checking position zero tolerates files with a few stray bytes in front of the header, which PDF readers generally accept. For office-zip, open it with zipfile and look for word/document.xml or xl/workbook.xml to tell Word from Excel.

Text Layer or Scan?#

PDFs come in two kinds that need completely different tools. A generated PDF (exported from Word, a reporting system, a browser) contains real text objects with positions. A scanned PDF is a stack of page images with no text at all. Check before choosing a strategy:

import pdfplumber

def has_text_layer(path: str, sample_pages: int = 3) -> bool:
    with pdfplumber.open(path) as pdf:
        for page in pdf.pages[:sample_pages]:
            if len(page.chars) > 50:
                return True
    return False

A handful of characters per page usually means a scan with a printed header or page number overlaid. Also look at the extracted text itself: output full of (cid:123) tokens means the font has no usable character mapping, and OCR is the only way to read it.

Choosing a PDF Library#

For most scraping work pdfplumber is the default, because tables and layout are where the value usually is, and it exposes the coordinates you need when automatic extraction fails.

Tables With pdfplumber#

extract_tables() finds tables using the ruling lines drawn on the page. On a clean, gridded table it returns a list of rows, each a list of cell strings (or None for empty or merged cells):

import pdfplumber

with pdfplumber.open("report.pdf") as pdf:
    page = pdf.pages[2]
    for table in page.extract_tables():
        header, *rows = table
        print(header)
        for row in rows[:3]:
            print(row)

['Region', 'Q1', 'Q2', 'Q3', 'Q4']
['North', '1,204', '1,310', '1,188', '1,402']
['South', '980', None, '1,045', '1,120']
['East', '1,511', '1,498', '1,620', '1,705']

Many reports draw tables without lines, using whitespace for alignment. Switch the detection strategy to text alignment, and crop to the region that contains the table so page headers and footnotes do not leak into it:

settings = {"vertical_strategy": "text", "horizontal_strategy": "text"}

# bounding box is (x0, top, x1, bottom) in PDF points from the top-left
region = page.crop((36, 180, page.width - 36, 640))
table = region.extract_table(table_settings=settings)

Finding the crop box is easier visually: page.to_image(resolution=100).debug_tablefinder(settings).save("debug.png") draws the detected cells over the page, so you can see what pdfplumber thinks the grid is and adjust.

Tables that span pages repeat the header row on every page and sometimes split a row across the page break. Drop rows equal to the header after the first page, and merge a row into the previous one when its key column is empty.

When Tables Are Not Tables: Rebuild From Word Positions#

Bank statements, invoices and price lists often look tabular but defeat every table finder. Fall back to the raw material: every word with its coordinates. Group words into lines by their vertical position, then assign each word to a column by its horizontal position.

from collections import defaultdict

COLUMNS = [("date", 0, 90), ("description", 90, 380), ("amount", 380, 470), ("balance", 470, 600)]

def rows_from_words(page, line_tolerance=3):
    lines = defaultdict(list)
    for w in page.extract_words(keep_blank_chars=False):
        key = round(w["top"] / line_tolerance)       # words on the same line share a key
        lines[key].append(w)

    for key in sorted(lines):
        record = {name: [] for name, _, _ in COLUMNS}
        for w in sorted(lines[key], key=lambda w: w["x0"]):
            for name, left, right in COLUMNS:
                if left <= w["x0"] < right:
                    record[name].append(w["text"])
        yield {k: " ".join(v) for k, v in record.items()}

The column boundaries come from measuring one real page: print extract_words() for a known row and read off the x0 values. This approach is brittle across different document templates but very stable within one publisher's series, which is the usual scraping situation. Store the template version with each record so a layout change shows up as a validation failure, not as silently shifted columns.

Cleaning Text That Came From a PDF#

PDF text is laid out for printing, not for reading, and it arrives with artefacts:

• Ligatures. "fi" and "fl" may come out as single characters (fi, fl). unicodedata.normalize("NFKC", text) turns them back into letters.
• Hyphenation. Words broken at line ends arrive as extrac- plus tion. Join a line ending in a hyphen with the next line when the combined word is in a dictionary, or accept some errors for free text.
• Reading order. Two-column layouts can interleave columns line by line. Crop each column and extract it separately.
• Numbers. Accounting formats write negatives as (1,234.50), and many European documents use 1.234,50. Parse per source with explicit rules, never with a guess:

from decimal import Decimal

def parse_amount(raw: str, decimal_comma: bool = False) -> Decimal | None:
    s = raw.strip().replace(" ", "")
    if not s or s in {"-", "–"}:
        return None
    negative = s.startswith("(") and s.endswith(")")
    s = s.strip("()")
    if decimal_comma:
        s = s.replace(".", "").replace(",", ".")
    else:
        s = s.replace(",", "")
    value = Decimal(s)
    return -value if negative else value

Then validate. Most financial documents contain their own check: the column total, the closing balance, a "total items" line. If the sum of your extracted rows does not equal the printed total, a row was missed or a number misread, and you know before the data ships. More on this in data cleaning and validation.

Scanned PDFs Need OCR#

When there is no text layer, you read pixels. The least effort route is OCRmyPDF, which runs Tesseract and writes a new PDF with an invisible text layer placed over the images. Everything above, including pdfplumber's word positions, then works on the output:

ocrmypdf --skip-text --deskew -l eng scan.pdf scan_ocr.pdf

--skip-text leaves pages that already have text alone, so mixed documents are safe to run through it. For more control (preprocessing, confidence scores, per-field OCR), render pages to images and call Tesseract directly, which is covered in OCR extraction. Treat OCR output as lower confidence than text-layer output: re-check numbers against totals, and keep the page number and file hash with each record so a doubtful value can be traced back to the image.

Word Documents and Spreadsheets#

python-docx reads .docx paragraphs and tables:

from docx import Document

doc = Document("tender.docx")
for para in doc.paragraphs:
    if para.style.name.startswith("Heading"):
        print("#", para.text)
for table in doc.tables:
    for row in table.rows:
        print([cell.text.strip() for cell in row.cells])

Merged cells in Word tables appear once per spanned grid position, so a merged cell can show up as repeated identical values in a row. Deduplicate adjacent repeats if that matters for your schema.

For .xlsx, use openpyxl directly or pandas. Load with data_only=True to get the cached results of formulas instead of the formula text. The catch: those cached values exist only if the file was last saved by an application that calculated them, so files generated by scripts can return None for every formula cell.

from openpyxl import load_workbook

wb = load_workbook("prices.xlsx", read_only=True, data_only=True)
ws = wb["Prices"]
for row in ws.iter_rows(min_row=2, values_only=True):
    sku, name, price, *_ = row
    print(sku, name, price)

openpyxl does not read the legacy binary .xls format; pandas can with the xlrd engine installed. For old .doc files, convert them first with LibreOffice in headless mode: soffice --headless --convert-to docx old.doc.

Route, Don't Special-Case#

Keep document handling out of your crawl logic. Build one extract_document(path) that sniffs the file, dispatches to the right reader (text-layer PDF, OCR, Word, spreadsheet), and returns a uniform list of dictionaries with provenance fields: source URL, file SHA-256, page or sheet, and extractor version. The rest of the pipeline then treats a PDF row exactly like a row from an HTML table, and adding a format is one new branch rather than a rewrite.

Practice#

Download a public statistical report as PDF. Extract one table with extract_tables(), then extract the same table with the word-position method, and compare the two outputs against the printed totals. Note which cells each method got wrong and why.