OCR: Text from Images and Scanned Documents

When the Data Is a Picture#

Some data only exists as pixels: scanned letters and invoices, PDFs that are page images with no text layer, charts and tables published as screenshots, photographed notices and labels. A parser cannot extract what the markup does not contain. Optical Character Recognition (OCR) converts images of text back into text. It is powerful and also the least reliable extraction method in this course, so the skills that matter are preparing images well, configuring the engine for the layout in front of it, and measuring and validating what comes out.

First, Make Sure You Need It#

OCR is the last resort. Before reaching for it, check for text you can read directly:

• A PDF may already have a text layer. Test with pdfplumber first, as shown in PDF and document extraction.
• Images on web pages often have alt text, captions, or a nearby data table with the same numbers.
• Charts rendered in the browser are usually drawn from a JSON payload you can capture in the Network tab.
• An image-only document may have an accessible or text version published elsewhere on the same site.

If none of that exists, OCR it.

Setting Up Tesseract#

Tesseract is the standard open-source OCR engine. pytesseract is a thin Python wrapper that calls the tesseract executable, so both must be installed:

sudo apt install tesseract-ocr tesseract-ocr-deu     # engine plus German language data
pip install pytesseract pillow opencv-python numpy
tesseract --version
tesseract --list-langs

On Windows and macOS, install the engine with the platform installer or package manager, then make sure tesseract is on your PATH, or point the wrapper at it: pytesseract.pytesseract.tesseract_cmd = "C:/Program Files/Tesseract-OCR/tesseract.exe". Forward slashes work fine in Windows paths from Python.

The simplest call reads a clean image:

import pytesseract
from PIL import Image

print(pytesseract.image_to_string(Image.open("notice.png"), lang="eng"))

On a crisp, high-resolution screenshot of printed text, this is often nearly perfect. On a phone photo of a crumpled receipt, it will be poor. The difference is almost entirely image quality.

Preprocessing Does Most of the Work#

Tesseract performs best on dark text on a light, uniform background, with characters large enough to have clear strokes, and lines that are horizontal. Preprocessing moves images toward that ideal:

import cv2
import numpy as np

def preprocess(path: str, scale: float = 2.0) -> np.ndarray:
    img = cv2.imread(path)
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    gray = cv2.resize(gray, None, fx=scale, fy=scale, interpolation=cv2.INTER_CUBIC)
    gray = cv2.medianBlur(gray, 3)                           # remove speckle noise
    binary = cv2.adaptiveThreshold(
        gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 31, 15
    )
    if binary.mean() < 127:                                 # mostly dark: light text on dark background
        binary = cv2.bitwise_not(binary)
    return cv2.copyMakeBorder(binary, 20, 20, 20, 20, cv2.BORDER_CONSTANT, value=255)

What each step is for:

• Upscaling. Small text is the most common cause of garbage output. Screenshots at normal screen resolution often have text only 10 to 12 pixels tall; doubling or tripling it helps markedly. Scans should be rendered at around 300 DPI.
• Median blur. Removes salt-and-pepper noise from scans without smearing letter edges as much as a regular blur.
• Adaptive threshold. Converts to pure black and white using a local threshold per region, which copes with shadows and uneven lighting where a single global cutoff fails. For evenly lit scans, Otsu's method (cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)) picks the global threshold automatically. The block size (31) and constant (15) are starting points to tune on your documents.
• Inversion. Tesseract expects dark text on a light background; white-on-dark banners need inverting.
• Border. Text touching the image edge is often missed; a white margin fixes it.

pytesseract accepts NumPy arrays directly, so pytesseract.image_to_string(preprocess("scan.png")) works.

Straighten Skewed Scans#

Even a few degrees of rotation make lines harder to segment. A simple, dependable deskew tries small angles and keeps the one where text rows are sharpest, measured by the variance of the horizontal projection (rows of text alternate cleanly with blank rows only when the page is straight):

from PIL import Image

def deskew(binary: np.ndarray, max_angle: float = 5.0, step: float = 0.5) -> np.ndarray:
    ink = Image.fromarray(255 - binary)                  # ink = white on black for rotation
    best_angle, best_score = 0.0, -1.0
    for angle in np.arange(-max_angle, max_angle + step, step):
        rotated = np.asarray(ink.rotate(angle, expand=True, fillcolor=0))
        score = rotated.sum(axis=1).var()
        if score > best_score:
            best_angle, best_score = angle, score
    straight = ink.rotate(best_angle, expand=True, fillcolor=0)
    return 255 - np.asarray(straight)

For pages rotated by 90 or 180 degrees (scanned sideways or upside down), Tesseract's orientation detection reports the needed rotation: pytesseract.image_to_osd(img) returns text including a Rotate: line. It requires the osd language data to be installed.

Tell the Engine What Layout to Expect#

The page segmentation mode (--psm) tells Tesseract how the text is arranged. The default assumes a full page and tries to find blocks, which is wrong for a cropped field or a single line.

Restricting the character set helps for known field types. Recent Tesseract versions honour a whitelist with the default neural engine:

digits = "--oem 1 --psm 7 -c tessedit_char_whitelist=0123456789.,-"
total = pytesseract.image_to_string(total_crop, config=digits).strip()

Specify languages explicitly (lang="eng+deu") for multilingual documents; the wrong language model turns accented characters into noise.

Words With Positions and Confidence#

image_to_string throws away where each word was and how sure the engine is. image_to_data keeps both, which is what you need for tables, for extracting fields by position, and for deciding what to send to human review:

from pytesseract import Output
from collections import defaultdict

data = pytesseract.image_to_data(img, lang="eng", config="--psm 4", output_type=Output.DICT)

lines = defaultdict(list)
low_confidence = []
for i, word in enumerate(data["text"]):
    conf = float(data["conf"][i])
    if conf < 0 or not word.strip():
        continue                                   # structural rows (blocks, lines) have conf -1
    key = (data["block_num"][i], data["par_num"][i], data["line_num"][i])
    lines[key].append((data["left"][i], word, conf))
    if conf < 60:
        low_confidence.append((word, conf, data["left"][i], data["top"][i]))

for key in sorted(lines):
    print(" ".join(w for _, w, _ in sorted(lines[key])))
print("low-confidence words:", low_confidence[:10])

Grouping by block, paragraph and line numbers rebuilds reading order. For tables, go one step further: take each word's left coordinate and assign it to a column using boundaries measured from a sample page, exactly like the word-position technique in the PDF lesson. The confidence value (0 to 100) is per word; a threshold around 60 is a reasonable first cut for flagging, to be tuned against your own measured errors.

Extract Fields, Not Pages#

For forms and documents from one template (invoices from one supplier, certificates from one registry), full-page OCR followed by text searching is fragile. Crop the regions where fields live and OCR each with the right mode and whitelist. Store crop boxes as fractions of the page size, so they survive different scan resolutions:

FIELDS = {
    "invoice_no": ((0.62, 0.08, 0.95, 0.12), "--psm 7"),
    "date":       ((0.62, 0.12, 0.95, 0.16), "--psm 7 -c tessedit_char_whitelist=0123456789./-"),
    "total":      ((0.60, 0.80, 0.95, 0.85), "--psm 7 -c tessedit_char_whitelist=0123456789.,"),
}

def extract_fields(page: Image.Image) -> dict:
    w, h = page.size
    out = {}
    for name, ((x0, y0, x1, y1), cfg) in FIELDS.items():
        crop = page.crop((int(x0 * w), int(y0 * h), int(x1 * w), int(y1 * h)))
        out[name] = pytesseract.image_to_string(crop, config=cfg).strip()
    return out

Validate Like You Expect Errors#

OCR confuses shapes: 0 and O, 1, l and I, 5 and S, 8 and B, and it drops or invents punctuation, which is disastrous for decimal points. Build validation in from the start:

• Field-specific repair. In fields known to be numeric, map O to 0, l and I to 1, S to 5 before parsing. Never apply this to free text.
• Format checks. Dates must parse, invoice numbers must match the known pattern, amounts must have at most two decimals.
• Cross-checks. Line items should sum to the subtotal; subtotal plus tax should equal the total. A mismatch pinpoints a misread digit.
• Review queue. Anything failing a check or containing low-confidence words goes to a human with the crop image attached, rather than into the database.

The typed-parsing side is covered in data cleaning and validation.

Measure Accuracy Before Trusting It#

Transcribe 20 to 50 representative images by hand as ground truth, including the worst ones. Then score every configuration change against them instead of eyeballing:

from difflib import SequenceMatcher

def accuracy(predicted: str, truth: str) -> float:
    norm = lambda s: " ".join(s.split())
    return SequenceMatcher(None, norm(predicted), norm(truth)).ratio()

Run the whole set with each preprocessing variant and --psm value, and keep the combination with the best average score and the best worst-case score. This is how you find out, for example, that upscaling helps screenshots but hurts clean 300 DPI scans.

Choosing an Engine#

For the last row, see LLM-assisted extraction; the validation rules above apply even more strongly there. When documents contain personal data such as IDs or medical records, sending them to an external service may be restricted, which often decides the choice by itself.

For large batches with Tesseract, run one process per CPU core with multiprocessing and set the environment variable OMP_THREAD_LIMIT=1, so each Tesseract process does not also start its own thread pool and oversubscribe the machine.

OCR is not a tool for reading CAPTCHAs; that crosses the line described in CAPTCHAs and challenges.

Practice#

Collect 20 scanned or photographed documents of one kind. Transcribe five by hand, run plain image_to_string and the preprocess plus deskew pipeline with two different --psm values, and record the accuracy score of each. Then add one cross-check (for example, totals) and count how many misreads it catches.