Media Scraping: Images, Video & Streams

Beyond Text: Capture the Media#

Catalogues, publishers and galleries keep much of their value in images, video and documents. Scraping media is a different job from scraping text: the URLs hide in more places than a plain <img src>, one picture exists in a dozen sizes, video usually arrives as hundreds of small segments, and every byte costs bandwidth. This lesson covers finding every media reference, choosing the right variant, downloading binaries safely, and proving that what you saved is really the file you wanted.

Where Media URLs Hide#

A browser shows you one image, but the markup often references many. A thorough extractor checks all of these:

• src on <img>, which is frequently a tiny placeholder on lazy-loading sites.
• srcset on <img> and <source> (inside <picture>), a comma-separated list of variants with width (800w) or density (2x) descriptors.
• Lazy-load attributes such as data-src, data-srcset, data-lazy-src or data-original, which a script copies into src when the image scrolls into view.
• Inline CSS: style="background-image: url(...)" on a <div>.
• <meta property="og:image"> and JSON-LD blocks ("image": [...]), which often point at the largest, canonical version.
• <video poster>, <video src> and <source src> children.

Here is an extractor that gathers all of them and resolves relative URLs against the page:

import json
import re
from urllib.parse import urljoin
from bs4 import BeautifulSoup

LAZY_ATTRS = ("data-src", "data-lazy-src", "data-original")
BG_URL = re.compile(r'url\(["\']?([^"\')]+)["\']?\)')

def parse_srcset(value):
    '''Return [(url, score)], where score is width in px or density * 1000.'''
    out = []
    for part in (value or "").split(","):
        bits = part.strip().split()
        if not bits:
            continue
        url, desc = bits[0], (bits[1] if len(bits) > 1 else "1x")
        if desc.endswith("w"):
            score = int(desc[:-1])
        elif desc.endswith("x"):
            score = int(float(desc[:-1]) * 1000)
        else:
            score = 0
        out.append((url, score))
    return out

def extract_images(html, base_url):
    soup = BeautifulSoup(html, "lxml")
    found = {}
    def add(url, score=0):
        if not url or url.startswith("data:"):
            return  # inline placeholders, usually a 1x1 GIF
        full = urljoin(base_url, url.strip())
        found[full] = max(score, found.get(full, 0))

    for tag in soup.find_all(["img", "source"]):
        add(tag.get("src"))
        for attr in LAZY_ATTRS:
            add(tag.get(attr))
        for attr in ("srcset", "data-srcset"):
            for url, score in parse_srcset(tag.get(attr)):
                add(url, score)
    for tag in soup.select("[style*=background]"):
        for url in BG_URL.findall(tag["style"]):
            add(url)
    og = soup.find("meta", property="og:image")
    if og:
        add(og.get("content"), 10_000)  # usually the canonical large image
    for script in soup.find_all("script", type="application/ld+json"):
        try:
            data = json.loads(script.string or "")
        except ValueError:
            continue
        images = data.get("image", []) if isinstance(data, dict) else []
        for url in [images] if isinstance(images, str) else images:
            if isinstance(url, str):
                add(url, 10_000)
    return sorted(found.items(), key=lambda kv: -kv[1])

Feed it a product card with <img src="/p/1-thumb.jpg" srcset="/p/1-400.jpg 400w, /p/1-1200.jpg 1200w"> and a base URL of https://shop.example/item/1, and the first entries come back as https://shop.example/p/1-1200.jpg with score 1200, then the 400w variant, then the thumbnail with score 0. The descending score is how you pick the largest variant, which is what you normally want. Skipping data: URIs matters: lazy-loading sites put a transparent 1x1 GIF in src, and saving thousands of those is a classic silent failure.

If the page builds its gallery with JavaScript, none of this markup exists in the raw HTML. Look for the gallery JSON in the page source or the network tab before reaching for a browser; the infinite scroll and lazy loading lesson covers both routes.

Recognise Video and Streams#

Video pages rarely hand you a single MP4. Streaming uses a manifest: HLS playlists end in .m3u8 and list .ts or .m4s segments, DASH uses an XML .mpd file, and some players only embed a JSON config with a list of stream URLs. Find the manifest by searching the raw page text, or by filtering the browser's Network tab for m3u8 or mpd:

MANIFEST = re.compile(r'https?://[^"\'\s<>]+\.(?:m3u8|mpd)[^"\'\s<>]*')
manifests = set(MANIFEST.findall(page_text))

HLS usually has two levels. The master playlist lists variants at different bitrates; each variant is a media playlist listing the actual segments:

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p/index.m3u8

Pick a variant by BANDWIDTH or RESOLUTION, then read its segments:

def best_variant(master_text, master_url):
    best, best_bw, pending = None, -1, None
    for line in master_text.splitlines():
        if line.startswith("#EXT-X-STREAM-INF"):
            m = re.search(r"BANDWIDTH=(\d+)", line)
            pending = int(m.group(1)) if m else 0
        elif line and not line.startswith("#") and pending is not None:
            if pending > best_bw:
                best, best_bw = urljoin(master_url, line), pending
            pending = None
    return best

def segments(media_text, media_url):
    lines = media_text.splitlines()
    urls = [urljoin(media_url, l) for l in lines if l and not l.startswith("#")]
    is_vod = "#EXT-X-ENDLIST" in lines
    return urls, is_vod

#EXT-X-ENDLIST marks an on-demand playlist with a fixed length. Without it the stream is live: the playlist keeps growing, and you must re-fetch it every few seconds and download only segments you have not seen. Segment URLs are relative to the playlist that lists them, not to the page, which is why urljoin uses media_url.

For assembly, ffmpeg is the dependable tool; it understands both HLS and DASH and remuxes segments into one file without re-encoding:

ffmpeg -i "https://media.example/v/123/1080p/index.m3u8" -c copy video.mp4

Two hard stops. If the playlist contains #EXT-X-KEY with a vendor DRM key format, or the DASH manifest declares ContentProtection for a DRM system, the content is deliberately protected; circumventing DRM is illegal in many jurisdictions regardless of what the scraping ToS says. And even unprotected video is usually copyrighted: collect metadata and links freely, but download the media itself only when you have rights to it (see legal considerations).

Download Safely: Stream, Name, Verify#

Images can be megabytes and video gigabytes, so never hold a whole body in memory. Stream to a temporary file, hash while writing, and rename only on success, so a crash never leaves a half-written file that looks complete. The file downloads and streaming lesson covers the streaming loop in depth; here is the media-specific version:

import hashlib
import os
from pathlib import Path

MAGIC = [
    (b"\xff\xd8\xff", "jpg"),
    (b"\x89PNG\r\n\x1a\n", "png"),
    (b"GIF87a", "gif"), (b"GIF89a", "gif"),
    (b"%PDF-", "pdf"),
    (b"\x1a\x45\xdf\xa3", "webm"),
]

def sniff(head: bytes):
    for sig, ext in MAGIC:
        if head.startswith(sig):
            return ext
    if head[:4] == b"RIFF" and head[8:12] == b"WEBP":
        return "webp"
    if head[4:8] == b"ftyp":
        return "mp4"
    return None

async def download(client, url, out_dir: Path):
    tmp = out_dir / (hashlib.sha1(url.encode()).hexdigest() + ".part")
    sha, head = hashlib.sha256(), b""
    async with client.stream("GET", url) as r:
        r.raise_for_status()
        with open(tmp, "wb") as f:
            async for chunk in r.aiter_bytes():
                if len(head) < 16:
                    head += chunk[:16 - len(head)]
                sha.update(chunk)
                f.write(chunk)
    ext = sniff(head)
    if ext is None:
        tmp.unlink()
        raise ValueError(f"not media: {url} starts with {head[:16]!r}")
    final = out_dir / f"{sha.hexdigest()}.{ext}"
    os.replace(tmp, final)
    return final

Three decisions in that code are deliberate:

• The file type comes from the bytes, not the URL. photo.jpg?sig=abc may be a WebP, and a 200 response may be an HTML error page. If sniff returns None and the head starts with <!DOCTYPE or <html, you received a login wall, a hotlink block or an expired signed URL, not media. Record it as a failure.
• The filename is the SHA-256 of the content. The same logo linked from 5,000 pages is stored once, and remote strings never become paths on your disk. A name built from the URL invites ../ path traversal and characters that Windows refuses.
• os.replace is atomic on the same filesystem, so a reader never sees a partial file.

Store the mapping separately: source_page, media_url, sha256, bytes, type, width, height, fetched_at. That table is what downstream code queries; the files are just blobs.

Download in Parallel, Politely#

Media downloads are IO-bound and byte-heavy, so concurrency pays off, but it must be bounded. A semaphore caps in-flight requests and one shared client reuses connections:

import asyncio
import httpx

async def main(urls, out_dir: Path):
    sem = asyncio.Semaphore(6)
    timeout = httpx.Timeout(10.0, read=120.0)  # slow reads are normal for video
    limits = httpx.Limits(max_connections=6)
    async with httpx.AsyncClient(timeout=timeout, limits=limits,
                                 follow_redirects=True) as client:
        async def one(url):
            async with sem:
                try:
                    return url, await download(client, url, out_dir)
                except (httpx.HTTPError, ValueError) as exc:
                    return url, exc
        return await asyncio.gather(*(one(u) for u in urls))

Media usually comes from a CDN host separate from the HTML host, and the CDN often tolerates more parallelism than the origin. Rate-limit per host rather than globally, as the concurrency lesson describes.

How Media Scrapes Fail#

Practice#

Pick a public-domain image gallery that allows crawling in its robots.txt. Extract every image reference from 20 pages, keep only the largest variant per image, download them with the bounded client above, and produce a table with sha256, type and bytes. Then count how many URLs collapsed into the same hash; that number tells you how much bandwidth deduplication will save on the full crawl.