Beyond Text: Capture the Media

Catalogues, publishers and galleries keep their value in images, video and documents. This lesson is about finding media URLs (they hide in more places than a plain <img src>), choosing the right variant, and downloading binaries safely and in parallel.

Find Every Image Reference

Modern pages stow images in several places: the src, the srcset (size variants separated by commas), lazy-loaded data-src/data-lazy-src attributes on <img> or <source>, background-image CSS on <div>s, and content: URLs in inline styles. A scrape has to look at all of them, because the browser only shows one variant but the page references dozens:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")
candidates = []
for img in soup.find_all("img"):
    candidates.append(img.get("src"))
    candidates += [c.strip().split()[0] for c in (img.get("srcset") or "").split(",")]
    candidates.append(img.get("data-src") or img.get("data-lazy-src"))
candidates = [c for c in candidates if c]

Resolve every candidate with urljoin (the url-handling lesson) and, for srcset, prefer the largest variant when a descriptor like 2x or 1024w says it exists — thumbnail-class tiny images are often the trap.

Recognize Video and Streams

Video pages rarely give you a single MP4. Modern streaming uses a manifest: HLS playlists end in .m3u8 and list .ts/.m4s segments, DASH uses .mpd, and some sites embed only a JSON config ("streams": [{"url": ...}]). Discover the manifest by searching the page for m3u8|mpd|playlist in raw text, or by looking at the Network capture for the manifest call:

import re
manifest = set(re.findall(r'https?://[^"'\s<>]+\.(?:m3u8|mpd)[^"'\s<>]*', page.text))

Downloading a stream means fetching the manifest, then each segment URL it lists, in order, and concatenating the bytes — the file-downloads lesson's streaming loop over the segment list. Some playlists are live (keep appending forever) vs on-demand (bounded); detect the boundary by whether the playlist stops appending a new segment ID.

Verify What You Saved

Do not trust a URL's extension — ?sig=abc hides the real type. Inspect the magic bytes of the downloaded header instead:

import io
from PIL import Image

data = client.get(url).content
fmt = Image.open(io.BytesIO(data)).format   # JPEG / PNG / GIF / WEBP

A .jpg that decodes as PNG gets its correct name; a body that fails to open as an image is an HTML error page wearing a media costume — record it as a failure, not as media. For video, check the container bytes (ftyp for MP4, ebml for WebM) the same way.

Download in Parallel, Politely

Media crawls are byte-heavy and IO-bound, so parallelize — but bound it. A shared semaphore caps concurrency, and a shared session keeps connection reuse without smashing the domain:

import asyncio, httpx

async def save(client, url, path, sem):
    async with sem:
        data = (await client.get(url)).content
    with open(path, "wb") as f:
        f.write(data)

async def main(urls):
    sem = asyncio.Semaphore(8)
    async with httpx.AsyncClient(timeout=60.0) as client:  # video is slow
        await asyncio.gather(*(save(client, u, p, sem) for u, p in urls))

Give video requests long read timeouts, treat every binary like the untrusted input it is (never open with names built from remote data without sanitizing), and you have the whole media pipeline: find every reference, resolve the best variant, verify the bytes, download bounded and parallel.