From Scheduled Script to Orchestrated Pipeline

A scraper that runs on cron and writes straight to a database works — until it has five stages, a stage that failed at 2am with no trace, or a retry that double-wrote rows. Orchestration is the discipline of running multi-stage data jobs as dependencies, not as one stubborn script: each stage knows its inputs, outputs, retry policy and schedule, and the system decides what to run next.

Why Cron Stops Being Enough

Cron runs a command on a clock. It cannot express "only run load after fetch succeeded, and rerun load if it died midway, but never run fetch again". As soon as a pipeline has more than one stage, you want three tools cron lacks: a recorded dependency graph, per-task retries with backoff, and a definition that is itself the run history. That is what orchestrators (Prefect, Airflow, Dagster) and lighter queued workers give you.

Structure the Job as Tasks

At minimum, split the pipeline into discrete, idempotent functions so any stage can rerun without side effects. The canonical web-data flow:

@dataclass
class Pipeline:
    fetch: Callable[[], list[dict]]   # crawl -> raw records (API or pages)
    clean: Callable[[list[dict]], list[dict]]  # normalize (data-cleaning lesson)
    load:  Callable[[list[dict]], None]        # upsert into storage

def run(p: Pipeline) -> None:
    raw = p.fetch()
    clean = p.clean(raw)
    p.load(clean)

Idempotent means: each task can be repeated safely. load must upsert by a stable natural key (the dedupe from the json-payloads lesson), fetch must be resumable (caching-and-resume), and clean must not mutate its input. With those three properties, an arbitrary retry graph is always safe.

Stage Outputs as Artifacts, Not Assumptions

A subtle orchestration bug: load reading the result of clean through a shared in-memory list. The moment a restarted stage runs in fresh memory, the data is gone. Make each stage's output a persisted artifact — a file, a table, an object-store blob — whose location is passed along rather than assumed. The pipe becomes inspectable at every step ("what did fetch actually return yesterday?") and any stage can rerun against a stored input:

def stage_dir(run_id: str, stage: str) -> Path:
    p = RAW_ROOT / run_id / stage
    p.mkdir(parents=True, exist_ok=True)
    return p

raw_path = stage_dir(run_id, "fetch") / "items.jsonl"
Path(raw_path).write_text("
".join(map(json.dumps, raw)))

With artifacts, "show me what fetch produced for the broken load" becomes ls on the run directory instead of a search through logs. The write-ahead principle from the caching lesson generalizes: a stage that survives only in memory did not survive.

Orchestrators: Fill the Orchestrator-Shaped Hole

A workflow engine takes those functions and adds scheduling, retries, backoff, and run history. Data-flow style frameworks (Prefect, Airflow with the scheduler, Dagster) solve it by making each task a unit with declared dependencies. A minimal Prefect flow looks exactly like your pipeline with a decoration:

from prefect import flow, task

@task(retries=3, retry_delay_seconds=60)
def fetch(): ...

@task
def clean(raw): ...

@task
def load(clean): ...

@flow
def nightly_crawl():
    load(clean(fetch()))

if __name__ == "__main__":
    nightly_crawl()

Even without adopting a framework, the ideas matter everywhere: declare dependencies so the graph is readable, give every task retries + backoff, and let each stage write a small summary row the monitoring lesson can query. State is the pipeline's contract with its operator.

Backfills, Partials and Observability

Two orchestration patterns are worth stealing by hand if you stay script-based. Backfill: to refresh a missed window, run the graph for a specific date range instead of touching today's data — looping run(for_date=...) over the missed days. Partials: a stage that fails should leave everything else intact; wrap each stage so a failure marks only that stage and its dependents, never the whole run. Pair with structured logging (run_id on every line) and one status table (started/finished/items/errors — the scheduling-and-monitoring lesson) and your pipeline explains itself when it wakes you at 4am. Tools are optional; the discipline is not.