Official APIs & Alternative Data Sources

The Best Scraper Is Often No Scraper#

The most advanced evasion technique is not evasion at all—it is choosing a data source that does not need to be evaded. After you have spent days fighting a WAF, it is worth asking whether the same data is available through an official API, a partner program, a public dataset, or a feed. Very often it is.

The Sunk-Cost Trap#

Once you have invested in a stealth pipeline, it is tempting to keep pushing it forever because stopping feels like admitting defeat. But every hour spent maintaining a bypass has an opportunity cost. A disciplined team evaluates data sources on total cost of ownership: engineering time, proxy spend, ban risk, legal exposure, and data quality. A paid API that costs $200 a month can be far cheaper than a fragile scraper that consumes an engineer's week every month.

Official APIs#

Always check for a public API first. Many platforms that appear scrape-hostile offer generous APIs: social networks, marketplaces, financial data providers, and SaaS tools. Even when an API is rate-limited or requires approval, its stability and legality usually beat a bypass. The catch is that APIs sometimes expose less data than the UI or charge per request—quantify both before deciding.

Partner and Licensing Programs#

Some data is available through formal data-licensing agreements, affiliate programs, or partner tiers. These are unglamorous but often give broader, higher-quality, and legally clean access than scraping ever could. For commercial use, a license frequently pays for itself in reduced legal risk alone.

Public Datasets and Open Data#

A surprising amount of data is already published: government open-data portals, academic datasets, company transparency reports, Common Crawl, and community-maintained mirrors. Before scraping, search for an existing snapshot. Reusing a public dataset eliminates the entire collection problem for that source.

Feeds: RSS, Atom, Sitemaps, Webhooks#

Feeds are push- or manifest-based ways to learn about new content without crawling. RSS/Atom give you recent items, sitemaps enumerate URLs, and webhooks deliver events in real time. For "tell me what changed" problems, feeds answer most of the need with none of the evasion.

User-Contributed and Community Sources#

Some ecosystems maintain community APIs or extractor libraries (for example, media tools with per-site extractors). These concentrate community maintenance and can be a legitimate, faster path than building your own. Verify the license and stability, and understand who maintains it before depending on it.

Buy, License, or Build: A Framework#

Score each candidate source on five axes:

1. Coverage: does it contain the fields you need?
2. Freshness: how current is it, and does that meet your use case?
3. Cost: money plus engineering maintenance.
4. Legal/ToS risk: is access sanctioned?
5. Reliability: what is the uptime and support like?

Build a scraper only when the alternatives genuinely fail on coverage, freshness, or cost. When you do scrape, prefer official endpoints and feeds first, and reserve WAF evasion for the narrow set of targets where no legitimate path exists. Choosing not to fight is not giving up—it is engineering judgment, and it is usually what separates a sustainable data operation from a perpetual maintenance burden.

Evaluating Data Sources Before Committing#

The decision to build against scraped data is usually framed as "is it legal/possible?" but the operational question is "is it reliable enough for the decision you are making?" A price you scrape for a dashboard is different from a price you use to charge a customer or to trigger a financial action. The cost of being wrong must be weighed against the cost of using an official channel.

Evaluate each candidate source against these dimensions:

The point of scoring is to make it explicit. A source with 99% availability but schema churn every two weeks may be more expensive than a licensed feed with 95% availability and a stable schema. Record the numbers, not just the verdict.

Source Redundancy and Independence#

Redundancy is not "four feeds". Redundancy is "four independent feeds". If three of your sources ultimately pull from the same official API with slightly different caching, they will all agree when the API is correct and all agree when the API is wrong. That is correlation, not redundancy.

Track lineage explicitly:

A useful rule: count roots, not endpoints. If two endpoints share the same data producer, a quota change or outage hits both. The trace below simulates four sources reporting the same product field; three share the same root, so the effective independence is 2 despite having 4 sources.

from collections import defaultdict

# Four candidate sources for one SKU on one day. A supplier who scrapes the same
# shop you do can hand you a decoy, so four sources is not about speed: it is
# about disagreement, and about knowing whose disagreement to believe.
OBSERVED = {
    "official_api":  {"price_minor": 18500, "currency": "USD", "in_stock": True,
                      "age_h": 1, "trust": 0.95, "root": "the shop"},
    "licensed_feed": {"price_minor": 18500, "currency": "USD", "in_stock": True,
                      "age_h": 3, "trust": 0.90, "root": "the shop"},
    "aggregator_a":  {"price_minor": 19900, "currency": "USD", "in_stock": True,
                      "age_h": 25, "trust": 0.55, "root": "an unnamed crawler"},
    "public_archive": {"price_minor": 17950, "currency": "USD", "in_stock": False,
                       "age_h": 63, "trust": 0.60, "root": "the shop"},
}

FRESHNESS_BUDGET_H = 48
SPREAD_TOLERANCE = 0.02
QUORUM = 0.75
FIELDS = ["price_minor", "in_stock", "currency"]


def live(name):
    return OBSERVED[name]["age_h"] <= FRESHNESS_BUDGET_H


def weighted_vote(names, field, sources=None):
    '''Tally the field across sources, weighted by trust. Returns the winner,
    the share of weight behind it, and who dissented.'''
    pool = sources if sources is not None else names
    tally = defaultdict(float)
    for name in pool:
        tally[OBSERVED[name][field]] += OBSERVED[name]["trust"]
    best = max(sorted(tally, key=repr), key=lambda v: tally[v])
    total = sum(OBSERVED[n]["trust"] for n in pool)
    agree = sum(OBSERVED[n]["trust"] for n in pool if OBSERVED[n][field] == best)
    dissent = sorted(n for n in pool if OBSERVED[n][field] != best)
    return best, agree / total, dissent


def main():
    names = sorted(OBSERVED)
    print("four sources, one SKU, one observation window, {}h freshness budget".format(FRESHNESS_BUDGET_H))
    print()
    print("{:<16} {:>8} {:>6} {:>6}  {}".format("SOURCE", "price", "age_h", "trust", "usable"))
    print("-" * 74)
    for name in names:
        row = OBSERVED[name]
        print("{:<16} {:>8} {:>6} {:>6.2f}  {}".format(
            name, row["price_minor"], row["age_h"], row["trust"],
            "yes" if live(name) else "no, older than the budget"))
    print()

    for label, pool in [("all four sources", names),
                        ("live sources only", [n for n in names if live(n)])]:
        best, share, dissent = weighted_vote(pool, "price_minor")
        print("vote on price_minor, {}".format(label))
        print("  consensus        {}".format(best))
        print("  agreeing weight  {:.0%} (quorum is {:.0%})".format(share, QUORUM))
        print("  dissenters       {}".format(dissent))
        print()

    # The spread between live sources is the alarm, not the mean.
    live_names = [n for n in names if live(n)]
    prices = sorted(OBSERVED[n]["price_minor"] for n in live_names)
    spread = prices[-1] - prices[0]
    mid = (prices[0] + prices[-1]) / 2.0
    naive = sum(OBSERVED[n]["price_minor"] for n in names) / float(len(names))
    print("live prices         {}".format(prices))
    print("live spread         {}  ({:.1%} of the midpoint)".format(spread, spread / mid))
    print("naive mean of four  {:.0f}  <- a number no source actually reported".format(naive))
    print("flag for review     {}".format(
        "no" if spread / mid <= SPREAD_TOLERANCE else "yes, one live supplier is off by more than 2%"))
    print()

    print("{:<14} {:>10} {:>10}  {}".format("FIELD", "CONFIDENCE", "QUORUM", "ACTION"))
    print("-" * 84)
    for field in FIELDS:
        best, share, dissent = weighted_vote(names, field)
        if field == "currency":
            action = "error, not a vote" if dissent else "clean, all sources agree"
        else:
            action = "accept {}".format(best) if share >= QUORUM else "hold, needs a second source"
        print("{:<14} {:>10.0%} {:>10.0%}  {}".format(field, share, QUORUM, action))
    print()

    # Four sources that share a root are one source with extra steps.
    roots = defaultdict(list)
    for name in names:
        roots[OBSERVED[name]["root"]].append(name)
    print("{:<16} {:<22} {}".format("SOURCE", "TRACES BACK TO", "INDEPENDENT"))
    print("-" * 82)
    for name in names:
        siblings = sorted(n for n in roots[OBSERVED[name]["root"]] if n != name)
        print("{:<16} {:<22} {}".format(name, OBSERVED[name]["root"],
                                        "yes" if not siblings else "no, same root as " + ", ".join(siblings)))
    print()
    print("{} sources, {} independent roots. Quorum over 4 is worth less than quorum over 2.".format(
        len(names), len(roots)))


main()

four sources, one SKU, one observation window, 48h freshness budget

SOURCE              price  age_h  trust  usable
--------------------------------------------------------------------------
aggregator_a        19900     25   0.55  yes
licensed_feed       18500      3   0.90  yes
official_api        18500      1   0.95  yes
public_archive      17950     63   0.60  no, older than the budget

vote on price_minor, all four sources
  consensus        18500
  agreeing weight  62% (quorum is 75%)
  dissenters       ['aggregator_a', 'public_archive']

vote on price_minor, live sources only
  consensus        18500
  agreeing weight  77% (quorum is 75%)
  dissenters       ['aggregator_a']

live prices         [18500, 18500, 19900]
live spread         1400  (7.3% of the midpoint)
naive mean of four  18712  <- a number no source actually reported
flag for review     yes, one live supplier is off by more than 2%

FIELD          CONFIDENCE     QUORUM  ACTION
------------------------------------------------------------------------------------
price_minor           62%        75%  hold, needs a second source
in_stock              80%        75%  accept True
currency             100%        75%  clean, all sources agree

SOURCE           TRACES BACK TO         INDEPENDENT
----------------------------------------------------------------------------------
aggregator_a     an unnamed crawler     yes
licensed_feed    the shop               no, same root as official_api, public_archive
official_api     the shop               no, same root as licensed_feed, public_archive
public_archive   the shop               no, same root as licensed_feed, official_api

4 sources, 2 independent roots. Quorum over 4 is worth less than quorum over 2.

The quorum is computed over all sources, but the "independent" line matters. When sources share a root, a unanimous vote is not four independent confirmations, it is one root confirmed through three mirrors. For high-stakes decisions, require quorum across independent roots, not across total sources.

Freshness, Timeliness and Versioning#

Freshness is not a single timestamp. Distinguish:

• Collection time: when the source observed the value
• Ingestion time: when you received it
• Validity window: how long the value is expected to remain correct

If a feed reports "price as of 02:00Z" but you ingest it at 10:00Z, the validity window matters. For fast-moving markets, a 30-minute-old value may be worse than no value. Versioning helps: prefer sources that publish ETags, last-modified, or a content hash so you can detect silent changes. Also track staleness alerts: alert when the age of the freshest usable value exceeds your SLO.

Spread Alarms and Data Quality#

Data quality failures rarely look like "all sources wrong". They look like "one source diverges". Set spread-based alarms:

The example shows price_minor at 62% agreement with the weighted quorum threshold of 75% -- a "hold, needs a second source" case. in_stock at 80% clears. These thresholds must be set per field, because not all fields have the same tolerance for error.

Risk Register for Scraping-Sourced Data#

If you choose to scrape, document the risks explicitly:

A decision record should capture these, along with the numbers that led to the choice. The official APIs and alternative data lesson is the correct counterweight: always evaluate a licensed channel before committing to scraping, and revisit that evaluation when costs or controls change.

Decision Records and Lineage#

For any data that feeds decisions with non-zero cost, keep a short decision record: source set, independence analysis, thresholds, freshness SLOs, and rollback criteria. Also log lineage with every ingested row: which sources contributed, their collection times, the quorum result, and the spread. When something goes wrong, lineage is what lets you answer "did we use bad data, or did our interpretation change?"

One operational habit: treat source additions like code changes. Require tests (schema validation, a canary comparison against ground truth), version pinning, and a staged rollout. This is the same discipline as keeping current: drift is inevitable, and you need a way to detect and roll back without guessing.

The bottom line: prefer fewer, independent, well-documented sources over many correlated ones. Require quorum across independent roots for high-stakes fields, alert on spread and staleness, and never trade independence for convenience.