Disclaimer: this is general information, not legal advice. Laws differ by country, change over time, and turn on specific facts. For any commercial project, talk to a lawyer who knows your jurisdiction.

There is no single "scraping law". Scraping touches several bodies of law at once, and whether a project is low-risk or dangerous depends on four questions:

  1. What are you collecting? Facts, creative works, or personal data?
  2. How are you accessing it? Publicly, behind a login, or by getting around a technical barrier?
  3. What did you agree to? Did you accept terms of service by creating an account?
  4. What will you do with it? Internal analysis, a derived product, republication, or resale?

Most real disputes come from a bad answer to one of these, not from the act of sending HTTP requests.

Computer Access Laws (US: the CFAA)#

The US Computer Fraud and Abuse Act prohibits accessing a computer "without authorization" or "exceeding authorized access". For years companies argued that scraping in violation of their terms, or after a cease-and-desist letter, met that standard.

Two developments narrowed that argument. In Van Buren v. United States (2021), the Supreme Court read "exceeds authorized access" as a question of whether you entered parts of a system that were off-limits to you, rather than whether you used permitted access for a forbidden purpose. In hiQ Labs v. LinkedIn, the Ninth Circuit held that scraping pages anyone can view without logging in likely does not count as access "without authorization" under the CFAA.

That is often summarized as "scraping public data is legal", which overstates it. hiQ's CFAA win did not end the case: later rulings went against hiQ on breach of contract, including its use of fake accounts, and the dispute ended in LinkedIn's favor. The CFAA question is only one of several. Other countries have their own computer misuse laws, such as the UK's Computer Misuse Act, and they are not interpreted the same way.

What still clearly creates computer-access risk: using credentials that are not yours, continuing after your access was explicitly revoked by technical means, and getting past authentication or other barriers designed to keep you out.

Contracts and Terms of Service#

Almost every site's terms forbid automated collection. Whether those terms bind you depends largely on how you encountered them:

  • Browsewrap: terms linked in a footer that you never actively accepted. Courts are often reluctant to enforce these against someone who never saw them.
  • Clickwrap: you ticked a box or clicked "I agree" when creating an account. These are generally enforceable, and scraping through that account can be a breach of contract.

In Meta v. Bright Data (2024), a US federal court found that Meta's terms did not prohibit logged-out scraping of publicly visible data, partly because the terms governed users' conduct while using their accounts. The practical lesson from this and the hiQ case is consistent: logging in changes your position. If a project needs an account, read the terms you are accepting, and assume they apply. The authentication sessions lesson covers the mechanics; this is the reason to think before using them.

Trespass to Chattels and Server Harm#

Trespass to chattels means interfering with someone else's property. In eBay v. Bidder's Edge (2000), a court enjoined an auction aggregator's heavy automated querying of eBay. Later, Intel v. Hamidi (California, 2003) held that the claim requires actual harm or impairment of the system. Either way, the lesson for engineers is the same: a crawler that measurably degrades a site's service is exposed, and a throttled crawler that behaves like a handful of users is not an attractive target. Rate limits are a legal control as well as a technical one; see rate limiting and politeness.

Facts are not protected by copyright. In the US, Feist v. Rural (1991) held that a phone directory's names and numbers were not copyrightable, although an original selection or arrangement of facts can be. Prices, dates, specifications, and statistics are facts. Articles, reviews, photos, product descriptions written by a copywriter, and user posts are creative works owned by someone.

Activity Copyright exposure
Extracting prices and specs to compute trends Low: facts
Storing full article text for internal search or analysis Depends on jurisdiction, purpose, and fair use or local exceptions
Republishing scraped articles, photos, or reviews High: that is reproduction and distribution
Training a model on scraped creative works Actively litigated in several countries; unsettled

Fair use (US) and similar exceptions elsewhere are defenses that apply to using protected expression. They are case-by-case, so "it is fair use" is a conclusion a lawyer reaches about a specific use, not a label you can apply in advance. Circumventing technical protections around copyrighted content, such as paywalls or DRM, can raise separate anti-circumvention issues on top of copyright itself.

Europe: Database Rights and Text and Data Mining#

The EU adds protections that US-trained engineers often miss:

  • The sui generis database right (Database Directive 96/9/EC) protects the investment in obtaining, verifying, or presenting database contents, even when the contents are uncopyrightable facts. Extracting a substantial part of such a database, or repeatedly and systematically extracting small parts, can infringe it. Bulk crawling of an EU directory or listings site is exactly the pattern this right targets.
  • Contract still matters. In Ryanair v. PR Aviation (2015), the Court of Justice of the EU held that where a database is not protected by copyright or the database right, the owner can restrict its use by contract.
  • Text and data mining exceptions in the Digital Single Market Directive (2019/790) allow TDM for scientific research (Article 3) and, more broadly, for anyone (Article 4) unless the rightsholder has reserved their rights in an appropriate way, for online content by machine-readable means. A robots.txt group excluding AI crawlers or a TDM reservation in site metadata can therefore have legal weight, which is one more reason to parse robots.txt properly; see robots.txt and ethics.

Personal Data: The Area Where Enforcement Actually Happens#

Under the GDPR, personal data that is publicly visible is still personal data. Scraping names, photos, emails, usernames, or anything tied to an identifiable person makes you a controller with obligations:

  • A lawful basis, usually legitimate interests, which requires a documented balancing test against the people's rights and reasonable expectations.
  • Transparency. Article 14 requires informing people when you obtain their data from somewhere other than them, with limited exceptions. Poland's data protection authority fined a company in 2019 for failing to notify people whose data it had collected from public registers.
  • Minimization and purpose limitation. Collect only the fields you need, for a defined purpose, and delete them when you are done.
  • Data subject rights. People can ask what you hold, and ask you to delete it.

Several European regulators have fined Clearview AI for building a facial recognition database from scraped public photos. Similar principles appear in other privacy laws, such as the CCPA in California, with different details. If your dataset can be keyed by a person, get privacy advice before you start, not after.

Records Are Your Defense#

An audit trail turns "you scraped us" into a factual conversation. Log, per run, what you fetched and how:

import json
import datetime as dt

def audit(run_id, url, status, logged_in, robots_allowed, delay_s, fields):
    entry = {
        "run": run_id,
        "at": dt.datetime.now(dt.timezone.utc).isoformat(),
        "url": url,
        "status": status,
        "logged_in": logged_in,
        "robots_allowed": robots_allowed,
        "delay_s": delay_s,
        "fields": fields,            # which fields were kept, e.g. ["title", "price"]
    }
    with open(f"audit-{run_id}.jsonl", "a", encoding="utf-8") as f:
        print(json.dumps(entry), file=f)

Keep it alongside a short written record of the project's purpose, legal basis for any personal data, and retention period. It costs almost nothing and is the first thing a lawyer will ask for.

Handling a Cease-and-Desist#

If an operator asks you to stop, stop crawling that site promptly, preserve your logs and data (do not delete evidence), and involve counsel before replying. Continuing after an explicit demand, especially by changing IPs or identities to get around a block, is the fact pattern that turns a manageable dispute into a serious one.

Before You Start a Project#

Answer these in writing:

  • Is every page I need reachable without logging in? If not, what terms did I accept?
  • Does the data include anything about identifiable people? If so, what is my lawful basis, and which fields can I drop?
  • Am I copying facts, or someone's creative expression? Will any of it be republished?
  • Is the source in the EU, and is it a substantial database?
  • Does robots.txt or site metadata reserve rights for my kind of use?
  • Is there an official API, licence, or data partnership that makes the question moot?

If the only way to reach the data is by defeating a login, a paywall, or a block aimed at you, the durable options are an authorized API, a negotiated agreement, or a different data source.