Is Web Scraping Legal? The Complete 2026 Guide to GDPR, CCPA, and robots.txt Compliance

By the Data Engineering Team at DataSOS Technologies

Quick Answer

Yes, in most cases. Scraping publicly available data is generally legal in the United States and most of Europe in 2026. Courts have repeatedly ruled that public web pages are not protected by anti-hacking statutes like the Computer Fraud and Abuse Act (CFAA). But legality shifts quickly once three variables change: what you scrape (facts vs. personal data vs. copyrighted content), how you access it (a logged-out public page vs. bypassing a login or CAPTCHA), and what you do with the data afterward (internal analysis vs. resale vs. AI training). This guide breaks down exactly where those lines sit under GDPR, CCPA, and robots.txt as of mid-2026.

Every few months, a procurement team emails us the same question before signing off on a data project: “Is this actually legal?” It usually comes after someone in legal or compliance has read a scary headline about a lawsuit, and now the entire initiative is on hold until someone can answer with certainty.

The honest answer is that web scraping law is not a single rulebook. It is a patchwork stitched together from computer-access statutes, contract law, copyright law, and a growing stack of privacy regulation, and it has been shaped almost entirely by litigation rather than by a clean piece of legislation. That makes “is web scraping legal” a harder question to answer well than most vendors let on. Most articles either oversimplify it (“public data is always fair game”) or bury you in case citations without ever telling you what to actually do.

This guide does both properly: it walks through the court cases that actually determine your legal exposure, translates GDPR and CCPA into plain rules you can apply, explains what robots.txt really does and does not protect you from, and gives you a practical framework for scoping any data collection project so legal risk gets assessed before a single line of code gets written.

The 3 Factors That Actually Determine Your Legal Risk

Most explanations of web scraping law start with case law, which is backwards. Before any court case matters to you, you need to know where your specific project sits on the risk spectrum. In our experience running data extraction programs across 30+ industries, legal exposure comes down to three variables, and they compound.

1. What you scrape

Public, factual data (prices, stock levels, publicly posted job listings, government filings) sits on the safest ground. The moment a data point identifies or could reasonably identify a person, whether that is a name, an email address, a job title paired with a company, or a device fingerprint, it becomes personal data under GDPR and potentially personal information under CCPA, regardless of how public the page looks. Copyrighted material (article text, images, video, original creative writing) is a separate problem again: facts are not copyrightable, but the specific expression of those facts usually is.

2. How you access it

There is a meaningful legal gap between reading a public page with no login and no technical barrier, and bypassing a password, a CAPTCHA, or an explicit access block to get to data that was not intended to be public. US courts have been consistent that the first category does not violate computer-access laws. The second category is where criminal and civil exposure both increase sharply.

3. What you do with the data afterward

Internal competitive analysis, market research, and lead enrichment for your own outreach are treated differently from reselling a scraped dataset, republishing copyrighted content, or feeding personal data into a model without a documented legal basis. Regulators and courts increasingly look past the collection step and ask what happened next.

The practical takeaway: run every data collection request through all three filters before scoping the project, not after. A single “high risk” answer on any one of them should trigger a compliance review before build.

The Court Cases That Built Web Scraping Law

Because there is no dedicated federal “web scraping law” in the US, the rules have been defined almost entirely by litigation. Four cases matter more than the rest.

hiQ Labs v. LinkedIn (2017–2022): the case everyone half-remembers

hiQ Labs was a workforce analytics company that scraped public LinkedIn profiles to build attrition-risk products. LinkedIn sent a cease-and-desist citing the CFAA, hiQ sued for a declaratory judgment, and the case ran for six years across a district court, the Ninth Circuit, and the US Supreme Court.

The part everyone quotes: in 2019, and again on remand in 2022, the Ninth Circuit held that scraping data from a logged-out, publicly accessible page does not violate the CFAA, because there is no unauthorized “access” when anyone with a browser can see the same page.

The part most summaries leave out: that CFAA victory did not end the case. In late 2022, the district court separately found that hiQ had breached LinkedIn’s User Agreement, a contract-law claim rather than a CFAA claim, and that hiQ had gone further by hiring contractors to create fake LinkedIn accounts to pull data that was not actually public. The case ended in a settlement: hiQ paid damages, agreed to a permanent injunction, and destroyed its scraped dataset.

What it means for you: scraping logged-out, public pages is on strong legal footing under the CFAA. Scraping through fake accounts, shared logins, or by circumventing a technical access control is a different legal category entirely, and it is the part of hiQ that actually sank the company.

Van Buren v. United States (2021): why the CFAA got narrower

This case was not about scraping at all. It involved a police officer who used his own database access for an improper purpose. But the Supreme Court’s ruling, that “exceeds authorized access” only applies when someone accesses parts of a system they have no right to access at all, not when they use legitimate access for a purpose the system owner would not approve of, directly shaped every scraping case that followed, including hiQ’s second Ninth Circuit ruling.

Meta Platforms v. Bright Data (2024): the sequel that confirmed the rule

Meta sued data provider Bright Data over scraping of public Facebook and Instagram data. In January 2024, the Northern District of California granted summary judgment to Bright Data, holding that Meta’s own terms of service did not bar logged-out scraping of public data, so Bright Data had not breached Meta’s terms in the first place. Meta dropped the case and waived its appeal the following month. Combined with a similar 2022 ruling in Meta v. BrandTotal, this reinforced that the hiQ logic applies well beyond LinkedIn.

Clearview AI (2020–2025): what happens when the data itself is the problem

Clearview AI scraped billions of face images from social platforms to build a facial recognition database. Unlike hiQ or Bright Data, the legal fight here was never really about the CFAA; it centered on biometric privacy statutes. Clearview settled a US class action in 2025 for roughly $51 million plus a 23% equity stake for plaintiffs, and separately faced enforcement actions across Europe. It is the clearest illustration that access-method legality (was scraping technically “hacking”) and data-type legality (was this a special, protected category of personal data) are two completely separate questions, and you can win one while losing the other.

The EU is a different story

The Court of Justice of the European Union has held that website operators can enforce anti-scraping terms of service through ordinary contract law, even when the underlying database does not qualify for special database-right protection. In practice, this means the CFAA-style “public data scraping is not hacking” shield that protects US scrapers does not automatically carry over to Europe. A scraper operating against EU-based sites needs to take contractual terms more seriously than a US-only operation would.

Quick-reference: risk posture by jurisdiction (public, non-personal data)

GDPR and Web Scraping: What Actually Applies

GDPR applies the moment your scraping touches the personal data of anyone in the EU or EEA, regardless of where your company or servers are based. And the bar for “personal data” is lower than most teams assume: a name, an email address, an IP address, a job title next to a company name, or even a pseudonymous forum post that could plausibly be linked back to a real person all qualify. “Publicly visible” does not mean “not personal data.” It just means the data is easier to find; the legal obligations still apply.

Under Article 6, every processing activity needs a lawful basis. For most commercial scraping, the only realistic option is legitimate interest, and using it correctly requires more than a footnote in your privacy policy:

• Conduct a Legitimate Interest Assessment (LIA) before scraping starts, not retroactively, weighing your business purpose against the individual’s privacy expectations.
• Practice data minimization: collect only the fields your use case actually requires.
• Provide a transparency notice under Articles 13/14 where feasible.
• Build a process for data subject access, correction, and erasure requests.
• Apply extra caution, and a separate Article 9(2) exception, to any special-category data (health, biometric, political or religious opinion, sexual orientation) that shows up incidentally in a scrape.
  
  

Two developments make this a live issue rather than theory. First, France’s CNIL ruled against a company scraping LinkedIn contact details, finding that LinkedIn members had not “manifestly made public” their contact information in the GDPR sense, and that the scraper’s legitimate interest did not outweigh individual privacy rights. That case is a direct warning for anyone treating professional social profiles as fair game simply because a login is not technically required.

Second, and more current: on July 8, 2026, the European Data Protection Board adopted new guidelines specifically addressing web scraping in the context of generative AI, clarifying the lawful basis question and confirming that scraping special categories of personal data requires both an Article 6 basis and an Article 9(2) exception. The Board had already signaled this direction in its December 2024 Opinion 28/2024 on AI models, which endorsed legitimate interest only under a strict three-step test and explicitly named respecting robots.txt and ai.txt as a mitigating factor regulators will look for. Non-compliance carries real teeth: fines up to €20 million or 4% of global annual turnover, whichever is higher.

CCPA/CPRA and Web Scraping: The US Privacy Angle

The California Consumer Privacy Act, as amended by the CPRA, applies to for-profit businesses that do business in California and either exceed $25 million in annual revenue, handle personal information for 100,000 or more California residents or households, or derive 50% or more of revenue from selling personal information.

The single most misunderstood part of CCPA for scraping teams is the “publicly available information” exemption. It is real, but it is much narrower than “anything visible on the internet.” It covers information lawfully made available through government records, and information the consumer themselves made available to the general public or through widely distributed media. It does not blanket-exempt everything sitting on a public web page just because no login was required.

Unlike GDPR, CCPA does not require a lawful basis before you collect data. Its focus is transparency and consumer rights instead:

• If you sell or share scraped personal information with a third party, that is a “sale” or “sharing” under CCPA, which triggers the consumer’s right to opt out.
• Current CCPA regulations give scrapers a practical carve-out on notice: if you collect personal information indirectly (i.e., by scraping rather than direct collection) and you do not sell or share it, you generally do not need to provide a notice at collection.
• The moment you package and sell scraped datasets that include personal information, that carve-out disappears and notice obligations kick back in.
  
  

Other US states are moving fast in the same direction. Virginia, Colorado, Connecticut, Utah, and more than a dozen others now have comparable privacy statutes, and federal privacy legislation remains stalled in Congress as of mid-2026. For any scraping program that touches US consumer data across multiple states, this is quickly becoming a 15-plus-jurisdiction compliance exercise rather than a single-law question.

robots.txt: What It Actually Protects You From (and What It Doesn’t)

robots.txt is one of the most misunderstood files on the internet. It is a voluntary protocol, not a statute. A website publishes it to tell automated crawlers which parts of the site it would prefer not be crawled. Ignoring it is not, by itself, a criminal or civil violation anywhere.

That said, treating robots.txt as irrelevant is a mistake for three practical reasons:

• Courts and regulators increasingly treat robots.txt compliance as evidence of good faith. In hiQ-style disputes, the questions of whether a scraper respected clear access signals and whether it caused harm to the target system both factor into trespass-to-chattels and breach-of-contract analysis.
• The EDPB’s December 2024 Opinion 28/2024 explicitly lists respecting robots.txt and the newer ai.txt convention as a mitigating factor in the legitimate-interest balancing test for AI training data. In effect, robots.txt now carries indirect GDPR weight even though it is not itself a GDPR mechanism.
• Ignoring robots.txt combined with aggressive request rates is the fact pattern most likely to trigger a trespass-to-chattels claim, since that tort turns on actual harm to the target system rather than on the legality of accessing public data in principle.
  
  

Bottom line: robots.txt will not get you sued on its own, and following it will not automatically make an otherwise risky scrape legal. But it is the cheapest, fastest signal of good-faith operation available, and it is increasingly read that way by regulators and courts alike.

Is Web Scraping Legal for AI Training in 2026?

This is the fastest-moving corner of the topic right now, and it deserves its own answer because the rules genuinely changed in 2026.

In the EU, the AI Act’s general-purpose AI (GPAI) provisions took effect on August 2, 2026. Providers of general-purpose AI models must now publish a summary of their training data sources, including the top 10% of domains used for training (or the top 5% or 1,000 domains for smaller providers), and must respect copyright opt-outs asserted under the EU Copyright Directive’s Article 4 text-and-data-mining exception. The Act also directly bans untargeted scraping of facial images to build facial recognition databases, closing off the exact pattern that drove the Clearview AI litigation.

Combine that with the EDPB’s July 2026 web-scraping guidelines and its earlier Opinion 28/2024, and the direction of travel in Europe is unmistakable: scraping for AI training is no longer treated as a gray area. It is a defined, documented, opt-out-respecting compliance process, or it is a liability.

In the US, the picture is messier and still being litigated case by case, largely around copyright and fair use rather than a dedicated AI-training statute. Data facts remain uncopyrightable under the long-standing Feist Publications precedent, but the specific creative expression, article text, images, and original prose, is protected, and several ongoing US lawsuits are testing exactly where AI training fits on that line. Any organization scraping data intended for model training, whether to sell as a dataset or to fine-tune an internal model, should treat that as a materially higher-risk category than scraping the same data for internal business intelligence, and scope the legal review accordingly.

A Practical Risk Framework Before You Scrape Anything

Rather than a generic legal checklist, here is how we actually triage new data collection requests before scoping a build. Use it as a quick self-assessment for your own projects.

Tier 1: Low risk

Public, factual, non-personal data. Logged-out access. robots.txt respected. Reasonable rate limits. Used internally for analysis. This is the everyday case for price monitoring, inventory tracking, and market research, and it sits on the strongest legal footing available.

Tier 2: Moderate risk, needs a compliance layer

Public data that includes personal information, B2B contact details, professional profiles, review author names. Still logged-out and technically accessible, but now GDPR and/or CCPA obligations apply. Requires a documented legitimate-interest basis, data minimization, and a defined retention and deletion process.

Tier 3: High risk, requires legal sign-off before build

Data behind a login wall, protected by CAPTCHA or explicit technical access controls, or covered by a clickwrap terms of service you or your provider actively accepted. This is where breach-of-contract and trespass-to-chattels exposure rises sharply, and where the hiQ settlement actually landed.

Tier 4: Do not proceed without specialist counsel

Special-category personal data (health, biometric, political or religious belief) without a clear Article 9(2) exception, facial recognition data collection, circumventing DRM or paywalls, or bulk collection intended for resale or model training without a documented legal basis and opt-out process.

How DataSOS scopes this in practice

Every engagement starts with a data-and-access audit before a single scraper is written: what fields are being collected, whether any qualify as personal data, whether the target requires a login, and what robots.txt and terms of service actually say. That audit determines the technical architecture, not the other way around. Our team applies schema validation and automated cleaning so the same pipeline that respects access boundaries also delivers accurate, structured data your BI tools can trust; see how that works on our Web Scraping & Data Extraction Services, and how it holds up against hardened anti-bot defenses in our breakdown of proxy chain architecture for Akamai-protected environments.