You need data from a public page and someone on your team says scraping is illegal. Someone else says it is fine because the page is public. Both are repeating something they half-remember, and the project stalls on an argument nobody in the room can settle.
Is Web Scraping Legal as an Activity?
Requesting a public page is ordinary browsing. Automating that request does not, by itself, change what happened — a server received a request and chose to answer it.
That is why "is scraping legal" is close to the wrong question. Nobody is prosecuted for issuing an HTTP request. The consequences attach to what the data is, who it belongs to, and what you do with it afterwards.
Three things genuinely shift the picture: copyright in what you collected, personal data inside it, and whether you got past a control to reach it. Everything else is detail.
Before going further: this is context rather than legal advice. Laws differ by country, outcomes turn on facts, and we are not lawyers.
What "Publicly Available Data" Actually Covers
Public means reachable without a login. It does not mean unowned. This is the most expensive assumption we see teams make.
A published article is protected by copyright the moment it exists. A product photograph belongs to whoever made it. A structured listing may attract database rights in some jurisdictions even where individual entries are plain facts.
Facts themselves generally sit differently from the expression of them. You can usually record that a product costs a certain amount. Reproducing the paragraph describing it is a different act.
In our experience the practical test is simple: could you have written this yourself from what you observed, or are you copying someone's work? The first is usually fine. The second needs a reason.
How Web Scraping Terms of Service Change the Question
Most commercial sites prohibit automated collection somewhere in their terms. Whether that binds you is a genuinely contested area, and it turns partly on how those terms were presented.
| Type | How it works | Why it matters |
|---|---|---|
| Clickwrap | You tick a box or press a button to continue | Explicit agreement. Looks much more like a contract. |
| Browsewrap | A link in the footer, no interaction required | Harder to argue someone agreed to something they never saw. |
The practical consequence is worth understanding. Creating an account to reach data means you almost certainly accepted terms explicitly. Reading pages that were open to anyone means you may never have encountered them.
Courts have not landed in one place on this, and we are not going to pretend otherwise by citing a case that suits the point. What is clear is that clicking "I agree" is a materially different position from not clicking anything.
Personal Data Brings Its Own Rules
Data protection obligations attach to the data, not to how you got it. That trips people up more than the copyright question does.
Collect names, email addresses, profile details or anything else identifying a person, and you have duties about why you hold it, how long you keep it, and whether those people should be told. None of that depends on scraping being permitted.
A useful habit: decide before you start whether you need the personal fields at all. Most projects that scrape profiles never use the names, and dropping them at collection time removes an entire category of obligation.
This is also why we do not build email extraction into our tools. The only real use is list building, and it turns a general utility into a personal-data collector.
Where the Risk Climbs Sharply
Three moves change the character of what you are doing.
- Getting past a block. Rotating addresses to defeat a ban means consent was withdrawn and you continued. That reads badly in every direction.
- Reaching behind authentication. Anything requiring a login is not public, and using someone else's credentials is a separate problem entirely.
- Volume that causes harm. Requests heavy enough to degrade a service turn a data question into an infrastructure complaint.
The last one is the easiest to avoid and the most commonly ignored. Our guide to practical scraping etiquette covers what actually causes trouble, which is rarely what people expect.
A Defensible Working Position
Collect the minimum, at human speed, from pages nobody asked you to stay out of.
Identify yourself in a user agent rather than pretending to be a browser. Respect a block the first time. Store facts rather than expression where you can. Drop personal fields you do not need.
None of that is a legal shield, and we would not claim it is. It does mean that if anyone ever asks what you did, the answer is short and reasonable.
For one-off collection at reading speed, a browser tool handles it without any of this becoming relevant — our website content extractor pulls a single page at a time for exactly that reason.
When the Technical Answer Is Simply No
Plenty of pages cannot be scraped regardless of permission. Sites that build content with JavaScript return an empty shell to any simple fetch.
That is worth knowing before a legal conversation, because it sometimes makes the question moot. The mechanics are in the technical reasons extraction fails.
If the goal is feeding content to a model rather than building a dataset, the volume is usually tiny and the questions above rarely bite. Preparing pages for a model covers that workflow, and a URL to markdown converter handles the conversion.
Where a project genuinely depends on the answer, get advice from someone qualified in your jurisdiction. A blog post — this one included — is not that.