Your requests start returning 403s. You check robots.txt, you were following it, and the block makes no sense. It usually is not robots.txt. It is how fast you asked, how many connections you opened, and what you did when the first error came back.
Does robots.txt Apply to Scraping at All?
It is a request published at a fixed address, and nothing enforces it. Any client can ignore it entirely and nothing technical will stop them.
What it does is state clearly what a site owner wants. That matters in a way separate from enforcement — disregarding an explicit request looks poor if anyone ever reviews what you did.
Two misreadings cause most confusion. A Disallow does not make a page private; it stays readable to anyone with the URL. And robots.txt itself is public at a predictable path, which is why listing sensitive directories in it is such a bad idea.
Rules also apply per host. blog.example.com serves its own file and is governed separately from the root domain — checking one and assuming it covers everything is a routine oversight.
Why Scraping Rate Limits Matter More Than Compliance
Bot defences watch traffic patterns, not whether you read a text file. That is the part people find surprising.
You can follow every rule perfectly and still get banned in under a minute by opening twenty parallel connections. To a firewall that is an attack, because in traffic terms it is one.
Reading speed is the useful benchmark. One page every few seconds disappears into normal traffic on any real site. Sequential requests with a pause look like a person; concurrency never does.
In our experience total volume matters far less than shape. A thousand pages over a day is unremarkable. The same thousand in ninety seconds trips everything.
Getting Crawl Delay and Backoff Right
The response codes tell you what to do, and almost nobody listens to them.
| Response | What it means | What to do |
|---|---|---|
| 429 | Too many requests | Back off exponentially, honour Retry-After |
| 503 | Temporarily unavailable | Wait, then retry with a longer gap |
| 403 | Refused | Stop. This is a decision, not a hiccup |
| Challenge page | Bot check served as 200 | Stop. Retrying confirms you are automated |
Retrying immediately after a 429 is the single most common mistake we see. It turns a temporary throttle into a permanent ban, because persistence after a rate limit is exactly the signature they are watching for.
The Crawl-delay directive is worth honouring where a site sets one. It is not in the original specification and Google ignores it, but plenty of other crawlers respect it, and matching it costs you nothing.
Why User Agent Identification Beats Blending In
Say who you are and how to reach you. Something like YourProject/1.0 (+https://yoursite.com/bot).
This feels backwards. A browser user agent gets fewer blocks in the short term, so why volunteer that you are automated?
Because of what happens when something goes wrong. An identifiable crawler causing load gets an email asking it to slow down. An anonymous one pretending to be Chrome gets a silent ban with nobody to appeal to.
It also removes an argument. Disguising your client is a choice someone can characterise later, and it is hard to explain as anything other than deliberate. The wider version of that reasoning is in the legal picture.
The Habit That Cuts Your Footprint Most
Cache everything locally while you develop. This is the least-used improvement available and the most effective.
Most requests during a build are the same pages fetched repeatedly while you fix a selector. Storing responses on disk means the site sees one request per page ever, and your own iteration gets faster because nothing waits on the network.
Request only what you need, too. Skipping images, fonts and stylesheets cuts bandwidth dramatically, and none of it matters if you only want the text.
For genuinely occasional work, a browser tool avoids all of this — our webpage to plain text extractor handles one page at a time, which is under any threshold worth worrying about.
When a Block Is Not a Rate Problem
Sometimes the page was never going to work. An empty result on a JavaScript-rendered site looks like a block and is not one.
Check the status code first. A 200 with no content is a rendering problem, covered in empty responses from dynamic pages. A 403 is a refusal, and slowing down will not change it.
Distinguishing the two saves real time. We have watched people add delays for an hour against a problem that no delay could fix.
There is also a newer question here worth knowing about. Some publishers now block named AI crawlers, and the llms.txt convention tries to address the same territory from the opposite direction — telling automated readers what is worth reading rather than what to avoid.
A Checklist Worth Keeping
Read robots.txt for the host you are actually requesting. Identify yourself honestly. Go sequentially at reading speed. Back off on 429 and stop on 403.
Cache locally, request only what you need, and take only the fields the project uses. None of that is onerous, and together it means you will rarely be blocked at all.
Checking your own site's response headers is a useful way to see what your defences look like from outside — an HTTP header inspector shows what a crawler is told before it ever reaches your content.