The text is right there on screen. You paste the URL into an extractor and get nothing back — no error, just an empty result. Nothing is broken. The page you are looking at was never in the file the server sent.
Why Web Scraping Fails on JavaScript Sites
A fetch downloads one file. A browser downloads that file and then runs everything it points at. The gap between those two is where the content lives.
On a React, Vue, Angular or Svelte application, the server often sends something close to this:
<body>
<div id="root"></div>
<script src="/static/app.js"></script>
</body>That is the whole page. Your browser downloads app.js, runs it, the code requests data from an API, and only then does anything appear. An extractor stops at the first step, because that is all a fetch does.
So the empty result is accurate. It is reporting what the server sent, which genuinely contains no article.
How to Check Whether a Scraper Returns Empty HTML for This Reason
Open the page, view source, and search for a sentence you can see on screen. Five seconds, and it settles the question.
Present in the source? The content is server-rendered and your problem is somewhere else — headers, a block, or a parsing mistake. Absent? It arrived by script, and no amount of header tuning will bring it back.
The mistake we see most often here is checking the element inspector instead. The inspector shows the live DOM after every script has run, so the text is always there. It tells you nothing about what the server sent.
A second check: disable JavaScript in your browser and reload. That approximates what a fetch receives. Not exactly — your browser still sends cookies and a full header set — but close enough to confirm the diagnosis.
Finding the Data Behind Client Side Rendered Content
If a script fetched the content, something answered that request. Going to the source directly is usually cleaner than fighting the page.
Open the network tab, filter to XHR or Fetch, and reload. Look for a response containing the text you want. Sites built this way almost always have a JSON endpoint behind them, and JSON is far nicer to work with than HTML.
It is also more stable. A redesign changes class names and breaks every selector you wrote; the data endpoint often survives untouched.
Two honest caveats. Some endpoints need a token or a session, which puts you back where you started. And an internal API is not a public one — it can change without notice, and using it may sit differently under a site's terms. Our note on the legal picture covers where that line sits.
Why the Page Is Blank When I Fetch It, But Not Always
Partial results usually mean the site renders some things on the server and some in the browser.
Frameworks like Next.js and Nuxt ship real HTML for the initial view, then hydrate the interactive parts. So the article body is there, and the comments, related items and anything below an infinite scroll are not.
The other cause of intermittent results is a rate limit. A run that worked ten minutes ago and now returns nothing has probably hit a challenge page, which parses as perfectly valid HTML containing none of your content. Check the status code before assuming the site changed.
Adding a browser user agent gets suggested constantly here and does not help. It may get you past a crude bot filter. It does not execute JavaScript, and if the text is not in the HTML, no header brings it back.
What Actually Works
Four options, roughly in order of effort.
- Find the API. Cheapest and most stable when it is available without a token.
- Copy from your own browser. For one page, this beats building anything. Your browser already ran the code.
- Check for a static alternative. Print views, AMP versions and RSS feeds are frequently plain HTML.
- Run a headless browser. It works. It also costs memory, time and a much larger attack surface if you expose it as a service.
We have deliberately not added headless rendering to our tools for that last reason. When our markdown conversion hits a client-rendered page it says so plainly rather than returning a blank file, which at least tells you which problem you have.
The SEO Cost Nobody Mentions
Search engines hit the same wall you did. Google does render JavaScript, but on a second pass that can lag well behind the initial crawl.
The practical result is that client-rendered sites often index slowly, partially, or with the wrong content in the snippet. If your own site is affected, running it through an extractor that reads the raw response shows you roughly what a crawler sees on its first visit.
That is a useful diagnostic even when you have no interest in scraping. An empty extraction of your own page is a warning worth acting on.
Where only the links matter rather than the prose, the link extractor makes the gap obvious quickly — a navigation menu built in JavaScript returns almost nothing.
Working Out Which Problem You Have
Check the source before changing anything. Most of the time lost to this goes on tuning headers for a problem headers cannot solve.
Content in the source means look at your request or your parsing. Content missing means look for the API, or accept that a browser is required.
And if a site keeps refusing you after several attempts, slow down before trying harder. What robots.txt actually governs covers the difference between being blocked and being throttled, which are fixed in opposite ways.