You paste a page into a chat window and the model answers about the cookie banner. Half your context went on navigation, scripts and class names. The article you wanted it to read got truncated somewhere around the footer.
How to Prepare Web Content for an LLM
Strip what carries no meaning, keep what carries structure. That single rule covers most of the work.
Raw HTML fails on both counts. Class names, wrapper divs and inline styles consume tokens and mean nothing to a model. Meanwhile the structure that does matter — where a section starts, which items form a list — survives conversion at almost no cost.
The order we use: convert to markdown, drop navigation and footers, remove links and images unless they earn their place, then check the size before pasting anything.
Worth measuring rather than assuming. Convert one of your own pages both ways and compare the character counts. The ratio varies enormously between a text-heavy article and a marketing page, and published averages will not match your case.
Markdown vs HTML for AI — What Each Costs
Markdown keeps the structure at a fraction of the size.
Compare the same heading three ways. In HTML: <h2 class="section-title">Pricing</h2>. In plain text: Pricing. In markdown: ## Pricing.
The markdown version costs two characters more than plain text and tells the model a section began. The HTML version costs far more and adds a class name nobody needs.
Lists show the same pattern more starkly. Plain text turns a six-item list into a paragraph, and the model has to infer that six separate things were listed. A markdown hyphen makes it explicit.
Our URL to markdown converter does this in one step and reports its rough token count, so you can check the size before committing context to it.
Building a Token Budget That Holds
Decide the ceiling before you extract, not after the context fills.
Three things compete for the same space: the content, your instructions, and the model's reply. Filling the window with content leaves nothing for the answer, which is a failure people rediscover repeatedly.
Where the savings are, roughly in order:
- Navigation and footers. The largest single cut on most pages, and pure noise.
- Links. Substantial on documentation. A model cannot follow them.
- Images. A long CDN URL for information the model cannot see.
- Repeated boilerplate. Cookie notices and legal footers, repeated across every page you feed it.
Keep tables. Flattening one into prose reliably confuses a model about which value belongs to which column, and the markdown version costs very little.
Any character-based token estimate is approximate, including ours. Dividing by four suits English prose and misleads on code or non-Latin scripts.
RAG Content Extraction and Where Chunking Goes Wrong
Split on headings, not on character count. This is the highest-value change most retrieval systems can make.
Cutting every thousand characters slices through the middle of sections. A chunk starts halfway into one topic and ends halfway into another, so neither is retrievable properly and both look partly relevant to everything.
Splitting on structure gives you chunks that are already about one thing. That is why converting to markdown first is worth doing even when a model never sees the markdown — the headings tell you where the boundaries are.
Add the page title and section heading to every chunk. It costs a handful of tokens and transforms retrieval, because a chunk reading "This applies to version 3 and later" is meaningless without knowing what "this" is.
Record when you extracted it, too. Stale content is worse than missing content — a model presents both with identical confidence.
What Stops This Working
Pages that build content with JavaScript return nothing to extract.
No conversion or chunking strategy helps when the fetch receives an empty shell. Check whether the text is in the page source before blaming your pipeline — the diagnosis is in pages that render client-side.
Permission is the other constraint, and it is easy to skip when the volume feels small. Extracting a page to read is one thing; building a retrieval corpus from someone else's content is another. Our note on permission to use what you collect covers where that shifts.
If you are on the publishing side of this, the emerging llms.txt convention is one attempt at pointing agents to clean versions of your own pages rather than leaving them to guess.
A Workflow That Holds Up
Convert, trim, check the size, chunk on headings, label every chunk.
For a single page into a chat window, the first three are enough. For a retrieval corpus, the last two are where quality actually comes from — and they are the steps most often skipped.
When you already hold the markup rather than a URL, strip markup before you paste it in without any fetch involved. And when the goal is a document to keep rather than context to spend, the full extractor gives you meta tags and images as separate groups.
Check what you captured before it reaches the model. Reading the first few lines of an extraction takes seconds and catches the cases where you captured a cookie banner instead of an article.