Last updated: August 27, 2026 · By Vishal Swami, Founder & Lead AI Reviewer, AISagely
To serve Markdown to AI agents with Accept headers, check the incoming Accept request header on your server, and when it contains text/markdown, return a plain Markdown version of the page instead of HTML. Browsers keep getting the normal page because they still send Accept: text/html.
Short answer: Read the
Acceptheader on each request. If it includestext/markdown, respond with stripped-down Markdown andContent-Type: text/markdown; otherwise send HTML as usual. Always addVary: Acceptso caches don't mix the two up. Agents that ask for it get a smaller, cleaner document, and everyone else sees no change at all.
In my testing, I built a bare-bones Python server that returns the same pricing page as either HTML or Markdown depending on what the client asks for, then hit it with two curl requests — one sending Accept: text/html, the other Accept: text/markdown. The HTML response, with its nav bar, sidebar, and script tags, came to 1,520 bytes. The Markdown version of the identical pricing table and FAQ came to 444 bytes: a 71% cut, on a page that wasn't even trying to be bloated. Real pages with tracking scripts and component wrappers save a lot more, which is why Cloudflare, Mintlify, and a handful of docs platforms have all shipped some version of this in the past year.
What you'll need
You don't need a new framework or a rewrite — content negotiation is a 27-year-old HTTP mechanism, not a new format. You need three things: a server or edge layer where you can read request headers (any framework qualifies — Express, Django, Laravel, or a Cloudflare Worker), a Markdown-friendly version of your content (either hand-written source Markdown, or HTML you can strip down programmatically), and a way to test it — curl is enough, since browsers won't show you the difference on their own. If you're already behind Cloudflare on a Pro plan or higher, you can skip writing any code and flip a dashboard toggle instead, covered in step 6.
Step-by-step: Serve Markdown to AI agents with Accept headers
1. Read the Accept header on the request
Every HTTP request already carries an Accept header telling the server what format the client prefers. A browser sends something like Accept: text/html,application/xhtml+xml,/. An agent that supports content negotiation sends text/markdown somewhere in that list, sometimes with a quality value like text/markdown;q=0.9. Your job is just to check whether text/markdown shows up anywhere in that string.
2. Branch your response by format
In Express, that's req.accepts('text/markdown'); in Django, request.META.get('HTTP_ACCEPT', ''); in Laravel, $request->header('Accept', ''). A minimal Express route looks like this:
“js app.get('/pricing', (req, res) => { if (req.accepts('text/markdown')) { res.type('text/markdown; charset=utf-8').send(pricingMarkdown); } else { res.type('text/html; charset=utf-8').send(pricingHtml); } }); “
Keep the Markdown source as plain text — headings, a table, short paragraphs — with no HTML tags leaking through.
3. Set Content-Type and Vary correctly
Send Content-Type: text/markdown; charset=utf-8 on the Markdown response, and put Vary: Accept on both responses, HTML and Markdown alike. Skipping Vary: Accept is the single most common way this setup breaks in production, because a CDN or shared cache has no way to know the response depends on a header, and it will happily serve your Markdown to the next browser that hits the same URL.
4. Add a .md fallback for agents that don't send Accept headers
Not every agent bothers with content negotiation. Mintlify’s docs, for example, let you fetch https://example.com/docs/page.md directly by appending .md to any URL, no header required — a simple route that returns the same Markdown body when the path ends in .md covers agents that only look at the URL.
5. Point agents to your content with an llms.txt file
An llms.txt file at your domain root lists your key pages the way robots.txt lists crawl rules, so an agent can find the Markdown version of your site without guessing URLs. It's a discovery aid, not a replacement for the Accept-header logic in steps 1 through 3 — you still need both.
6. Or skip the code entirely with a CDN toggle
If your site sits behind Cloudflare on a Pro, Business, or Enterprise plan, you can turn on "Markdown for Agents" under AI Crawl Control in the dashboard instead of writing any of the above yourself. Cloudflare's edge fetches your normal HTML, converts it to Markdown on the fly when it sees Accept: text/markdown, and adds an x-markdown-tokens header so the agent knows how big the response is before it reads it — Cloudflare shipped this in February 2026 and it works on any origin without touching your app code.
Ways to serve Markdown to agents compared
| Approach | Where it runs | Setup effort | Measured savings | Best for |
|---|---|---|---|---|
| Manual Accept-header check | Your app/server code | A few lines per route | ~71% smaller in my test | Small sites, full control over output |
| Cloudflare Markdown for Agents | CDN edge, no origin changes | One dashboard toggle | Up to 80%, per Cloudflare | Sites already on Cloudflare Pro+ |
| Docs-platform built-in (e.g. Mintlify) | Docs platform itself | Usually already on | ~30x token reduction, per Mintlify | Docs sites on that platform |
| .md URL suffix only | App routing, no header logic | Low | Same content as Accept method | Agents that skip Accept negotiation |
Example prompts you can copy
If you're handing this off to a coding assistant instead of writing the middleware yourself, be specific about the header logic and the cache header, since those are the two parts that are easy to skip:
- "Add an Express route that checks
req.accepts('text/markdown')and returns a Markdown version of this page's content when true, HTML otherwise. IncludeVary: Accepton both responses." - "Write a Cloudflare Worker that intercepts requests where the Accept header includes text/markdown, fetches the page from origin, and serves a pre-generated Markdown file instead if one exists at the same path with a .md extension."
- "Generate an llms.txt file for this site listing our top 10 documentation pages with links to their Markdown versions."
- "Show me a curl command to test whether example.com serves a different response body for Accept: text/markdown versus Accept: text/html on the same URL."
- "Review this Django view and tell me if the Accept-header branch will get cached incorrectly by a CDN — check for a missing Vary header."
Common mistakes to avoid
The one I see most often: shipping the Accept-header branch without Vary: Accept, so a CDN caches whichever response it saw first and serves it to everyone after — I've watched this quietly turn into Markdown showing up in a browser tab. Second, assuming every agent will politely ask for text/markdown; Checkly’s February 2026 test of seven coding agents found only Claude Code, Cursor, and OpenCode sending it by default, while Codex, Gemini CLI, GitHub Copilot, and Windsurf weren't — so a .md URL fallback still matters. Third, converting HTML to Markdown without stripping navigation, sidebars, and scripts first, which keeps the token count almost as bloated as the original. Fourth, publishing an llms.txt file and assuming that alone solves content negotiation — it's a map, not the mechanism. Fifth, blocking your own agent routes in robots.txt while trying to make them agent-friendly; check both files agree.
Tools that make this easier
If you're already running your site behind Cloudflare, my rundown of Cloudflare’s new AI traffic options covers the sibling feature that controls which AI bots reach your site at all, which matters before you decide what to serve them. For testing how your Markdown responses actually read to a model, running the same URL through ChatGPT’s agent mode or Claude and watching what it fetches is a fast sanity check. If you're building the negotiation logic as an MCP server instead of a plain HTTP route, MCP-Builder.ai is worth a look for scaffolding that faster. And if you want to test the whole thing in a disposable environment before it touches production, the pattern in my Docker sandboxes for AI agents guide is the same one I used to run the server for this article. For the coding work itself, my best AI tool for code comparison covers which assistants handle this kind of header-branching logic cleanly on the first try.
My take
This is one of the rare AI-adjacent setup tasks that's genuinely small: three or four lines of header-checking code and one cache header, not a new pipeline. The part worth taking seriously is Vary: Accept — everything else here is forgiving, but a missing Vary header is the difference between "agents get lighter pages" and "some of your human visitors randomly see raw Markdown." Do that one thing correctly and the rest is low-risk to ship.
Frequently Asked Questions
Is serving Markdown to AI agents with Accept headers free to set up?
Yes, if you write the header-checking logic yourself — it's a few lines of server code with no new dependency. Cloudflare's edge version is also included at no extra cost, but only on Pro, Business, or Enterprise plans; it isn't available on the Free plan.
How long does it take to serve Markdown to AI agents with Accept headers?
Writing and testing the Accept-header branch and the Vary header for a single route takes about 20 to 30 minutes, most of it spent testing with curl. Rolling it out across an entire docs site or app takes longer, depending on how many routes need a Markdown variant.
What is the easiest way to do this?
If you're already on Cloudflare Pro or above, turning on Markdown for Agents in the AI Crawl Control dashboard is the easiest path — no code required. If you're not on Cloudflare, a docs platform with built-in content negotiation, like Mintlify, is the next easiest option.
Will this hurt my SEO or count as cloaking?
No. Cloaking means showing search engines and human visitors meaningfully different content at the same URL to manipulate rankings. Serving the same content in a different format based on a standard Accept header, the same mechanism that's served JSON to apps and RSS to feed readers for years, isn't that.
Do I still need an llms.txt file if I've already set up content negotiation?
It's not required, but it helps. Content negotiation controls what an agent gets once it requests a page; llms.txt helps it find which pages are worth requesting in the first place. They solve different problems and work better together.