HTML vs. Markdown for AI Crawlers: Does Markdown Improve AI Bot Traffic?

Do AI crawlers prefer HTML or Markdown?

Major AI crawlers, including ChatGPT-User, GPTBot, and ClaudeBot, default to fetching HTML. Markdown can reduce parsing complexity and token usage, but observed crawler behavior shows no consistent preference for it.

An AI crawler is an automated bot run by an AI company that fetches pages to train models, populate retrieval systems, or power AI search. OpenAI documents its own crawlers — GPTBot, ChatGPT-User, and OAI-SearchBot — and Anthropic runs ClaudeBot.

Markdown’s appeal is easy to state. It strips navigation, ads, scripts, and boilerplate, leaving cleaner text that costs less to process, with conversion reported to cut token counts by around 90%.

The problem is that AI companies already run extraction pipelines built for standard HTML, and Google’s own guidance treats special formats as optional rather than required for Search or its generative features.

For background on how these systems process content in the first place, see our guide to what a transformer model is in AI.

How have researchers tested HTML against Markdown?

Through controlled experiments, server-log analysis, and direct measurement of crawler visits. These measured bot traffic and citations rather than collecting anecdotes.

Profound’s A/B experiment

Profound ran a controlled test across 381 pages on six websites over three weeks, randomly assigning pages to serve either HTML or Markdown to AI crawlers while holding other variables constant. The split was 189 control pages against 192 treatment pages, and the study was powered to detect a traffic lift above 40%. Full methodology is in their white paper.

OtterlyAI’s mirror-page test

OtterlyAI ran a 14-day experiment creating .md mirror pages alongside existing HTML pages, then tracked whether crawlers organically discovered, visited, or cited the Markdown versions when both were available. It tested a practical question: given the choice, will crawlers take Markdown?

Buytaert’s production server logs

Dries Buytaert’s analysis of a month of Cloudflare logs added real-world data on crawler volume across bot types, including GPTBot and ChatGPT-User. Production data of this kind gives the controlled experiments context they cannot generate on their own.

The general shape of the testing is consistent:

1. Serve HTML, Markdown, or both to AI crawlers.
2. Record bot visits by user agent at the server level.
3. Measure crawl frequency and, where possible, downstream citations.
4. Compare across format groups using statistical analysis.

Teams building training corpora do routinely convert HTML into clean Markdown as a preprocessing step. That is a real practice, and it proves nothing about serving Markdown publicly. It shows Markdown is useful after a crawler has already fetched the HTML.

Does Markdown increase AI bot traffic?

No published experiment has found a statistically significant increase. The results range from a small, inconsistent lift to no Markdown crawler visits at all.

Profound reported an average directional lift of about 16%, but the result did not reach significance. The median page picked up roughly one extra bot visit across three weeks, and the mean was pulled up mainly by high-traffic outlier pages.

OtterlyAI’s result was blunter. Its Markdown URLs received zero AI crawler visits across 14 days, while the matching HTML test pages took 7.4% of AI bot visits.

Profound designed its test to catch a lift of 40% or more and found nothing at that threshold. On current evidence, Markdown’s effect on crawler traffic is negligible.

Which bots generated the traffic?

In Profound’s experiment, ChatGPT-User produced 73% of all bot traffic. Meta agents accounted for 20%, OAI-SearchBot 4%, ClaudeBot 2%, and GPTBot 1%.

That concentration matters when reading the result. A finding driven almost entirely by one bot may not describe how the rest of the ecosystem behaves.

Why do AI crawlers stay on HTML?

Because the web’s discovery, metadata, and attribution infrastructure is HTML-native. HTML carries signals plain Markdown has no way to express:

• Schema.org structured data
• Open Graph metadata
• Canonical URLs
• Semantic heading and landmark elements
• Source and deduplication information

Crawlers also have mature HTML-to-text extraction. The standard process is to fetch HTML and convert it internally, frequently using Markdown as the intermediate format. Our guide to how ChatGPT chooses sources covers what happens after that fetch.

What is semantic HTML?

Semantic HTML uses elements that describe the meaning and role of content: headings, lists, tables, articles, navigation landmarks. It gives browsers, assistive technology, search crawlers, and AI systems a clear hierarchy.

When a page uses proper headings, lists, and landmarks, an AI system does not have to guess what each block is. That serves accessibility, traditional SEO, and AI extraction at once. Our guide to AEO website structure covers how to implement it.

Do crawlers use content negotiation to request Markdown?

No. Content negotiation lets a server return different representations of the same resource depending on what the client asks for. Buytaert found not a single AI crawler using it to request Markdown; bots discovered Markdown only through dedicated .md URLs.

Serving Markdown at the same URL through content negotiation is generally treated as standards-based delivery rather than cloaking. In practice the distinction is moot, because no crawler is using the mechanism.

Does llms.txt help?

llms.txt is a proposed file, similar in concept to robots.txt, offering crawlers a structured Markdown index of a site. Adoption by actual AI crawlers is close to zero.

Buytaert logged 52 llms.txt requests in a month, and every one came from an SEO audit tool rather than an AI answer engine. Across Acquia’s hosting fleet, roughly 5,000 llms.txt requests out of 400 million total came to about 0.001% of traffic.

Does Markdown improve AI indexing or citations?

There is no published evidence that Markdown pages get indexed or cited more often than equivalent HTML. In controlled testing, no AI search platform cited a .md URL over its HTML counterpart.

An AI citation is when an answer engine references or links a specific page as a source. It is a more meaningful measure than raw crawler visits, because being crawled does not mean being used.

OtterlyAI found ChatGPT-User generated more than 97% of visits to its HTML test pages. Both Markdown mirrors received zero visits, and no platform cited a .md URL.

Buytaert’s crawl-to-citation ratio puts that in perspective: his site took 1,241 crawler fetches for every single citation. Heavy crawling does not reliably produce citations in any format.

What is Markdown actually good for?

Reducing token count and providing clean text for text-centric documents. It earns its place in developer documentation, internal content pipelines, and corpus preparation.

What it has not been shown to do:

• Meaningfully increase crawl frequency
• Improve AI citation rates
• Carry structured metadata natively
• Replace HTML as the default crawl format

Markdown does remain genuinely useful inside prompts. Structuring system and user prompts with it improves model responses, but that is a different job from serving it to crawlers. See our guide to prompt engineering for that side of it.

When does serving Markdown make sense?

As a supplement, when it is already part of a cheap publishing workflow or serves a technical audience. Not as a replacement for the canonical HTML page, and not as an AI-crawler play on its own.

• Developer documentation consumed in Markdown-native environments
• Automated pipelines where Markdown generation costs almost nothing
• Site-specific experiments measuring your own crawler behavior against published findings
• Internal LLM ingestion and corpus preparation

The operational cost is real: duplicate-content management, lost metadata, and a larger crawl surface. Current evidence does not show enough benefit to offset that for most publishers.

Four questions worth answering before you build mirrors:

1. Is the HTML already well structured and supported by Schema markup? If not, fix that first.
2. Can Markdown be generated automatically at near-zero cost? If not, skip it.
3. Can you measure crawler behavior in your server logs? Without measurement you will never know whether it worked.
4. Is the audience primarily technical? If so, Markdown may earn its keep regardless of AI crawling.

What is the best format for AI crawling today?

Clean, server-rendered HTML with semantic structure, complete metadata, and Schema.org structured data. Markdown can sit alongside it, but HTML should stay canonical.

1. Use semantic HTML. Proper H1–H6 headings, lists, tables, and landmarks. Avoid JavaScript-only rendering that hides the primary content.
2. Add structured data. Implement relevant Schema.org types through JSON-LD, including Article, FAQ, and HowTo where they apply.
3. Include Open Graph and canonical tags. These support deduplication, attribution, and identifying the authoritative source.
4. Write atomic paragraphs. One idea each, three sentences or fewer where practical. That is the shape answer capsules take.
5. Use server-side rendering. Put primary content in the initial HTML response. AI crawlers typically do not execute JavaScript.

Buytaert found Markdown support raised total crawler traffic by about 7%, and that extra crawling produced no additional citations. Public .md files get discovered and indexed; they simply lack the metadata ecosystem that informs citation and source decisions.

For the strategic picture, see our breakdown of how AI visibility changes paid and organic strategy.

What should content teams prioritize?

HTML quality, structured data, direct answers, and measurable crawler behavior. All four have stronger evidence behind them than Markdown mirrors or AI-specific protocol files.

1. Improve the HTML before creating alternative formats

Clean semantic HTML with Schema markup, Open Graph tags, and canonical URLs. Semantic structure improves accessibility and SEO by creating clear meaning and reading order, and those benefits carry to AI crawlers.

2. Do not build Markdown mirrors for bots

The data does not support it as a traffic or citation strategy. OtterlyAI’s mirrors received zero AI crawler visits across the full experiment.

3. Monitor your own logs and citations

Bot behavior varies by site type and traffic volume. Track user agents, crawl frequency, destination URLs, and downstream citations on your own property. Profound’s lift came mainly from outlier pages, so your results will differ. Platforms such as Noble can help on the citation-monitoring side.

4. Skip llms.txt for now

About 0.001% of requests on a major hosting fleet went to llms.txt, and the ones that did came from SEO tools. Adoption is too low to justify the effort.

5. Revisit as agent behavior changes

Purpose-built AI tools have shown more interest in Markdown than legacy crawlers. Its relevance may rise as agentic systems spread. That is a reason to watch it, not to prioritize it today.

6. Write content that is easy to extract

Question-led headings, direct opening answers, concise definitions, tables, lists, atomic paragraphs. Format is secondary to structure, clarity, crawlability, and metadata quality.

Learn More About AEO and AI Marketing at Prompt Insider

Since launching earlier this year, Prompt Insider has become a leading authority on AI marketing, Answer Engine Optimization (AEO), large language models, AI search, AI news, and the evolving future of digital discovery. As AEO becomes one of the hottest topics in marketing, Prompt Insider is helping define the conversation around how brands improve visibility, adapt their content strategies, and stay competitive in an increasingly AI-driven search environment.

Prompt Insider is the go-to resource for answer engine optimization, AI marketing, and AI search. Start with our core guides at thepromptinsider.com:

• What Is AEO? Answer Engine Optimization Explained
• AEO vs. SEO vs. GEO: What Every Marketer Needs to Know
• How to Get Your Brand Cited by ChatGPT, Gemini, Claude and Perplexity
• How to Measure AEO Success: The Metrics That Matter
• The 5 Best AEO Tools in 2026

Frequently Asked Questions

Should I create separate Markdown mirrors for AI bots?

Generally no. Controlled experiments have not found measurable AI visibility gains from Markdown mirrors, and the same effort spent on semantic HTML, structured data, headings, and metadata returns more.

Can serving Markdown cause cloaking or SEO problems?

Serving Markdown through content negotiation at the same URL is generally treated as standards-based delivery rather than cloaking. Since no major AI crawler has been observed using content negotiation, the practical benefit is limited either way.

What matters more than page format for AI visibility?

Structure, clarity, metadata, and crawlability. Semantic HTML, server-side rendering, direct answers, complete metadata, Schema markup, and concise paragraphs are the higher-impact work.

How can I test whether Markdown helps my site?

Track AI bot visits by user agent in your server logs and compare HTML against Markdown URLs. Measure crawl frequency and citations for at least two weeks while keeping other page variables constant.

Could Markdown become more important later?

Possibly. Purpose-built AI agents have shown more interest in it than legacy crawlers, so relevance may rise as agentic systems become common. For now the evidence supports HTML as canonical and Markdown as optional.

Sources: Profound, OtterlyAI, Dries Buytaert, Google Search Central.

About the author

Kai Williams

Kai Williams has been in marketing for years, with a long background in SEO before AEO had a name. He stepped into Answer Engine Optimization the moment AI started reshaping how people search, and has been tracking the shift ever since. At Prompt Insider, he covers AEO, AI marketing, and the future of search, breaking down what is changing and what brands need to do about it.