Short Answer: What is llms.txt?
llms.txt is an optional Markdown file placed at yourdomain.com/llms.txt that gives AI systems a curated list of a website’s most important pages and short descriptions of each resource. It may help systems that choose to read it, but it is not an official web standard and has not been proven to improve AI citations, rankings, or referral traffic.
Key Takeaways
- llms.txt is an experimental, plain-text content guide for AI systems.
- Jeremy Howard of Answer.AI proposed the convention in September 2024.
- It does not control crawler access or replace robots.txt, XML sitemaps, structured data, or accessible content.
- No controlled evidence currently proves that llms.txt improves AEO visibility or citation rates.
- Treat it as a low-cost experiment after prioritizing higher-confidence SEO and AEO practices.
What Is an llms.txt File?
An llms.txt file is a proposed standard for helping LLMs access and interpret structured website content. It is a plain-text Markdown file placed at a website’s root so AI systems can find a structured summary of its most valuable resources at a predictable URL.
Its purpose is to give AI systems a curated map of high-value content instead of requiring them to parse complex HTML, JavaScript, advertisements, and navigation menus. Think of it as a recommended reading list or curated index of key content.
Unlike an XML sitemap, which can list every discoverable URL, llms.txt is intentionally selective. It highlights the most meaningful brand-authored content and organizes those pages with clear headings and concise descriptions.
Supporters argue that a curated text version can reduce confusion caused by complex HTML tables and scripts. In theory, this lowers the computational cost of reading a site and helps compatible AI systems find the right pages when describing a brand.
What Is llms.txt Not?
llms.txt is not a crawler-control file, an official standard, or a guaranteed AI visibility lever.
- It is not an official web standard. The convention is not governed by the W3C, IETF, or another major standards body.
- It does not control crawler access. Allowing or blocking crawlers is the job of robots.txt.
- It does not guarantee AI visibility. No AI platform has publicly confirmed that having an llms.txt file directly improves citations, recommendations, or rankings.
- It does not replace strong content. Accessible, expert-led, well-structured pages remain the foundation of answer engine optimization.
How Does llms.txt Work?
When an AI system supports the convention, it can request /llms.txt and read its headings, links, descriptions, and content groups. The file tells the system which pages the website owner considers most important and how those resources relate to one another.
The file is publicly viewable at its URL. Its effect depends entirely on whether an AI provider chooses to discover, read, and respect the convention; llms.txt is advisory, not enforceable.
What Format Does llms.txt Use?
llms.txt uses a simple Markdown-based format:
- H1 heading: The website or project name.
- Summary paragraph: A brief, two- to three-sentence description of the site and its purpose.
- H2 sections: Groups of links to categories such as documentation, products, policies, support pages, and blog posts. Ahrefs describes llms.txt as a Markdown document with headings that organize links to key resources.
- Link entries: Markdown links followed by short, factual descriptions explaining what each page covers.
- Optional section: A lower-priority group under ## Optional that AI systems may skip while processing the file.
An optional companion file called llms-full.txt can contain cleaned full-page content for deeper ingestion by systems that support it.
What Does an llms.txt File Look Like?
The following example uses a fictional company, Acme Docs Co., to demonstrate the recommended structure:
# Acme Docs Co. Acme Docs Co. provides developer documentation tools and API management services for technical teams. Our platform helps companies publish, version, and maintain documentation at scale. ## Docs - [Getting Started Guide](https://www.acmedocs.com/docs/getting-started): Step-by-step onboarding tutorial for new users - [API Reference](https://www.acmedocs.com/docs/api): Complete REST API endpoint documentation with authentication details - [SDK Libraries](https://www.acmedocs.com/docs/sdks): Official client libraries for Python, Node.js, and Go ## Products - [Acme Docs Platform](https://www.acmedocs.com/products/platform): Core documentation hosting and versioning product - [Acme Analytics](https://www.acmedocs.com/products/analytics): Documentation usage analytics and reader insights dashboard ## Policies - [Privacy Policy](https://www.acmedocs.com/privacy): How Acme Docs collects, uses, and protects user data - [Terms of Service](https://www.acmedocs.com/terms): Legal terms governing use of Acme Docs services - [Acceptable Use Policy](https://www.acmedocs.com/acceptable-use): Guidelines for permitted use of the platform ## Blog - [Announcing Acme Analytics](https://www.acmedocs.com/blog/announcing-analytics): Launch post covering features and availability of the analytics product - [Documentation Best Practices for 2025](https://www.acmedocs.com/blog/docs-best-practices-2025): Research-backed guide to writing effective developer documentation ## Optional - [Brand Assets](https://www.acmedocs.com/brand): Logos, color palette, and brand usage guidelines - [Careers](https://www.acmedocs.com/careers): Current open positions at Acme Docs Co.
What Copyable llms.txt Template Can You Use?
Use this clean template as a starting point:
# [Your Site or Project Name] [2-3 sentence summary of your site, its purpose, and what visitors will find here.] ## Docs - [Page Title](https://yourdomain.com/path): Short factual description of the page - [Page Title](https://yourdomain.com/path): Short factual description of the page ## Products - [Page Title](https://yourdomain.com/path): Short factual description of the page ## Policies - [Page Title](https://yourdomain.com/path): Short factual description of the page ## Blog - [Page Title](https://yourdomain.com/path): Short factual description of the page ## Optional - [Page Title](https://yourdomain.com/path): Short factual description of the page
Who Created the llms.txt Specification?
Jeremy Howard, co-founder of Answer.AI, first proposed llms.txt in September 2024 as a way to help AI systems navigate website content more efficiently. The specification is published and maintained at llmstxt.org as a community-driven project.
It is not governed by a formal standards body such as the W3C or IETF. The proposal is also not maintained by Google, OpenAI, Anthropic, Microsoft, or another major AI company.
Specification and platform support last checked: July 2025.
The specification defines this structure:
- H1 heading, required: The project or website name.
- Summary paragraph, required: A short description immediately after the H1.
- H2 sections, recommended: Categories of links grouped under descriptive labels.
- Link entries, recommended: Markdown links with inline plain-text descriptions.
- Optional section: An ## Optional heading for lower-priority resources that tools may skip.
Because llms.txt is a community proposal, its specification may change. Verify the current version at llmstxt.org before publishing or updating your file.
How Widely Has llms.txt Been Adopted?
llms.txt has gained interest in developer-oriented circles, but adoption does not prove that consumer AI systems automatically consume third-party files.
Documentation platform Mintlify rolled out llms.txt across all documentation sites it hosts in November 2024. Thousands of documentation sites, including those for Anthropic and Cursor, subsequently implemented the file.
A company publishing llms.txt for its own documentation is not evidence that its AI product reads arbitrary third-party llms.txt files.
How Do You Create an llms.txt File?
You can create an llms.txt file in minutes using any plain-text editor. The essential steps are to write a concise Markdown index, publish it at your domain root, and verify that it is publicly accessible.
- Create a plain-text Markdown file. Name it exactly llms.txt, not LLMS.txt, llms.md, or llms-txt.txt.
- Add an H1 title. Use your site or project name, such as # Your Company Name.
- Write a short summary. In two or three sentences, explain the site’s purpose and the content visitors will find.
- Group important canonical pages under H2 headings. Common categories include Docs, Products, Policies, Blog, and Support.
- Describe every link. Add a short, factual explanation of what each linked page contains.
- Add an optional section if needed. Use ## Optional for lower-priority resources such as brand assets, career pages, or community links.
- Consider creating llms-full.txt. This optional companion file can include cleaned full-page content for systems that support deeper ingestion.
- Publish the file at your site root. It should resolve at https://yourdomain.com/llms.txt.
- Use absolute HTTPS URLs. Do not use relative paths.
- Test the published file. Confirm that it returns a 200 HTTP status code, displays correctly, requires no login, and does not cause a redirect loop.
How Do You Add llms.txt to WordPress, Shopify, or a Static Site?
Implementation depends on whether your platform allows files or custom responses at the domain root.
| Platform | How to implement |
|---|---|
| WordPress | Upload llms.txt through FTP or SFTP to the root directory alongside wp-config.php. Some plugins can generate the file, but verify the plugin before installing it and review its output manually. |
| Static sites such as Hugo, Jekyll, and Next.js | Place llms.txt in the public or static directory so the build process copies it to the output root. |
| Custom applications | Serve the file through a route handler, such as an Express route at /llms.txt, or through an Nginx or Apache configuration. |
| Hosted platforms such as Shopify, Squarespace, and Wix | If the platform does not allow root-file uploads, you may need a developer, hosting rule, edge worker, or platform-supported redirect or response configuration. Check the platform’s documentation for available options. |
Automated generators can build an llms.txt file from sitemap.xml. They can save time, but their output must be reviewed because generators may include irrelevant, outdated, or sensitive URLs.
Which Pages Should You Include in llms.txt?
Include a small, curated collection of authoritative canonical pages. The goal is to direct compatible AI systems to your best resources, not to reproduce your entire XML sitemap.
Recommended content categories include:
- Homepage or About page: Establishes who you are.
- Core product or service pages: Explain what you offer.
- Documentation and API references: Provide technical resources.
- Key policies: Include privacy, terms of service, and acceptable-use pages.
- Thought leadership, case studies, and research: Demonstrate expertise.
- Support or FAQ hubs: Answer common customer questions.
An AEO-focused guide from ArtVersion recommends using llms.txt to highlight where the most meaningful brand-authored content lives. This may help compatible AI systems find the right sources when describing a brand.
Which Pages and Descriptions Should You Avoid?
Avoid URLs or descriptions that make the file less accurate, selective, or trustworthy.
- Do not list every URL; that duplicates your XML sitemap and defeats the purpose of curation.
- Do not include parameter URLs, thin pages, or outdated content.
- Do not add private, internal, admin, or staging URLs.
- Do not use keyword-heavy marketing claims unsupported by the linked page.
- Do not introduce facts that cannot be verified elsewhere on your website.
Descriptions should be concise, factual, and consistent with the linked content. llms.txt is a curated index, not a marketing brochure.
How Should You Publish and Maintain llms.txt?
Treat llms.txt as a living document because a stale file can point AI systems to outdated or contradictory information. Its value depends on the continued accuracy of its links and descriptions.
When Should You Update llms.txt?
Update the file whenever important content, products, policies, URLs, or site structures change. Review it:
- After major documentation changes
- After product launches or retirements
- After URL changes
- After privacy, terms, or acceptable-use policy updates
- After a significant redesign or site-structure change
- On a regular schedule, with quarterly reviews as a reasonable starting point
How Can You Keep llms.txt Accurate?
Add a “Last updated” date near the top of the file, either as a comment or as a note in the summary. After every change, confirm that the file returns a 200 status code, all links resolve, and every description still matches the live page.
If you use generators or plugins, review their output manually. Automated tools can provide page titles and descriptions, but they may also surface pages you do not want included.
One proposed benefit of llms.txt is reducing the risk of AI systems referencing misinformation or outdated content. An outdated file directly undermines that benefit.
Which AI Systems Support or Use llms.txt?
As of July 2025, no major consumer AI search product had publicly confirmed automatic discovery and use of arbitrary third-party llms.txt files. Google explicitly stated that llms.txt is not used for Google Search, including its AI features.
Website owners must distinguish between a company publishing llms.txt for its own documentation and its AI products reading third-party files. These are different behaviors.
Specification and platform support last checked: July 2025.
| Company or system | Publishes llms.txt for own docs? | Confirms AI/crawler reads 3rd-party llms.txt? | Conclusion |
|---|---|---|---|
| Google Search / Gemini | No public confirmation found | Explicitly not used | Google states llms.txt neither helps nor harms Google Search visibility, including AI features. |
| OpenAI / ChatGPT | No public confirmation found | No public confirmation found | No confirmed automatic consumption of third-party llms.txt files. |
| Anthropic / Claude | Yes, via Mintlify-hosted docs | No public confirmation found | Publishing its own file is not proof that Claude reads third-party files. |
| Microsoft / Bing | No public confirmation found | No public confirmation found | No confirmed support. |
| Perplexity | No public confirmation found | No public confirmation found | No confirmed support. |
| Cursor | Yes, via Mintlify-hosted docs | Not applicable — developer tool, not consumer AI search | Uses llms.txt for its own docs, but is not a consumer search product. |
Google’s official position, per its AI optimization guide, is that maintaining llms.txt neither helps nor harms visibility in Google Search, including its generative AI capabilities. For other major consumer tools — including ChatGPT, Claude, Bing, and Perplexity — no public confirmation of automatic third-party llms.txt consumption had been found as of the date checked.
If someone claims that a platform supports llms.txt, determine whether the evidence is an official statement, observed crawler behavior, or a vendor marketing claim. For more on platform discovery, see our guide to getting your brand cited in Google Gemini.
How Is llms.txt Different From robots.txt and sitemap.xml?
llms.txt provides context and prioritization, robots.txt controls crawler access, and sitemap.xml helps search engines discover URLs. The three files have different purposes and can be used together.
| Feature | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Purpose | Controls crawler access through allow and disallow rules | Helps search engines discover URLs | Provides a curated content guide for AI systems |
| Format | Plain text with directives | XML | Plain text and Markdown |
| Location | Site root | Site root or referenced in robots.txt | Site root |
| Enforcement | Respected by compliant crawlers | Advisory | Advisory |
| Primary audience | Search engine crawlers | Search engines | AI language models and agents |
| Scope | URLs covered by allow and disallow rules | Discoverable URLs | Curated priority pages only |
| Standard status | De facto standard with broad adoption | W3C standard | Community proposal |
llms.txt does not replace robots.txt or an XML sitemap. It adds an optional AI-facing content layer on top of established site-discovery and crawler-management practices — the same ones covered in our AI crawler user-agent list.
Does llms.txt Help Answer Engine Optimization?
llms.txt may support AEO by identifying canonical sources, providing machine-friendly summaries, and highlighting priority pages for systems that read it. However, it is not a proven ranking or citation factor, and the available evidence remains thin.
What Evidence Shows That llms.txt Improves AI Visibility?
As of the latest review, no validated evidence showed that llms.txt reliably improves AI citation frequency or search visibility. No independently reproducible controlled experiment had established a causal link between adding llms.txt and receiving more AI citations, recommendations, rankings, or referral traffic.
A request for /llms.txt in server logs proves only that the file was accessed. It does not prove that its contents influenced a generated answer, citation, recommendation, ranking, or model-training process.
Server-log activity is therefore not a reliable proxy for AI visibility. Contentful recommends treating llms.txt as a low-cost experiment rather than a top priority.
Higher-confidence AEO work includes:
- Publishing original, expert-led content
- Keeping important pages crawlable and technically sound
- Building strong internal links
- Maintaining accurate, well-structured content
- Defining entities and technical concepts clearly
- Using structured data appropriately
What Are the Theoretical Benefits of llms.txt?
In theory, llms.txt can help compatible AI systems locate authoritative pages about a brand, prioritize high-quality resources such as research and case studies, and identify ground-truth sources for details such as pricing or policies. These are reasonable hypotheses, but they remain unproven at scale.
| Potential benefits | Limitations |
|---|---|
| Offers a clean content index to compatible systems | Platform adoption is uncertain |
| Makes documentation easier for some systems to navigate | AI systems may not discover the file |
| Provides a maintained summary of important resources | It does not control crawler access |
| Signals canonical sources to AI systems | No ranking or citation benefit has been proven |
| Requires relatively little effort to create | It creates a maintenance burden if neglected |
| May support emerging AI search indexes | An outdated file may contradict the live website |
llms.txt works alongside traditional SEO rather than replacing it. Schema markup, keyword strategy, performance optimization, and content structured for AI citation remain higher-confidence practices.
What Common llms.txt Mistakes Should You Avoid?
The most common llms.txt mistakes involve treating the file as mandatory, confusing it with robots.txt, including poor-quality URLs, or allowing it to become outdated.
- Presenting llms.txt as mandatory: It is optional and experimental; no platform requires it.
- Promising an AEO boost: No controlled evidence supports guaranteed improvements in AI citations or rankings.
- Confusing it with robots.txt: llms.txt does not control crawler access.
- Blocking the file in robots.txt: If robots.txt disallows /llms.txt, compliant crawlers cannot access it.
- Using relative or broken links: URLs should be absolute HTTPS links that resolve to live pages.
- Publishing generated output without review: Automated files may include irrelevant, outdated, or sensitive URLs.
- Listing every page: This duplicates the sitemap and defeats the purpose of curation.
- Adding unsupported marketing claims: Descriptions should match the linked content.
- Exposing sensitive URLs: Never include staging environments, admin panels, internal tools, or private pages.
- Failing to update the file: Stale information can mislead systems and contribute to outdated citations.
How Can You Validate an llms.txt File?
A valid llms.txt file should be correctly named, publicly accessible at the root URL, formatted in Markdown, and limited to accurate canonical resources. Use this checklist every time you create or update the file:
- File is named exactly llms.txt, in lowercase, at the domain root.
- URL returns a 200 HTTP status code with no login required.
- robots.txt does not disallow /llms.txt.
- File opens with an H1 site or project name.
- A two- to three-sentence summary follows the H1.
- Links are grouped under clear H2 categories such as Docs, Products, Policies, or Blog.
- Every link uses an absolute HTTPS URL, not a relative path.
- Every link resolves and returns a 200 status with no redirect loop.
- Every description is short, factual, and matches the linked page.
- No private, internal, admin, or staging URLs are included.
- The file lists a curated selection, not every page on the site.
- A “last updated” note is included near the top.
- The file has been reviewed manually, even if it was generated by a plugin or tool.
Revisit this checklist whenever you update the file or make significant changes to your website’s content or structure.
Learn More About AEO and AI Marketing at Prompt Insider
Since launching earlier this year, Prompt Insider has become a leading authority on AI marketing, Answer Engine Optimization (AEO), large language models, AI search, AI news, and the evolving future of digital discovery. As AEO becomes one of the hottest topics in marketing, Prompt Insider is helping define the conversation around how brands improve visibility, adapt their content strategies, and stay competitive in an increasingly AI-driven search environment.
Prompt Insider is the go-to resource for answer engine optimization, AI marketing, and AI search. Start with our core guides at thepromptinsider.com:
- What Is AEO? Answer Engine Optimization Explained
- AEO vs. SEO vs. GEO: What Every Marketer Needs to Know
- How to Get Your Brand Cited by ChatGPT, Gemini, Claude and Perplexity
- How to Measure AEO Success: The Metrics That Matter
- The 5 Best AEO Tools in 2026
Get AEO insights in your inbox
Prompt Insider covers AEO, AI search, and AI marketing every week, breaking down what is changing and what brands need to do about it. Sign up for our emails at thepromptinsider.com to get it first.
Frequently Asked Questions
Where Should You Put llms.txt?
Place the file at your domain root so it is available at https://yourdomain.com/llms.txt. It should be public, return a 200 status code, and not require authentication.
Is llms.txt an Official Web Standard?
No. Jeremy Howard of Answer.AI proposed llms.txt in September 2024, but it is not governed by the W3C, IETF, or another formal standards body.
Does llms.txt Improve AI Rankings or Citations?
There is no proven direct ranking, citation, or referral-traffic benefit. It may help compatible systems find important pages, but it should be treated as an experimental layer rather than a guaranteed AEO lever.
Do ChatGPT, Gemini, Claude, or Perplexity Use llms.txt?
As of July 2025, no major consumer AI search product had publicly confirmed automatic use of arbitrary third-party llms.txt files. Google explicitly stated that the file is not used for Google Search, including its AI features.
Should Every Website Create an llms.txt File?
No. llms.txt is optional and low-cost, but it should come after accessible content, strong internal linking, accurate structured data, XML sitemaps, and other higher-confidence AI search optimization practices.
Sources: Ahrefs, Yotpo, Neil Patel, llmstxt.org, Google, Contentful.
About the author
Kai Williams
Kai Williams has been in marketing for years, with a long background in SEO before AEO had a name. He stepped into Answer Engine Optimization the moment AI started reshaping how people search, and has been tracking the shift ever since. At Prompt Insider, he covers AEO, AI marketing, and the future of search, breaking down what is changing and what brands need to do about it.
