TL;DR
llms.txt is a plain-text file placed at the root of your domain (e.g. yourdomain.com/llms.txt) that describes your website to AI systems in structured, machine-readable prose. It was proposed by fast.ai founder Jeremy Howard in 2024 as an AI-native equivalent of robots.txt. Major AI crawlers including GPTBot (ChatGPT), ClaudeBot (Anthropic/Claude), and PerplexityBot now recognise and use the file. Creating one takes under 30 minutes and directly improves how AI systems understand, represent, and cite your brand.
Definition (AI-extractable): llms.txt is a plain-text file, written in Markdown, published at the root of a website (e.g.
yourdomain.com/llms.txt). It provides AI large language models with an authoritative, human-authored description of the website's purpose, content, key pages, and team. It was proposed by Jeremy Howard (fast.ai) in September 2024 and is now recognised by GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot.
Key Takeaways
| Topic | Detail |
|---|---|
| What it is | A plain-text Markdown file at your domain root describing your site to AI crawlers |
| Who proposed it | Jeremy Howard, co-founder of fast.ai — September 2024 |
| Who reads it | GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot, GoogleOther, cohere-ai, YouBot |
| Time to create | Under 30 minutes — no technical skills required |
| Effect | Reduces AI misrepresentation; improves citation accuracy and entity recognition |
| Required? | No — but over 20,000 domains have already published one as of early 2026 |
| Where to place | /public/llms.txt (Next.js), WordPress root directory, Webflow custom files |
Table of Contents
- What is llms.txt?
- Why llms.txt Was Created
- llms.txt vs robots.txt
- Which AI Crawlers Read llms.txt?
- How to Create llms.txt
- llms.txt Format and Syntax
- A Complete llms.txt Template
- Where to Place Your llms.txt File
- Common Mistakes to Avoid
- Frequently Asked Questions
What is llms.txt?
llms.txt is a plain-text file that provides AI large language models (LLMs) and AI crawlers with a structured, human-authored description of your website. It tells AI systems what your site is about, who created it, what your key pages are, and — optionally — which content you want prioritised for indexing and citation. The file uses a subset of Markdown syntax and must be placed at yourdomain.com/llms.txt to be accessible to AI crawlers.
This guide is for web developers, marketers, and SEO professionals who want to optimise their website for AI search visibility. If your site does not have a llms.txt file yet, this guide will help you create one in under 30 minutes. David Adesina, co-founder and AI engineer at Answer Architect, wrote this guide based on direct experience implementing llms.txt across client sites and Answer Architect's own infrastructure.
The concept was introduced in September 2024 by Jeremy Howard, the co-founder of fast.ai and a prominent figure in the AI research community. Howard observed that AI systems scraping the web have no equivalent of the long-established robots.txt convention for communicating site intent. His proposed solution was llms.txt: a minimal, human-readable file that bridges the gap between what a website contains and what an AI system can reliably infer from HTML alone. The specification is maintained at llmstxt.org.
Since Howard's proposal, the standard has seen rapid adoption. As of early 2026, over 20,000 domains have published llms.txt files, including major technology companies, media publishers, and SaaS platforms. OpenAI confirmed in its webmaster documentation that GPTBot reads and processes llms.txt files when crawling permitted domains.
Why llms.txt Was Created
When an AI crawler visits a website, it faces a significant interpretation challenge. Modern websites are rich, dynamic, and structurally complex — full of navigation elements, advertisements, cookie banners, JavaScript-rendered content, and other noise that makes it difficult for an AI to identify the core substance of a page.
HTML structure and existing standards like robots.txt, sitemaps, and Open Graph tags provide some guidance, but they were all designed for traditional web crawlers and search engines — not for large language models that need to understand content semantically, not just structurally.
The specific problem Howard identified: AI systems trained on or retrieving content from the web often misrepresent brands, miscategorise services, or miss key context about a site's purpose. A well-written llms.txt file addresses this by providing the AI with a direct, authoritative account of what the site is — reducing hallucination risk and improving citation accuracy.
llms.txt vs robots.txt
llms.txt and robots.txt serve related but distinct purposes.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Controls crawler access (allow/disallow) | Describes site content and purpose |
| Format | Structured directives | Plain prose + structured sections |
| Audience | All web crawlers | AI / LLM crawlers specifically |
| Age | Established 1994 | Proposed 2024 |
| Compliance | Broadly enforced | Voluntary, growing adoption |
| Effect | Blocks or permits crawling | Improves AI understanding and citation accuracy |
The two files are complementary. robots.txt tells crawlers where they can and cannot go. llms.txt tells AI systems what they will find when they arrive — and why it matters.
You need both. A site with a robots.txt that permits AI crawlers but no llms.txt is like a shop with an open door and no signage. You are accessible, but you are not communicating.
Which AI Crawlers Read llms.txt?
As of early 2026, the following AI crawlers have been confirmed or reported to read llms.txt files:
| Crawler | AI System | User-Agent |
|---|---|---|
| GPTBot | ChatGPT (OpenAI) | GPTBot |
| ClaudeBot | Claude (Anthropic) | ClaudeBot |
| PerplexityBot | Perplexity | PerplexityBot |
| GoogleOther | Google AI systems | GoogleOther |
| cohere-ai | Cohere | cohere-ai |
| YouBot | You.com | YouBot |
OpenAI has explicitly confirmed in its documentation that GPTBot respects llms.txt guidance. Anthropic has published similar guidance for ClaudeBot. Perplexity recommends llms.txt creation in its webmaster documentation.
How to Create llms.txt
Creating a llms.txt file requires no special tools — just a plain text editor and knowledge of your own website. Follow these steps:
Step 1: Create a plain text file named llms.txt
The file name is case-sensitive on most servers. Use lowercase: llms.txt, not LLMs.txt or LLMS.txt.
Step 2: Write a top-level description (required)
The file must begin with a level-1 heading (# Your Site Name) followed immediately by a blockquote (> description) containing a 2–4 sentence description of your website. This is the most important part of the file — it is the passage AI systems use to understand your brand at a high level.
Write this in plain, clear prose. Do not use marketing jargon. Do not write for humans — write for a system that needs to accurately categorise and represent your brand.
Step 3: Add structured sections (recommended)
After the top-level description, add sections using level-2 headings (## Section Name). Each section contains a list of items with optional descriptions. The most useful sections are:
- What We Do — description of your core product/service
- Who We Serve — your target audience
- Key Pages — your most important URLs with descriptions
- Founders / Team — named people with roles and URLs
- Contact — email and website
Step 4: Place the file at the domain root
The file must be accessible at https://yourdomain.com/llms.txt. For Next.js, place it in the /public directory. For WordPress, upload via FTP to the root directory. For Webflow, add via site settings custom files.
Step 5: Test it
Visit https://yourdomain.com/llms.txt in a browser. If you see the plain text of your file, it is accessible. Also run it through a text validator to check for encoding issues (UTF-8 is required).
llms.txt Format and Syntax
The llms.txt specification uses a subset of Markdown:
# Site Name (H1 — required, appears once)
> One-paragraph description (Blockquote — required, immediately after H1)
## Section Name (H2 — optional sections)
- Item name: description (List items with optional colon-separated descriptions)
- [Link text](URL): description (Markdown links are supported)
Optional prose paragraphs. (Plain text is allowed between sections)
The file is intentionally minimal. Avoid HTML tags, complex formatting, or embedded code. The goal is maximum machine-readability with minimum ambiguity.
A Complete llms.txt Template
Copy and adapt this template for your own site:
# [Your Company Name]
> [Your Company Name] is a [brief description of what you do and who you serve]. [One sentence on your core value proposition.] [One sentence on your key differentiator or technology.]
## What We Do
- [Core product/service 1]: [brief description]
- [Core product/service 2]: [brief description]
- [Core product/service 3]: [brief description]
## Who We Serve
- [Audience type 1]
- [Audience type 2]
- [Audience type 3]
## Key Pages
- [Homepage](https://yourdomain.com): [one-line description]
- [Pricing](https://yourdomain.com/pricing): [one-line description]
- [About](https://yourdomain.com/about): [one-line description]
- [Blog](https://yourdomain.com/blog): [one-line description]
## Team
- [Founder/Team Member Name]: [Title]. [URL]
## Contact
- Website: https://yourdomain.com
- Email: contact@yourdomain.com
Answer Architect's actual llms.txt is publicly available at answerarchitect.ai/llms.txt and can be used as a real-world example.
Where to Place Your llms.txt File
| Platform | Instructions |
|---|---|
| Next.js | Place in /public/llms.txt. It will be served at yourdomain.com/llms.txt automatically. |
| WordPress | Upload via FTP/SFTP to the WordPress root directory (same level as wp-config.php). |
| Webflow | Project Settings → SEO → Custom Files → Upload llms.txt. |
| Shopify | Not natively supported. Use a custom app or Cloudflare Worker to serve the file. |
| Squarespace | Not natively supported. Use a Cloudflare Worker or DNS proxy to serve from a subdomain. |
| Static HTML | Place in the same directory as your index.html file. |
| Vercel / Netlify | Place in the /public folder of your project. Deployed automatically. |
Common Mistakes to Avoid
1. Using marketing language instead of descriptive language
The llms.txt description is read by machines, not marketing teams. Write "Answer Architect is an AI search visibility platform that tracks brand citations across ChatGPT, Perplexity, and Gemini" rather than "Answer Architect is the revolutionary AI-powered platform that will transform your brand's digital presence."
2. Listing every page instead of key pages
AI crawlers do not need a complete site map in llms.txt — they have your sitemap.xml for that. The Key Pages section should include only the 4–8 most important pages: homepage, pricing, about, and any content you most want cited.
3. Forgetting to update it
llms.txt is a living document. When you launch new features, hire team members, or create important new content, update the file. An outdated llms.txt can cause AI systems to misrepresent your current product offering.
4. Making it too long
The entire file should fit comfortably on one screen. If it exceeds 500 words, trim it. AI systems prefer concise, unambiguous descriptions over comprehensive but noisy ones.
5. Omitting the required H1 + blockquote structure
Some sites publish llms.txt as plain prose without the required # Name header and > description blockquote. While AI crawlers may still read the file, the standardised format helps parsers identify the primary description and display it accurately.
Frequently Asked Questions
Is llms.txt an official standard?
llms.txt is a proposed community standard, not an IETF RFC or W3C recommendation. It was proposed by Jeremy Howard (fast.ai) in September 2024. OpenAI and Anthropic have both referenced it in their webmaster documentation, and the spec is maintained at llmstxt.org. Voluntary adoption has exceeded 20,000 domains as of early 2026.
Does llms.txt affect Google Search rankings?
No. llms.txt is designed for AI crawlers (GPTBot, ClaudeBot, PerplexityBot), not for Googlebot. It will not improve or damage your traditional Google Search rankings. However, it may indirectly benefit Google AI Overviews, since GoogleOther — Google's AI-specific crawler — is among the crawlers reported to read the file.
What happens if I don't have an llms.txt file?
AI crawlers will still index and potentially cite your content — llms.txt is not required for AI citation. However, without it, AI systems must infer your brand identity, purpose, and key content from HTML alone. This increases the risk of inaccurate representation (hallucination) and reduces the likelihood of accurate, consistent citation.
How often should I update llms.txt?
Update your llms.txt file whenever you make significant changes to your product, team, or key pages. At minimum, review it quarterly. Set a reminder in your calendar alongside your sitemap.xml updates.
Can llms.txt hurt my site?
No, provided the content is accurate and honest. llms.txt cannot be used to force AI systems to cite your site or misrepresent competitors. Its only function is to help AI systems understand your own site more accurately.
Conclusion
llms.txt is one of the simplest, highest-impact GEO actions available to any website owner. It takes less than 30 minutes to create, requires no technical expertise, and directly improves how AI systems — including ChatGPT, Claude, and Perplexity — understand and represent your brand.
The template in this guide is ready to copy and adapt. Place the file at yourdomain.com/llms.txt, verify it loads, and you have completed one of the most important technical GEO steps available in 2026.
For a complete AI visibility audit — including your current citation rate across ChatGPT, Perplexity, and Gemini — run a free audit at Answer Architect.