llms.txt explained: what it is, why it matters, and how to set it up
llms.txt is a simple text file you place in your website's root directory that tells AI assistants (ChatGPT, Claude, Gemini, Perplexity) which content you want them to read and cite. It's not mandatory, but without it, AI crawlers may miss your best pages or cite competitors instead—and you have no way to know.
What is llms.txt and why does it matter?
llms.txt is a plain-text file that lives at the root of your domain (e.g., example.com/llms.txt) and serves as a manifest for AI language models. It tells crawlers from OpenAI, Anthropic, Google, and Perplexity which pages on your site are authoritative, which content is off-limits, and where to find your best answers. Think of it as a more granular version of robots.txt, but designed specifically for AI assistants rather than traditional search engines. Traditional SEO optimizes for Google's crawler. AI optimization (AEO/GEO) optimizes for AI assistants—and the two are not the same. Google's crawler indexes pages for ranking; AI crawlers are looking for source material to cite in chat responses, answer cards, and AI-generated summaries. If your llms.txt is missing or poorly configured, AI assistants may skip your content entirely, cite your competitors instead, or include outdated information.
How AI crawlers use llms.txt vs. robots.txt and sitemaps
llms.txt is NOT a replacement for robots.txt or XML sitemaps. All three serve different purposes: **robots.txt** tells search engines (Google, Bing) and other crawlers what to index and what to ignore. It's been the standard for 25+ years and remains mandatory for traditional SEO. **XML sitemaps** list every page on your site and signal their importance to search engines. They speed up indexing and help Google understand site structure. **llms.txt** is newer and purpose-built for AI crawlers. It tells them: "Here's the content I want you to read first, here's what's off-limits, and here's where my canonical answers live." It's a hint layer—AI crawlers will still follow your robots.txt rules, but llms.txt lets you be explicit about priority and context. AI crawlers like OpenAI's crawler, Anthropic's Claude crawler, and Perplexity's bot will check for llms.txt. If it exists, they read it and adjust their crawl strategy. If it doesn't exist, they fall back to robots.txt and sitemaps. Either way, they crawl—but without llms.txt, you're not steering them toward your best content.
| File | Purpose | Audience | Required? | Content type |
|---|---|---|---|---|
| robots.txt | Tell crawlers what to index / ignore | All crawlers (Google, Bing, AI bots) | Yes (for SEO) | Allow/disallow rules |
| Sitemap.xml | List all pages and signal priority | Search engines + AI crawlers | Recommended | XML with URLs and metadata |
| llms.txt | Prioritize content for AI citation | AI crawlers only (ChatGPT, Claude, Gemini, Perplexity) | No, but recommended | Plain text with URLs, policies, and context |
What goes into an llms.txt file: structure and best practices
llms.txt follows a simple, human-readable format. There's no official spec yet (it's still emerging), but the de facto standard includes: 1. **A header comment** explaining the file's purpose. 2. **A policy section** stating what you allow AI crawlers to do (crawl, cite, train on, etc.). 3. **A content section** listing the URLs you want AI assistants to prioritize. 4. **Optional metadata** like your business name, content categories, and contact info. Here's the anatomy of a functional llms.txt: ``` # llms.txt for [Your Business Name] # Last updated: [Date] Policy: - Allow: crawl, cite, answer-generation - Disallow: training-data - Prefer: answer-first content Canonical URLs: - https://yoursite.com/about - https://yoursite.com/services - https://yoursite.com/blog/how-to-guides Business Info: Name: [Your Business] Category: [Industry] Contact: [email] ``` The key sections: **Policy** tells crawlers what they can and cannot do. Most sites allow crawling and citation (that's the goal) but may disallow training-data use if they're concerned about their content being used to train AI models. You can be permissive ('allow all') or restrictive ('disallow training'). **Canonical URLs** are the pages you want AI assistants to prioritize. These should be your best, most authoritative pages—how-to guides, product pages, your about section, core service descriptions. Limit to 10–20 high-value URLs; too many dilutes the signal. **Business Info** is optional but helpful. It gives crawlers context about who you are and what you do, which can improve citation accuracy.
Step-by-step: how to create and deploy llms.txt
- Create a plain-text file named llms.txt
Open a text editor (Notepad, VS Code, or your CMS's code editor). Do NOT use Word or a rich-text editor—it will add formatting and break the file. Create a new file and save it as 'llms.txt' (no extension after .txt).
Why: AI crawlers expect plain text. Formatted files will cause parsing errors and the crawler will ignore it.
✓ Checkpoint: You have a file on your computer named 'llms.txt' with zero formatting (confirm by opening it in a plain-text editor and seeing only raw text).⚠ Pitfall: Most people accidentally save it as llms.txt.txt or use a rich-text editor, which adds invisible formatting. Always double-check the filename and use a plain-text editor. - Write your policy and canonical URLs
Copy the template below into your llms.txt file and customize it with your business name, canonical URLs, and policy. Start with 'allow: crawl, cite' unless you have a specific reason to restrict training-data use. Template: ``` # llms.txt for [Your Business Name] # Last updated: [Today's Date] Policy: - Allow: crawl, cite, answer-generation - Disallow: training-data - Prefer: answer-first content, guides, reference material Canonical URLs: - https://yoursite.com/about - https://yoursite.com/services/core-offering - https://yoursite.com/blog/how-to-setup-guide Business Info: Name: [Your Business Name] Category: [Your Industry] Contact: [your email] ```
Why: This structure is recognized by major AI crawlers. The policy section tells them what you allow; canonical URLs guide them to your best content.
✓ Checkpoint: Your llms.txt file contains a policy section, at least 5–10 canonical URLs, and basic business info. No syntax errors (lines are clean, no stray characters).⚠ Pitfall: People often list too many URLs (50+), which dilutes the signal. Stick to 10–20 of your absolute best pages. Also avoid typos in URLs—crawlers will skip broken links. - Upload llms.txt to your website root
Use FTP, SFTP, or your hosting/CMS dashboard to upload the file to the root directory of your domain. For most sites, this is the same directory where your index.html or index.php lives. The file must be accessible at https://yoursite.com/llms.txt (or http://yoursite.com/llms.txt if you're not HTTPS—though HTTPS is strongly recommended).
Why: AI crawlers look for the file at the root. If it's in a subdirectory (e.g., /blog/llms.txt), crawlers won't find it.
✓ Checkpoint: Open a browser and navigate to https://yoursite.com/llms.txt. You should see the raw text file displayed in the browser (not a 404 error). If you see your file content, deployment is successful.⚠ Pitfall: Uploading to the wrong directory (e.g., /public_html/blog instead of /public_html) is the most common mistake. Verify the root directory with your hosting provider if unsure. Also, don't password-protect the file—crawlers can't authenticate. - Verify the file is publicly accessible
Use a tool like curl (command line: curl https://yoursite.com/llms.txt) or an online HTTP checker to confirm the file returns a 200 status code and the content is visible. Alternatively, open it in an incognito browser window to simulate how a crawler sees it.
Why: Crawlers won't use a file they can't access. A 404, 403, or redirect will cause the file to be ignored.
✓ Checkpoint: The HTTP response is 200 OK. The file content is fully visible and readable (no authentication required, no redirects).⚠ Pitfall: Files behind a login wall or redirected through a CDN misconfiguration will return errors. Test from outside your network to confirm public access. - Submit your site to AI crawler services (optional but recommended)
Visit OpenAI's crawler documentation, Anthropic's crawler page, Google's crawler info, and Perplexity's bot page. Some allow you to submit your domain or notify them of new content. You can also add a reference to your llms.txt in your robots.txt file with a comment like: '# See llms.txt for AI crawler policies.' This signals to crawlers that you have an llms.txt and they should check it.
Why: Proactive submission speeds up discovery. Some crawlers check robots.txt for hints about llms.txt.
✓ Checkpoint: You've visited at least one major crawler's documentation and noted their submission process. (Not all require submission; many crawl automatically.)⚠ Pitfall: Waiting passively for crawlers to find your file. Proactive notification, especially if you have a new site or niche content, can accelerate discovery by weeks. - Monitor citations and update llms.txt quarterly
Track where your content appears in AI assistant responses. Tools like Zaduky show you how often ChatGPT, Perplexity, Gemini, and Claude cite your site vs. competitors. Every 3 months, review your canonical URLs in llms.txt. If new high-value pages have launched, add them. If old pages are no longer relevant, remove them. Update the 'Last updated' date at the top of the file.
Why: AI crawlers respect signals that you're actively maintaining your content. Stale or irrelevant URLs waste crawler bandwidth and dilute your signal.
✓ Checkpoint: You've set a calendar reminder to review llms.txt in 3 months. You have a baseline understanding of which of your pages are being cited by AI assistants.⚠ Pitfall: Set-and-forget approach. llms.txt is not a one-time task. As your content evolves, your llms.txt should too. Outdated URLs and missing new content reduce effectiveness.
Common llms.txt policies and what they mean
Your policy section is where you set boundaries. Here are the most common policies and when to use them: **Allow: crawl, cite, answer-generation** This is the most permissive policy. It tells AI crawlers they can read your content, cite it in responses, and use it to generate answers. Use this if you want maximum visibility in AI assistants. It's the recommended default for most businesses. **Allow: crawl, cite; Disallow: training-data** This allows crawlers to read and cite your content but forbids using it to train AI models. Use this if you're concerned about your proprietary content being absorbed into AI model training. Note: this is a polite request, not a legal guarantee. Crawlers that respect llms.txt will honor it; crawlers that don't may ignore it. **Allow: crawl; Disallow: cite, training-data** This is very restrictive. It tells crawlers they can see your content but shouldn't cite it or use it for training. Use this only if you have a specific reason to keep your content out of AI responses (e.g., you're testing private content). This is rarely recommended. **Allow: crawl (specific URLs only)** You can be granular by listing only certain URLs in your canonical section and implying that others should not be crawled for AI purposes. This is useful if some of your content is sensitive (e.g., internal documentation) and you want to steer crawlers away from it. Most businesses should start with 'Allow: crawl, cite, answer-generation' and only add restrictions if there's a compelling reason.
Troubleshooting: why AI crawlers may not be citing your content
llms.txt is a signal, not a guarantee. AI crawlers may not have crawled your site yet, or your content may not rank high enough in the crawler's relevance model. Check: (1) Is llms.txt accessible at yoursite.com/llms.txt? (2) Are your canonical URLs correct and live? (3) Is your content answer-first and authoritative? (4) Have you been indexed by Google? AI crawlers often use Google's index as a starting point. If you're new or low-authority, it may take weeks or months for citations to appear. Use a tool like Zaduky to track citations over time.
How to track whether AI crawlers are citing your content
Publishing llms.txt is half the battle. The other half is measuring whether it's working—i.e., are AI assistants actually citing your content? Traditional SEO has Google Search Console to track rankings and clicks. AI citation tracking is newer and less standardized, but tools are emerging. Here's what to track: **Manual tracking**: Ask ChatGPT, Claude, Gemini, and Perplexity questions in your industry and note whether your site is cited. For example, if you're a SaaS company, ask 'How do I set up [your product]?' and see if your help docs appear. This is labor-intensive but gives you a quick sense of visibility. **Automated tracking platforms**: Tools like Zaduky monitor how often ChatGPT, Perplexity, Gemini, and Claude cite your site vs. competitors. They track citations over time, show which of your pages are cited most, and alert you when a competitor starts outranking you in AI responses. This is the most reliable way to measure ROI. **Traffic attribution**: Check your analytics for traffic from AI crawlers. Look for user agents like 'GPTBot', 'Claude-Web', 'Perplexity Bot', and 'Googlebot-Extended'. If you see increased traffic from these bots after publishing llms.txt, it's a good sign crawlers are engaging with your content. The goal is to move from 'Is anyone citing us?' to 'How often are we cited, and how does that compare to competitors?'
llms.txt as part of a broader AI optimization strategy
llms.txt is a necessary but not sufficient condition for AI visibility. Think of it as one lever in a broader AI optimization (AEO/GEO) strategy. Here's what else matters: **Answer-first content structure**: Write pages that answer the user's question in the first 1–3 sentences, then elaborate. AI crawlers prioritize pages that get to the point fast. A 5,000-word guide that buries the answer in the third section will rank lower in AI responses than a 2,000-word guide that answers immediately. **Structured schema markup**: Use JSON-LD schema (Organization, Article, FAQPage, HowTo) to help crawlers understand your content's structure and intent. This is especially important for how-to guides, FAQs, and product pages. **Authority and topical depth**: AI crawlers weight content from authoritative sources. If you're new or low-authority, llms.txt alone won't make you citable. Build authority through: (a) publishing consistently in your niche, (b) earning backlinks, (c) getting cited by other authoritative sources, (d) being mentioned in industry publications. **Freshness**: Keep your content updated. If an AI crawler visits a page that hasn't been updated in 2 years, it may deprioritize it in favor of fresher sources. Review and update your canonical URLs every quarter. **Mobile-friendly and fast-loading pages**: AI crawlers, like Google's, prefer fast, mobile-friendly pages. Slow or broken pages may not be crawled fully or may be deprioritized. llms.txt is the signal that says 'Here's my best content.' But the content itself has to earn the citation through quality, authority, and clarity.
Quick setup checklist for llms.txt
0/12 complete
Next steps: from llms.txt to AI-ready content
You've published llms.txt. Now make sure your content deserves to be cited. **Immediate (this week)**: Review your canonical URLs in llms.txt. Are they truly your best, most authoritative pages? If you have blog posts or guides that are thin, outdated, or don't answer a clear question, remove them from the list. Crawlers should see only your strongest content. **Short term (this month)**: Audit the pages listed in your llms.txt for answer-first structure. Does the first paragraph answer the user's question? If not, rewrite the opening. AI crawlers and AI assistants both favor pages that get to the point immediately. **Medium term (this quarter)**: Track citations. Use a tool like Zaduky or manual testing (ask ChatGPT and Perplexity questions in your industry) to see if your pages are being cited. If you're not seeing citations after 4–8 weeks, it may mean: (a) your content isn't authoritative enough yet, (b) crawlers haven't visited, or (c) your content doesn't rank high enough in the crawler's relevance model. If that's the case, focus on building authority (backlinks, mentions, consistent publishing) and improving content quality. **Ongoing**: Update llms.txt every quarter. Add new high-value pages, remove outdated ones, and keep your content fresh. AI crawlers respect sites that actively maintain their content. llms.txt is the door. Your content is what's behind it. Make sure both are worth the visit.