What AI gets wrong about your site, and why it’s not your fault: meet llms.txt  • Yoast

AI instruments are in every single place — from chatbots that reply buyer inquiries to language fashions that summarize all the pieces from documentation to authorized textual content. However in the event you’ve ever requested a mannequin like ChatGPT to clarify your website, your product, or your API, the outcomes won’t really feel fairly proper. The truth is, generally they’re approach off. And no, that’s not your fault. 

The disconnect between web sites and LLMs 

Massive language fashions (LLMs) like ChatGPT, Claude, or Gemini are skilled to know a variety of content material. However after they attempt to interpret your web site at runtime, that’s, when somebody is actively asking them a query, they run into a number of core issues: 

• HTML is noisy. Navigation bars, cookie banners, modal popups, and analytics scripts litter the web page. 
• Context home windows are restricted. Most web sites are too massive for an LLM to course of abruptly. 
• Necessary particulars are unfold throughout a number of pages or hidden in tables, code blocks, or feedback. 
• Markdown docs could exist, however the mannequin usually can’t find them, and even know they exist. 

So, while you ask an AI device to “clarify what this firm does” or “summarize this library API”, it usually will get caught. It both skips key context or grabs the fallacious indicators from cluttered markup. 

It’s not dangerous intent; it’s a design limitation. 

Why it’s not your website positioning’s fault, both 

You’ve most likely invested effort and time into SEO. Perhaps your robots.txt and sitemap.xml are in place. You’ve received meta tags, structured knowledge, and clear inside hyperlinks. Good, however LLMs don’t all the time work like Google. 

Conventional website positioning helps your website get discovered. Nevertheless, it doesn’t assure that AI instruments will perceive what a human consumer would. That’s the place a brand new proposal is available in. 

Meet llms.txt: A easy approach to assist AI perceive your website 

A rising variety of builders and AI researchers are adopting a light-weight, human-readable normal referred to as llms.txt.  

What’s llms.txt? 

llms.txt is a plain Markdown file positioned on the root of your website that gives language fashions with a abstract of your mission and direct hyperlinks to scrub, LLM-readable variations of necessary pages. It’s designed for inference-time use, serving to AI instruments shortly perceive a website’s construction, objective, and content material with out counting on cluttered HTML or metadata supposed for search engines like google and yahoo. 

What it does: 

• Offers a brief abstract of your website or mission 
• Hyperlinks to scrub, LLM-ready Markdown variations of key pages 
• Helps AI instruments discover precisely what issues, with out parsing messy HTML

Is it broadly supported? Not but 

Proper now, no main LLM supplier formally helps llms.txt. Instruments like GPTBot (OpenAI), Claude (Anthropic), and Google’s AI crawlers don’t reference or observe it as a part of their crawling conduct. Some firms like Anthropic publish llms.txt recordsdata themselves, however there’s no proof that any crawler is actively utilizing them in retrieval or coaching. 

Nonetheless, it’s a low-effort, no-risk addition that helps put together your website for a future the place structured LLM entry turns into extra standardized. And LLM-facing instruments, and even your personal AI brokers, could make use of it in the present day. 

Instance use instances: 

• A dev library hyperlinks to .md-formatted API docs and utilization examples. 
• A college website highlights course descriptions and educational insurance policies. 
• A private weblog affords a simplified timeline of key tasks or matters. 

You management the content material and the construction. LLMs profit from curated, LLM-aware context. And customers asking questions on your website get higher solutions. 

Utilizing our Yoast website positioning plugin? 

When you’re already utilizing our Yoast website positioning (free or Premium) plugin, generating a llms.txt file is easy. Simply allow the function in your settings, and the plugin will robotically create and serve an entire llms.txt file to your website. You’ll be able to view it anytime at yourdomain.com/llms.txt. 

An LLM-friendly internet isn’t the identical as a Google-friendly internet 

This doesn’t exchange website positioning. Consider llms.txt as a companion to robots.txt. It tells AI bots: “Right here’s the great things. Skip the noise.” 

Sitemaps assist crawlers discover all the pieces. llms.txt tells LLMs what to focus on. 

It’s particularly helpful for: 

• Builders and open-source maintainers 
• Product entrepreneurs trying to cut back help load 
• Groups that need chatbots to drag solutions from docs, not guess 

You don’t want a brand new CMS or tech stack 

All this requires is creating two issues: 

1. A primary llms.txt file in Markdown
2. Ideally, you’d even have Markdown variations (.html.md) of key pages included alongside the originals, with the identical URL plus .md added. 

No new instruments, plugins, or frameworks wanted, though some ecosystems are already including help. 

Right here’s an instance of a file robotically constructed by Yoast website positioning, because it has an llms.txt generator in-built:

Generated by Yoast website positioning v25.3, that is an llms.txt file, meant for consumption by LLMs. That is the [sitemap](https://everydayimtravelling.com/sitemap_index.xml) of this web site. 
 
# everydayimtravelling.com: Tales from our travels 
 
## Posts 
- [Test video](https://everydayimtravelling.com/test-video/) 
- [A Journey Through Portugal’s Wine Country: A Suggested Wine Tour Route](https://everydayimtravelling.com/a-wine-tour-through-portugal/) 
- [Travel essentials for backpackers FAQ](https://everydayimtravelling.com/travel-essentials-for-backpackers-faq/) 
 
## Pages 
- [Checkout](https://everydayimtravelling.com/checkout/) 
- [Contact us](https://everydayimtravelling.com/contact-us/) 
- [How we started this blog](https://everydayimtravelling.com/pagina-harry-potter/) 
- [My account](https://everydayimtravelling.com/my-account/) 
- [Cart](https://everydayimtravelling.com/cart/) 
 
## Classes 
- [Europe](https://everydayimtravelling.com/class/europe/) 
- [Asia](https://everydayimtravelling.com/class/asia/) 
- [South America](https://everydayimtravelling.com/class/south-america/) 
- [Food](https://everydayimtravelling.com/class/meals/) 
- [Western Europe](https://everydayimtravelling.com/class/europe/west-europe/) 
 
## Tags 
- [Budget](https://everydayimtravelling.com/tag/price range/) 

Serving to AI show you how to 

So, if AI is misinterpreting your web site, producing faulty summaries, or skipping important content material, there’s a cause, and it’s fixable. 

It’s not all the time your copy. Not your design or your metadata. It’s simply that these language instruments want a bit steerage. Sooner or later, llms.txt may very well be the way in which to provide it to them, and also you accomplish that in your phrases. 

Do you want assist creating an llms.txt file or changing your present content material to Markdown for LLMs? Yoast SEO can robotically generate an llms.txt file for you. 

Edwin Toonen

Edwin is an skilled strategic content material specialist. Earlier than becoming a member of Yoast, he labored for a top-tier internet design journal, the place he developed a eager understanding of easy methods to create nice content material.

---