What is llms.txt & should you use it?

AI platforms like ChatGPT and Google’s AI Overviews depend on crawlers to collect the content material they cite. However JavaScript-heavy pages, sprawling navigation, and years of accrued content material make it arduous for a crawler to search out your most necessary materials — and content material a crawler misses cannot present up in AI solutions.

An llms.txt file is supposed to provide AI crawlers a curated place to begin to allow them to perceive your website’s hierarchy and discover related content material for solutions. 

Under, we cowl what it really entails, whether or not any main AI platform has confirmed utilizing it, and learn how to construct and publish your personal model if you wish to check it.

What’s llms.txt?

The llms.txt file is a proposed customary meant to assist massive language fashions (LLMs) higher perceive and use content material from web sites.

The concept is fairly simple: As a substitute of letting AI crawlers wander round your website, you give them a curated listing of your most necessary content material. This curated listing (your llms.txt) tells AI which content material in your website it ought to take note of.

We have already got requirements like robots.txt and sitemaps to assist engines like google navigate web sites extra effectively. What’s totally different about llms.txt is that it’s particularly designed for AI fashions that may use your content material to reply questions or generate responses for customers.

There’s hypothesis that llms.txt may enhance visibility in AI-generated responses and drive referral site visitors. As of now, there is no proof that it does. 

What’s rising is the tooling round it, like Google Lighthouse, which now checks for llms.txt in website audits. That is an indication the file is being taken severely, though no mannequin supplier has confirmed utilizing it but.

What drawback is llms.txt attempting to resolve?

An llms.txt file is attempting to assist AI crawlers browse web sites extra successfully. Proper now, these crawlers run into two main challenges:

• Trendy web sites are arduous to learn. Most AI crawlers can solely learn your pages’ primary HTML — not content material that will get loaded by JavaScript. Meaning llms.txt offers a transparent, structured format that helps AI crawlers shortly digest the data.
• Most web sites have an overabundance of knowledge. AI crawlers don’t know which content material is necessary after they go to your website. In the event that they spend time scraping pages that aren’t helpful (like older weblog posts), they could generate responses primarily based on suboptimal data. That is the issue llms.txt helps clear up.

The file might also cut back crawl inefficiencies. Bots do not have limitless persistence together with your website. If a crawler burns its funds parsing your nav menu and cookie banner, it might by no means get to your precise product pages — those you need it to quote.

That is the place llms.txt is available in. It factors crawlers straight at your finest content material, so extra of it’s more likely to get learn, listed, and represented precisely in AI answers.

How are llms.txt information structured?

Your llms.txt information needs to be structured and formatted in Markdown.

Markdown is a light-weight markup language that makes use of plain textual content formatting syntax to create structured paperwork. (It is the identical format builders use in GitHub README information, and it’s simply parseable by AI methods.)

Save the file as lowercase llms.txt, which is the filename the official specification makes use of.

Some widespread Markdown components you will use in your llms.txt file embrace:

• # for the H1 heading, ## for H2s, ### for H3s, and so forth
• > for blockquotes to focus on necessary descriptions
• – or * for bullet factors in unordered lists
• [text](url) for hyperlinks to your content material
• : for including descriptions subsequent to hyperlinks to assist clarify what they result in 
• “` for code blocks when sharing technical examples

Most llms.txt information comply with the identical primary sample: an H1 together with your website or product title, an non-compulsory blockquote description, then a number of H2 sections that group associated hyperlinks, every written as a bulleted hyperlink with an non-compulsory colon-separated description.

The official llms.txt specification offers a really primary instance of how your file may look. But when your web site is massive or complicated, you may wish to add extra construction — utilizing H3s and H4s to create subsections, incorporating tables to arrange knowledge, or together with code snippets to exhibit API use.

There’s nothing fallacious with that. Markdown information are utterly readable by AI crawlers, so that you’re secure so long as you are utilizing legitimate syntax. The extra construction may really give AI crawlers extra context.

Here is a easy instance following the essential specification:

Are manufacturers utilizing the llms.txt customary?

Sure, manufacturers are already utilizing llms.txt information on their web sites, and adoption is rising.

In July 2025, solely 951 domains had printed an llms.txt. However there have been 16,670 as of September 2026. That is roughly 17.5x development in about 14 months — genuinely quick adoption for a spec that did not exist earlier than late 2024.

However zoom out: 16,670 domains remains to be a small proportion of the a whole bunch of tens of millions of lively web sites on the market. Which implies the expansion curve is actual, however the window remains to be broad open. Publishing an llms.txt right now does not put you in a crowded area — it places you forward of the overwhelming majority of your opponents.

You possibly can view any website’s llms.txt file by including /llms.txt to the tip of its area (for instance, instance.com/llms.txt). Here is how just a few manufacturers strategy theirs:

Model	What the file focuses on	The general construction
Hugging Face	Developer docs	Makes use of a number of ranges of headings (#, ##, ###, ####) to interrupt content material into separate sections. It additionally contains full code examples, a number of hyperlinks, and useful notes all through. Total, it looks like a complete data base.
Vercel	Developer docs	Begins with descriptive strains on the prime, resembling title:, description:, and tags: to provide an concept in regards to the explicit documentation that follows. And it makes use of clear headers (#, ##, ###) to arrange content material into logical sections. Beneath every part, you’ll discover step-by-step directions and sensible code examples.
Zapier	Developer docs	Makes use of headings and creates a really primary construction. It largely consists of an extended listing of hyperlinks and descriptions subsequent to them to assist clarify what they result in.
Cal.com	Developer docs	Makes use of headings on the prime (#, ##) after which jumps straight into an inventory of hyperlinks grouped into totally different sections with subheadings.

Discover how in a different way every firm approaches its llms.txt file.  

There’s nothing fallacious with this. So long as they’re utilizing legitimate Markdown, the file is machine-readable and will be simply processed by AI methods.

Do you have to use llms.txt in your website?

You should utilize llms.txt in your website as a low-cost experiment, but it surely is not necessary — your content material can nonetheless seem in AI solutions with out one. Having a file prepared prices you little if adoption grows. Simply do not anticipate it to maneuver the needle but.

No main AI platform, together with ChatGPT, Google’s AI Overviews, or Anthropic’s Claude, has confirmed that it reads or depends on llms.txt.

Google’s John Mueller additionally confirmed this on Bluesky:

That mentioned, there are some fascinating alerts.

For instance, Anthropic has published an llms.txt file by itself web site. That doesn’t imply its AI crawler is definitely utilizing these information — but it surely suggests they’re most likely not less than open to the thought.

Google additionally launched Open Knowledge Format (OKF), an open specification for packaging content material for AI brokers. However OKF is not a substitute for llms.txt. An llms.txt file faces outward, towards the crawlers visiting your website, whereas OKF is constructed for a company’s personal inside brokers.

OKF remains to be early, with the present model at 0.1. However a Google-backed customary for AI brokers reveals that making content material readable for AI is getting actual consideration.

Tooling has additionally began to catch as much as the format, even with out crawler affirmation. 

Google added an experimental Agentic Searching class to Lighthouse, Chrome’s site-auditing software, and one in all its checks appears for an llms.txt file. However that is not proof that Google’s personal crawlers are utilizing the file to reply queries, since Lighthouse audits and dwell crawling are separate methods.

We’re nonetheless within the early hypothesis part the place persons are implementing the file and hoping it’d turn into helpful sometime.

Again in March 2025, we applied this file on one in all our sister websites, Search Engine Land, to see whether or not it gives any significant benefits when it comes to AI visibility and site visitors.

In our testing, we did not discover a correlation between implementing llms.txt and improved efficiency in AI outcomes.

We additionally analyzed the server logs to see whether or not AI crawlers had been really accessing the file. From mid-August to late October 2025, the llms.txt web page obtained zero visits from Google-Prolonged bot (Google’s AI crawler), GPTBot (OpenAI’s crawler), PerplexityBot, or ClaudeBot.

Whereas conventional crawlers like Googlebot and Bingbot did go to the file, the file obtained just a few hits. Meaning they didn’t deal with the file with any particular significance.

Should you’re primarily curious what a completed file appears like, a handful of free llms.txt generator and validator instruments exist that can assist you draft and examine one shortly — they do not change the underlying uncertainty about whether or not any AI crawler reads it, however they’ll prevent time if you happen to determine to experiment.

Should you additionally wish to experiment with llms.txt by yourself website, comply with the step-by-step directions beneath on learn how to implement it.

Additional Studying: How can you do LLMO? and How can marketers adapt to LLM-powered search?

Tips on how to create an llms.txt file (step-by-step)

Create an llms.txt file by deciding which content material you wish to function, creating the file, and importing it to your web site.

1. Resolve which content material you wish to function

Resolve which content material you wish to function by choosing crucial pages to your website to incorporate in your llms.txt file.

On the very least, contemplate your:

• Services or products pages
• Up-to-date weblog posts
• Pricing web page
• About us web page
• Contact web page

These are usually the pages that may give AI a good suggestion of what your enterprise does and the way you assist clients.

Favor evergreen, canonical content material over something time-sensitive or gated. Good candidates are pages you’d need an AI system to deal with as authoritative months from now. Skip pages which might be personalised, login-gated, duplicated, or more likely to go old-fashioned quickly, like a single occasion announcement or a limited-time promotion.

2. Create the file

Create the file in a textual content editor like Notepad or Visible Studio Code and title it llms.txt.

You must format the file utilizing Markdown. A developer is not strictly required for this step — it is a plain textual content file — however one may also help make certain the Markdown syntax and construction maintain up, particularly on a bigger website.

Right here’s how the file’s construction may look:

3. Add the file to your web site

Add the file to your web site in the proper location so AI crawlers can theoretically discover it.

The precise location relies on the scope of your llms.txt file:

• If it covers your complete web site, add it to your root listing (i.e., “https://[yourdomain.com]”) so it is accessible at “https://[yourdomain.com]/llms.txt”
• If the file is particularly about documentation, place it within the corresponding subdomain (e.g., “https://[docs.yourdomain.com]/llms.txt”)

You could want a developer’s assist to really add the file. This file needs to be positioned in your server — often by your hosting management panel, resembling cPanel.

Log in to your internet hosting supplier after which navigate to cPanel > File Supervisor.

Then, go to the right listing. In case your llms.txt file is supposed for the entire website, go to the public_html/ folder. (That’s the basis listing for many domains.)

If it’s for a subdomain like “https://[docs.yourdomain.com],” navigate to the folder assigned to that subdomain — usually named /docs/ or comparable.

Add your llms.txt file there and save the modifications.

As soon as the file is uploaded, affirm the whole lot is working by opening a brand new tab and visiting the URL instantly.

You too can run a fast audit of your web site in Semrush Site Audit to substantiate that your llms.txt is being picked up appropriately.

Should you publish an llms.txt, examine your server logs periodically to see whether or not any AI crawlers are requesting it. And preserve the file up to date as you add or take away pages.

Assist AI methods entry your content material

An llms.txt file is not required for AI visibility, however extra instruments and corporations are beginning to take it severely — so including one is an easy option to hedge your bets if it does turn into customary.

It isn’t the one lever, although. Site Audit can present you what else could be limiting your AI visibility proper now: JavaScript rendering points, crawler blocks, skinny content material, and different barriers that preserve AI methods from studying your website in any respect.

Merely run the audit and click on “Points” > “AI Search” to see points impacting your AI visibility. 

Then, repair every concern to enhance your visibility in AI search.