The agent-ready approaches that are catching on, and what AI agents actually use

In my last post, I checked out how the online is altering for brokers and what agent-ready means: a website that lets an agent learn its content material, question it for particular solutions, and act on it, whereas website homeowners can inform which brokers are visiting and determine what they could do. This publish goes by means of the approaches website homeowners can use to get there, how far alongside each is, and what we find out about how brokers use them.

A handful of proposals have made it off the drafting board and onto plenty of actual websites, though most are nonetheless not printed requirements:

1. llms.txt and markdown variations of pages assist brokers learn
2. Structured information and Mannequin Context Protocol (MCP) servers assist them question
3. Discovery recordsdata inform them what’s there
4. Content material Alerts covers a part of the belief query
5. WebMCP, which can get its personal publish, is aimed toward taking motion

llms.txt: the agent “index”

llms.txt was initially proposed by Jeremy Howard at llmstxt.org, and it’s a markdown file on the root of a website, like instance.com/llms.txt, that provides language fashions a brief, curated map of the location. It opens with the location’s title and a one-line abstract, then lists essentially the most helpful pages below headings, every with a hyperlink and a sentence about what’s there. As an alternative of getting to crawl each web page, an agent can learn it in a single go and know what the location is about.

Regardless of the same naming conference, it doesn’t work like robots.txt. robots.txt tells crawlers what they could fetch, whereas llms.txt doesn’t management something – it’s simply an index which acts as a type of FYI to brokers.

Though it hasn’t been adopted by any requirements physique, and AI corporations don’t say whether or not their crawlers really learn it, its utilization has unfold. Even OpenAI, Anthropic, and Google every publish one for his or her developer docs, and documentation platforms like Mintlify and GitBook generate theirs routinely. Lighthouse’s Agentic Looking class checks for one as effectively.

The proposal reached model 2 this yr, which provides one requirement: the location ought to level to the file with a Hyperlink header or a <hyperlink rel=”describedby”> tag, so brokers know precisely the place it sits (extra on each within the discovery part beneath).

Markdown variations of pages: smaller and cheaper

Websites can provide markdown variations of their pages for brokers to devour as a substitute of the frontend HTML. The identical web page in markdown is a fraction of the dimensions of its HTML, as a result of the navigation, scripts, styling, and advertisements are gone and solely the core content material is left. In an instance shared by Cloudflare, one web page dropped from 16,180 tokens as HTML to three,150 as markdown. The benefit of a smaller web page is that it’s cheaper for the agent to parse, and leaves extra room in its context window for the precise query.

Websites can provide markdown in two methods:

1. Separate .md net handle: this often sits on the web page URL with .md added to the tip. This method is extensively used, together with in Anthropic’s and Stripe’s documentation. An agent has to know that handle exists, so websites often level to it with a <hyperlink rel=”alternate”> tag, a Hyperlink header, and/or an inventory in llms.txt.
2. Content material negotiation: the agent requests the unusual web page URL with an Settle for: textual content/markdown header, and the server solutions in markdown as a substitute of HTML. Coding brokers like Claude Code and OpenCode already ship that header.

In apply, the 2 usually work from the identical content material. A website, or the plugin that handles this for it, can generate the markdown model of a web page as soon as, when the web page is printed or up to date, and retailer it. Each the .md handle and a negotiated request then pull from that one saved copy, and the one distinction is how the agent requested for it. Working from one supply is extra environment friendly than changing the web page on each request, and it means an agent will get the identical, up-to-date content material whichever approach it asks.

Provide each the .md web page and content material negotiation

It’s a good suggestion to supply each the .md handle and content material negotiation, so an agent will get markdown no matter its desire. Based mostly on some evaluation, brokers appear to fetch .md addresses instantly extra usually than they negotiate, presumably as a result of websites level to these addresses with <hyperlink> tags, Hyperlink headers, and llms.txt recordsdata, whereas negotiation catches the brokers that ask for markdown on the regular URL. When each routes pull from the identical saved copy, providing the second doesn’t price something and isn’t any extra sophisticated.

Negotiation does want cautious caching, although. The HTML and markdown variations share one URL, so the server has to ship a Differ: Settle for header, and any web page cache or CDN (content material supply community) in entrance of the location has to respect it and retailer the 2 variations individually. If it doesn’t, an unusual customer may be served the cached markdown as a substitute of the HTML web page, and the browser could deal with it as a file to obtain. Cloudflare provides Differ: Settle for to its transformed responses because of this. Test together with your host whether or not its web page cache retains the 2 variations aside.

A unique method: Cloudflare’s Markdown for Brokers

Cloudflare takes a distinct route with Markdown for Agents, which it launched in February 2026. It additionally makes use of content material negotiation, but it surely doesn’t ask for a markdown model saved in your website – it fetches the web page’s regular HTML out of your server and converts it at its edge. The conversion strips the navigation, scripts, and styling, places the web page’s title and outline on the prime, and provides any JSON-LD structured information in a block on the finish. Every transformed response additionally carries an x-markdown-tokens header with an estimate of what number of tokens the web page will use, so an agent can determine how a lot of its context window to spend on it.

That makes it the best choice for a website already utilizing Cloudflare, since it really works on any platform and there’s no second copy of the content material to maintain updated. Nonetheless, the markdown is barely pretty much as good because the HTML it attracts from (for instance, if the web page is difficult to parse or the content material solely seems by means of JavaScript), and you may’t edit or management what Cloudflare produces.

Discovery: telling brokers what’s there

Discovery is how an agent finds out what a website gives for machines with out guessing at URLs. Most of it makes use of elements of the online that exist already.

Two sources for agent discovery have already come up on this publish: the <hyperlink> tag and the Hyperlink header, which internet sites use to level brokers to llms.txt and to markdown variations of pages. The hyperlink tag is the older of the 2. It sits in a web page’s <head>, and WordPress has used <hyperlink rel=”alternate”> to level to its RSS feeds because the early 2000s. The Hyperlink header carries the identical data within the response headers, so an agent can see every part a website makes out there with out downloading or parsing the web page, and one header can level to llms.txt, the sitemap, the feed, and an API catalog directly.

The newer sort of discovery supply is a set of agent-focused recordsdata below /.well-known/, a folder the online put aside in 2010 for site-wide recordsdata at mounted addresses. It already holds older recordsdata, like safety.txt, and the agent recordsdata are the most recent additions:

• /.well-known/api-catalog lists a website’s APIs. It’s outlined in RFC 9727, printed in 2025, and it’s the one one of many new agent-focused proposals on this publish that has grow to be a printed commonplace, reviewed and agreed upon by the IETF (Web Engineering Job Pressure).
• /.well-known/agent-skills/index.json lists Agent Abilities, that are brief markdown recordsdata that educate an agent use the location. The talent format got here from Anthropic and is documented at agentskills.io. The discovery convention for this can be a draft led by Cloudflare.
• MCP server cards describe a website’s MCP server, together with the place to attach and which protocol variations it helps, and a website lists them in an AI catalog at /.well-known/ai-catalog.json. A2A (Agent2Agent) agent cards do the identical for a website’s personal AI agent, at /.well-known/agent-card.json. Each are current, and few websites or brokers use them but.

These recordsdata assist an agent that’s constructed to search for them, however most mainstream brokers aren’t but. Agent-ready checkers and rankers do rating websites on these recordsdata, so that you’ll see them in agent-readiness studies both approach.

Content material Alerts: agent permissions

robots.txt has at all times mentioned who could fetch what, and now Content Signals provides a line saying what brokers and bots could or could not do with the data they fetch:

Person-agent: *
Enable: /
Content material-Sign: search=sure, ai-input=sure, ai-train=no

• search covers search indexing
• ai-input covers utilizing a web page as reside enter to an AI reply
• ai-train covers coaching fashions

The above instance says sure to look and AI solutions, and no to coaching. Cloudflare launched this format in 2025 and provides it to the robots.txt recordsdata it manages for its prospects. The identical indicators will also be despatched as a Content material-Sign header on every web page.

Cloudflare’s Markdown for Brokers (talked about above) makes use of this header, and in case your server doesn’t ship one, the markdown Cloudflare serves on your pages defaults to ai-train=sure, search=sure, ai-input=sure. Its docs don’t point out checking your robots.txt first, so for those who don’t need your content material used for coaching, your server must ship its personal header as effectively to be on the protected aspect.

The IETF’s AI Preferences working group is writing a typical vocabulary for a similar three concepts. Brokers don’t must respect these indicators, since they’re voluntary for now, and most crawlers don’t learn them but, however they’re nonetheless the clearest solution to state your place, so that they’re a “might-as-well” sort of implementation.

Structured information: previous, easy, and nonetheless helpful

Schema.org structured information has been round since 2011, when Google, Bing, and Yahoo launched it collectively so whereas it’s not among the many new approaches to agent-readiness, it continues to matter. Schema.org is a shared vocabulary of varieties, like Product, Occasion, Article, Group, and FAQPage, every with its personal properties. A website makes use of it to label the details on a web page, often in a block of JSON-LD, which is the format Google recommends. The block sits in a <script sort=”utility/ld+json”> tag, separate from the web page’s design, and appears like this:

<script sort="utility/ld+json">
{
  "@context": "https://schema.org",
  "@sort": "Product",
  "title": "Ceramic espresso mug",
  "gives": {
    "@sort": "Provide",
    "value": "24.00",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}
</script>

For an agent, this implies the details are labeled and straightforward to know, and it doesn’t must attempt to work out from the format which quantity on the web page is the worth, or whether or not the date close to the highest is when an article was printed or when it was final up to date. Properties like sameAs additionally assist make it clear which firm or particular person a web page is about by linking them to their Wikipedia or LinkedIn web page. It’s the identical markup Google makes use of for wealthy outcomes, just like the star rankings and costs in search listings, so many websites have already got some, and for shops, occasions, and recipes it’s essentially the most extensively deployed machine-readable layer on the internet.

Structured information does have a disadvantage, although: lots of the instruments brokers use to fetch pages flip the HTML into plain textual content or markdown and drop each <script> block on the way in which, JSON-LD included. Cloudflare’s Markdown for Brokers is an exception, because it retains the JSON-LD and provides it to the tip of the markdown. Google already requires structured data to match what’s seen on the web page, and AgentReady.org, a brand new open framework for agent readiness that Elementor is a contributor to, offers related steering for brokers: hold the details that matter within the seen textual content too, and don’t go away them solely within the markup.

Structured information exhibits how far fundamental on-page hygiene can go for brokers. It’s search engine optimization work that websites have been doing for 15 years, and brokers can use it as is. A whole lot of WordPress SEO plugins generate most of it routinely from content material that’s already on the location, and Elementor’s widgets include schema settings of their very own, so many website homeowners have it already.

MCP servers

MCP, the Mannequin Context Protocol, was launched by Anthropic in late 2024, and most main AI assistants and coding instruments now help it. The place llms.txt and markdown assist an agent learn a website, an MCP server lets it ask the location direct questions and get solutions again. The server publishes a listing of instruments, every with a reputation, an outline, and a schema for its inputs, and an agent related to it might probably name these instruments with out clicking by means of the location. A server can even provide sources, that are paperwork or information the agent can learn, and prompts, that are ready-made directions, although instruments are what most servers lead with. Most site-level servers now run remotely over HTTP, at an handle on the location, so nothing needs to be put in on the consumer’s pc.

Firms largely run one so individuals can work with them from the AI assistant they already use. For instance, Microsoft runs a free, public MCP server for its Be taught documentation, so a developer can search and browse Microsoft’s docs from inside their coding device. Stripe’s server at mcp.stripe.com lets a signed-in consumer work with the purchasers, funds, and invoices in their very own account. In response to Shopify, each Shopify retailer has a storefront MCP server at /api/mcp, which an agent can use to look the shop’s catalog, reply questions on its insurance policies, and handle a purchasing cart.

These instruments give brokers reside and exact solutions like the present value or stock degree, and a request like “discover in-stock mugs below $30” is usually a single name with typed inputs as a substitute of the agent having to learn by means of a number of pages.

To make use of any of those, somebody has to attach the server by including its handle to their AI assistant, and in some instances the consumer additionally has to sign up to an account. Discovery is beginning to assist: the official MCP Registry, launched in preview in September 2025, lists public servers, and the MCP server playing cards from the invention part let brokers discover a website’s server by means of its AI catalog. An individual nonetheless makes the connection, although, so whereas MCP fits website homeowners, groups, and prospects who use a service commonly, it isn’t sensible for an unusual customer. WebMCP, a proposal from engineers at Google and Microsoft, is aimed toward filling that hole: it lets an online web page provide instruments on to an AI agent working within the customer’s browser, with no server to run and nothing for the customer to attach. We’ll take a look at WebMCP extra intently within the subsequent publish.

Early-stage agent-ready proposals

This house is altering quick, and new concepts hold being proposed. Listed here are some early-stage ones which are additionally value understanding about:

• Web Bot Auth lets an agent signal its requests so a website can test who it truly is, as a substitute of counting on the user-agent string, which may be spoofed. It’s nonetheless in IETF drafts, and addresses an essential space associated to agent belief and safety that wants consideration.
• NLWeb, from Microsoft, offers a website an /ask endpoint that solutions questions in pure language. Adoption appears low for this.
• DNS-based discovery proposes a solution to discover a website’s agent providers by means of its DNS information, earlier than loading something. It’s nonetheless an early IETF draft.
• Commerce has its personal protocols for brokers that purchase issues, from OpenAI, Google, and the cost corporations. They’re transferring quick and can matter lots for on-line shops.

What you are able to do now

Most of what helps brokers immediately helps individuals too, and it’s all value doing no matter the place these specs land:

• Make the web page itself simple to learn. Use actual headings so as, hyperlinks which are hyperlinks, buttons which are buttons, and content material that’s within the HTML when the web page masses moderately than loaded later by JavaScript. Brokers learn the HTML web page greater than the rest, so this has the largest impact.
• Repair your types for accessibility: labels, autocomplete values, error messages tied to their fields with aria-describedby, and a standing message that says success. Accessibility work helps brokers use types too, and Elementor’s Ally Checker is one solution to discover what wants fixing.
• Add structured information. Merchandise, occasions, organizations, articles, and FAQs all have schema.org varieties, and JSON-LD is the format each search engines like google and brokers learn.
• Test what you’re blocking. Have a look at your robots.txt, and likewise on the bot settings in your host and CDN, which may block AI brokers with out the location proprietor understanding. Determine which brokers you need, then use Content material Alerts to say what they could do together with your content material.
• Provide markdown variations of your pages, each as .md addresses and thru content material negotiation on the conventional URL. There are a lot of options for this, together with Elementor’s Generate website markdown characteristic. Whichever you employ, be certain your web page cache retains the markdown and HTML variations separate and equally out there.
• Add llms.txt, and level to it with a Hyperlink header or <hyperlink rel=”describedby”> tag. Whereas it doesn’t change a lot by itself, you would possibly as effectively add it because it takes minutes to arrange.
• Promote what you publish. When you have feeds, a sitemap, markdown pages, or an API, record them in Hyperlink headers and <hyperlink> tags so an agent can simply discover them.
• When you run WordPress, you’ve obtained a head begin for all the explanations detailed in my post on WordPress being born ready: feeds, sitemaps, and a REST API all inbuilt. The Talents API in core, with the MCP Adapter, lets plugins describe what an agent can do on the location and expose it over MCP.
• Measure what brokers do. Your logs present which brokers go to and what they request. When you control them, you may get extra perception into what brokers really do in your website, and the place you need to put your focus.

For a full guidelines, The Website Specification lists what a great website ought to have, agent readiness included, with every merchandise tagged as required, really useful, elective, or keep away from, and a hyperlink to the usual behind it.

The place this leaves us

A whole lot of what brokers really use immediately is expounded to the common net web page: HTML that’s simple to learn, structured information that identifies the details, and markdown variations of the identical pages. The newer recordsdata, like llms.txt and those below /.well-known/, take little effort so as to add, and can assist brokers which are constructed to search for them. Belief is the world that also wants work, since Content material Alerts is dependent upon crawlers selecting to respect it and Internet Bot Auth remains to be in drafts. MCP handles querying, and even performing, as soon as somebody connects it, which makes it a greater match for a website’s common customers than for a first-time customer.

My next post will decide up from right here and take a look at WebMCP, which goals to deal with the various use instances the place an agent ought to be capable of submit a kind, ebook a time, or purchase one thing for an unusual customer on a website.