For a few years we had two audiences for our web sites: people, and bots. People wanted an intuitive, user-friendly interface that helped them discover the content material or actions they had been on the lookout for; and bots like Googlebot wanted clear, semantic, machine-readable recordsdata that they might glide by to grasp what a website was about.
Now, we’re at a distinct stage of the online, and there’s a brand new sort of customer who additionally wants consideration: AI brokers. Sure, AI brokers are simply one other kind of bot, however the best way they function and “assume” is totally different, and due to this fact their wants are distinctive.
With brokers within the image, we’re seeing that an increasing number of requests to web sites are coming from software program despatched there by brokers to learn pages, reply questions, and do issues on behalf of individuals. In my last post, I wrote about why WordPress was born prepared for this: feeds, a REST API, structured content material, and 20 years of letting different software program learn and write a website ready WordPress to be a number one answer for the agentic net. On this submit we’ll check out the online as a complete: the way it’s altering, who’s writing the brand new guidelines, and what agent prepared really means. Within the subsequent submit, I’ll get into the actions you’ll be able to take already so your web site is agent prepared and forward of the curve.
Bots now outnumber individuals
In June 2026, Cloudflare CEO Matthew Prince printed findings exhibiting that automated visitors had handed human visitors. Cloudflare Radar put bots at about 57% of requests for HTML pages on Cloudflare’s community, versus about 43% from individuals, and the pattern continued over the next months. Prince had initially projected that this shift would solely occur in 2027, however the agentic net is transferring quicker than anticipated.
Individuals are visiting much less
Whereas this knowledge does depart some open questions (prefer it’s not all the time potential to distinguish a bot from an individual, and plenty of that bot visitors is crawlers, not brokers doing duties on behalf of people), what is evident is that individuals are visiting websites much less. LLMs and brokers now reply many questions immediately, so an individual can get what they want from a website with out really viewing it, which implies that in an increasing number of instances brokers are visiting, and people aren’t.
We’ve constructed for bots earlier than
Optimizing websites for this new actuality jogs my memory plenty of how we constructed our websites for web optimization: ensuring we had readable URLs (like WordPress’ “fairly permalinks”), optimized title tags, sitemaps, alt textual content on pictures, and readable content material provided primarily in indexable HTML. We had been constructing for bots then too, ones that crawled pages and listed them, and the bots we’re constructing for now, brokers, do the identical but additionally learn, reply questions intelligently, and take actions.
Who’s writing the principles for the agentic net?
From the inception of the online, it has adhered to openness. It’s due to this open method, and requirements crafted over years to maintain it that method, that the online developed into the wild and fantastic factor that it’s. Simply as nobody owns the online, no one owns the agentic net both, and its guidelines are additionally being written within the open by a mixture of requirements our bodies, browser makers, AI firms, platforms, and particular person builders.
Requirements our bodies and browser makers
A few of that work occurs in additional formal settings: the W3C’s Web Machine Learning Community Group is growing WebMCP, which engineers at Google and Microsoft first drafted collectively in August 2025; and the Web Engineering Process Power (IETF) has a working group on AI Preferences, a shared vocabulary for what AI could do with content material, and one other on bot identification that we’ll take a look at beneath.
Browser makers are additionally exploring these agentic instructions, and every of them takes a stand on new proposals: Chrome and Edge are each testing WebMCP with actual websites by origin trials; Safari and Firefox don’t assist it.
Apple’s WebKit group and Mozilla, have printed formal positions on it: WebKit opposes it, and Mozilla’s place is labeled impartial.
Grassroots proposals
A number of this work additionally begins much less formally and in a extra grassroots method. Jeremy Howard proposed llms.txt on an internet site of its personal in 2024, whereas Cloudflare printed Content Signals and a draft for listing Agent Skills on GitHub. Anthropic released MCP as an open protocol in late 2024, and it was adopted rapidly by the opposite AI firms.
Audits and checkers
The instruments that examine websites form what will get adopted, too. In Might 2026, Google’s Lighthouse added an Agentic Browsing category that checks if a web page works properly for brokers: whether or not its buttons and fields have names an agent can learn, whether or not the format stays nonetheless so an agent doesn’t click on the improper factor, whether or not the positioning has an llms.txt file, and whether or not the web page registers WebMCP instruments and marks up its kinds for them.
In September 2026, Lighthouse 13.5 added a examine for AI catalogs, that are recordsdata at /.well-known/ai-catalog.json that record a website’s agent companies, corresponding to its MCP servers. As a substitute of a rating out of 100, the Agentic Searching class exhibits what number of of its checks a web page passes. Be aware that the entire class is presently labeled as experimental. Chrome’s DevTools for agents also can run Lighthouse from inside a coding agent, so the agent can audit a web page, repair what it finds, and run the audit once more.
Cloudflare’s Is It Agent Ready? scores a website’s agent recordsdata and robots.txt, and provides you a immediate on your coding agent to repair every failing examine. The Website Specification collects all of those checks, and lots of extra, into one guidelines which you could run in opposition to your website, with every merchandise tagged required, really helpful, optionally available, or keep away from.
Nonetheless, regardless of all this work, no new requirements have emerged as formally required, and it’s all nonetheless very a lot in flux.
What agent prepared means
OK, we all know agent interactions with our web sites are growing, however how will we prepare, or “agent prepared”, for them?
Let’s take a look at how websites are constructed immediately to attempt to perceive it. An internet web page is a set of directions for “drawing” one thing. The HTML says the place the heading goes, the CSS says the way it seems to be, and JavaScript typically builds half the web page after it hundreds. An individual wanting on the web page can see a product, a value, and an “Add to cart” button, whereas an agent sees hundreds of nested components and has to work out which quantity is the value and which is the transport price, the previous value, or the evaluate rely.
Brokers cope with this in certainly one of two methods:
- Instruments like the online instruments in ChatGPT and Claude fetch a web page to reply a query: they obtain the HTML and convert it to markdown textual content, discarding the format.
- Browser brokers, like Gemini in Chrome, Claude in Chrome, or ChatGPT’s agent mode, learn the web page’s accessibility tree or take screenshots, then click on by the positioning the best way a gradual, cautious particular person would.
Each depend upon the agent studying the web page accurately, however they don’t (or can’t) all the time.
Why can’t instruments or brokers all the time precisely seize and perceive net pages? There are lots of causes, together with all of the issues standing between an agent and the content material like cookie banners, CAPTCHAs, bot safety that blocks something that doesn’t seem like a traditional browser, content material that solely seems after scrolling, and kinds whose error messages are pink textual content with nothing within the code indicating which discipline they’re referring to. An individual can principally work round all of those obstacles comparatively simply, however an agent typically will get caught, or offers up and fills within the hole with a (dangerous) guess.
Learn, question, act
An agent prepared website makes three issues simple for an agent: studying, querying, and appearing.
Studying means getting the content material with out having to dig it out of convoluted navigation, adverts, and theme markup. As well as, all of that stuff associated to the visible look of a web page can typically make it a number of instances the dimensions of its precise content material. Since brokers burn tokens looking for what they’re on the lookout for, these bigger pages can find yourself being costly, so providing the content material to an agent in an environment friendly, optimized method saves cash, and likewise saves room of their context home windows.
Querying means asking a particular query, like “when is the following occasion?” or “is that this in inventory in measurement 9?”, and getting a direct, structured reply quite than having to go looking by a web page.
Appearing means submitting a kind, reserving a time, or including one thing to a cart.
Studying and querying are principally solved, and plenty of what makes a website helpful for brokers is figure you might already be doing for accessibility. An agent that falls again to shopping reads the accessibility tree, the identical construction a display screen reader makes use of, so actual headings and buttons which are really actual buttons assist a blind customer and an agent by the identical code. Making a website accessible for individuals is cause sufficient by itself, however it has by no means been the one profit: semantic markup has helped with seek for twenty years, and brokers are the latest kind of bot that’s assisted by it.
Appearing is the place the online has the furthest to go: an agent normally has to discover a kind’s fields on display screen and sort into them, which breaks every time the design adjustments. Labeled kind fields assist, since an agent can discover a discipline by its label as a substitute of its place on the web page, however a named motion with a transparent description and typed inputs, like “search merchandise by key phrase and most value,” retains working nonetheless the web page seems to be.
Understanding which brokers to let in
Being agent prepared isn’t nearly letting all brokers in, however quite letting the suitable brokers in, and that basically issues as soon as brokers aren’t solely studying websites, however begin appearing on them. A website can be eager about welcoming a client’s assistant, however would additionally need to block a scraper, and to do this it could want to have the ability to inform them aside.
As well as, website homeowners have the suitable to know who and what precisely is crawling and accessing their website, however that’s arduous to do immediately, as a result of a bot solely introduces itself with its user-agent string, which is a line of textual content despatched with each request that the bot writes itself. Nothing prevents a script from calling itself GPTBot or Chrome, and I’ve seen this in motion alone website, the place one tackle known as itself Chrome on a Mac, then node, then GPTBot, all inside 12 seconds.
When names can’t be trusted, websites find yourself blocking based mostly on guesses, which might backfire. For instance, an agent readiness scanner examined my website by pretending to be OpenAI’s GPTBot. Cloudflare observed the request didn’t really come from OpenAI and rightfully blocked it, however the scanner then reported that my website blocks AI brokers – an issue that wouldn’t have occurred if it had been trustworthy about who it was!
It’s potential to determine plenty of bots by evaluating a request’s tackle with the tackle ranges that OpenAI, Anthropic, Google, and others publish for his or her crawlers. Nonetheless, whereas that works for crawlers operating on an organization’s personal servers, it received’t for an agent operating in somebody’s browser or on their very own laptop computer. The repair being labored on is Web Bot Auth, an IETF draft wherein an agent indicators every request with a non-public key and the positioning checks the signature in opposition to a public key that the agent’s firm publishes, so a bot attempting to faux it’s related to a specific firm received’t move.
Net Bot Auth is already in use: OpenAI’s ChatGPT agent indicators its requests this manner. In October 2025, Visa and Cloudflare introduced the Trusted Agent Protocol, which builds on Net Bot Auth so shops can acknowledge buying brokers appearing for actual prospects.
Agent prepared doesn’t imply open season
Be aware that “agent prepared” doesn’t imply it needs to be open season for AI brokers in your content material. Some website homeowners assume that “make your website readable by AI” means “let AI take all of your stuff,” however being arduous for brokers to learn doesn’t cease your content material from getting used, and principally simply leaves you out of the solutions individuals now get as a substitute of visiting. Letting brokers learn your website and letting AI firms prepare on it are separate selections, and you may resolve to do both or each, as I’ll present in my subsequent submit.
Constructing for each audiences
As I discussed earlier, web sites have all the time had two audiences: individuals, and the bots that index pages for them. Brokers are bots, however as a result of they cause about what they learn, in some methods they behave extra like a human viewers. They sit someplace in between, studying and appearing on a website the best way software program does, however on behalf of a particular one that needs a solution or wants one thing achieved.
web optimization already taught us the best way to construct for most of these audiences, and what to keep away from. The tips aimed solely at bots, like key phrase stuffing and cloaking, which meant exhibiting search engines like google totally different content material from what individuals noticed, labored for some time however ultimately received websites penalized. The websites that did properly in the long term had been at the beginning good for individuals, which normally meant they had been additionally simple for machines to learn (a properly constructed, semantic and accessible website is simpler for bots to learn too).
I count on agent readiness to go the identical method. Websites which are properly constructed and have already got machine-readable content material (like WordPress, as I wrote in my earlier submit) shall be simpler for brokers to make use of, and agent recordsdata and markdown variations of pages have to match what individuals see, not cloak a distinct message for the bots.
An opportunity to form the requirements
The world of web optimization additionally exhibits how requirements for bots come collectively. For years every search engine did issues its personal method, till Google, Yahoo, and Microsoft agreed on the sitemaps protocol in 2006 and on schema.org in 2011. Each had been printed for anybody to make use of, however the various search engines labored them out amongst themselves first. Brokers are in the same stage now, with one distinction: a lot of the work is occurring in public whereas it’s nonetheless a draft. Anybody can touch upon a GitHub difficulty, be a part of a W3C group group, or submit to an IETF mailing record. The WordPress group runs a giant share of the websites these specs are about, and we will participate whereas the drafts are nonetheless works in progress.
In my subsequent submit, I’ll undergo the agent prepared approaches which are catching on, from llms.txt and markdown variations of pages to MCP servers, and what you are able to do by yourself website now.
