/** * TH3STAR - Force WordPress Blog Posts INDEX, FOLLOW * * Applies to individual published blog posts only. * Overrides Yoast NOINDEX and NOFOLLOW settings. */ /** * Force INDEX, FOLLOW in Yoast robots string. */ function th3star_force_blog_robots_string( $robots ) { if ( ! is_singular( 'post' ) ) { return $robots; } $directives = is_string( $robots ) ? array_map( 'trim', explode( ',', $robots ) ) : array(); // Remove conflicting indexing directives. $directives = array_filter( $directives, function ( $directive ) { $name = strtolower( trim( explode( ':', $directive, 2 )[0] ) ); return ! in_array( $name, array( 'index', 'noindex', 'follow', 'nofollow', 'none' ), true ); } ); $directives = array_values( $directives ); // Enforce INDEX, FOLLOW. array_unshift( $directives, 'index', 'follow' ); return implode( ', ', array_unique( $directives ) ); } /** * Force INDEX, FOLLOW in Yoast robots array. */ function th3star_force_blog_robots_array( $robots ) { if ( ! is_singular( 'post' ) ) { return $robots; } if ( ! is_array( $robots ) ) { $robots = array(); } // Force indexing. $robots['index'] = 'index'; $robots['follow'] = 'follow'; // Remove conflicting directives. unset( $robots['noindex'], $robots['nofollow'], $robots['none'] ); return $robots; } // Apply final Yoast robots overrides. add_filter( 'wpseo_robots', 'th3star_force_blog_robots_string', PHP_INT_MAX ); add_filter( 'wpseo_robots_array', 'th3star_force_blog_robots_array', PHP_INT_MAX ); Websites Are As Relevant As Ever, They Just Have A New Type Of Visitor: AI Agents | FREELANCE WEB DESIGNER KUALA LUMPUR
elementskit logo

Websites are as relevant as ever, they just have a new type of visitor: AI agents

new type of visitor.webp.webp

Many are questioning if it’s the top of the web site period, now that individuals are interacting an increasing number of with info throughout the confines of their AI shoppers like ChatGPT, Claude, and so on. as an alternative of visiting the websites themselves.

When common analytics, it might certainly seem that it’s the top of the net as we all know it as a result of individuals aren’t visiting web sites. Nevertheless, once we look intently at what’s actually occurring on web sites, who (or what) is visiting them, and the way usually, a special story emerges: web sites are very related, and can proceed to be, however they now need to be able to welcome a brand new sort of customer: AI brokers.

I gave a discuss this final week on the CloudFest Americas conference, explaining the place the net is at, and the way website house owners can put together to maximise this new actuality and alternative. The discuss was titled “Is the Web site Dying?” and right here’s a recap of what I shared.

Miriam Schwab speaking on stage at CloudFest Americas 2026

Photograph credit score: Marcus Burnette

The web site has been declared lifeless earlier than

Each few years somebody broadcasts that the web site, and WordPress together with it, is lifeless. Apps had been going to kill it, then social media, and at one level individuals even mentioned voice assistants like Amazon Echo and Google House would end it off! With web site analytics wanting the best way they do now, I perceive why individuals are predicting the demise of the net once more. The numbers don’t look so good.

Pew Research Center studied the true looking knowledge of 900 US adults to see how Google’s AI summaries have an effect on clicks. When Google didn’t present a abstract, individuals clicked by means of to an internet site in 15% of visits. When Google confirmed an AI abstract on the high of the outcomes, that dropped to eight%. The AI abstract provides individuals the data proper on the outcomes web page, so fewer of them want to go to the location it got here from.

Bar chart: with no AI summary 15% of visits click through to a website, with an AI summary 8% (Pew Research Center, July 2025)

For about 30 years, the net ran on a easy deal. You let serps crawl your website, they despatched you guests in return, and also you turned these guests into clients. That deal is breaking, and it isn’t ever going to be the identical once more.

Site visitors is definitely going up – simply not from individuals

Whereas click-throughs are happening, total site visitors to web sites is altering, and in some methods it’s even growing. In September, Cloudflare reported that greater than half of web site visitors is not human, and each day requests from AI brokers on its community grew by greater than 1,700% in a single yr!

Line chart: daily AI agent requests grew 1,700% in one year (Cloudflare, September 2025)

Individuals usually present that quantity as proof that the web site is dying, as a result of they see it because the demise of the human internet customer. However I learn it the alternative approach: bots exit of their approach to go to web sites, and in growing numbers and frequency, as a result of web sites have what they want. They arrive to learn pages and test info, and more and more to do one thing on a website on behalf of an individual. When somebody asks an AI assistant a query, the assistant might learn a number of pages to reply it whereas the individual themself by no means visits any of them. So whereas technically individuals could also be visiting websites much less, they’re nonetheless visiting websites by way of their proxies, AI brokers. So the web site isn’t dying; its viewers is just evolving.

What brokers need: learn, question, and act

If AI brokers are a big and rising share of your guests, how do you make your website a great place for them? Begin with what they’re attempting to do. Brokers go to websites for 3 causes, and getting a website prepared for them, which we name agent-readiness, comes right down to these three: learn, question, and act.

The three things agents want from a website: read, query, and act

Studying comes first, as a result of the opposite two rely upon it. If an agent can’t learn your web page, it might probably’t reply a query about you, and it positively can’t take an motion in your website. An agent doesn’t want your navigation, your cookie banner, or your advert slots. It wants the precise content material in a clear format it might probably course of, and when a website presents a markdown model of a web page, brokers usually select it over the HTML.

Question is narrower. When somebody asks an assistant what your plan prices or whether or not you’re open on Sunday, the agent isn’t after your entire web page. It desires one reality, and it must be certain it has the proper one. If that reality seems solely in a picture, or sits subsequent to final yr’s costs, the agent has to guess, and it might probably simply give the unsuitable reply.

Performing is the place the net has the furthest to go. Right now, an agent that wishes to fill in your contact kind has to have a look at the web page the best way an individual would, work out which field is which, and kind into it. That’s sluggish, and it usually fails. WebMCP is a proposal that goals to repair this. It lets a web page inform the agent in your browser which actions it presents, like “ship a message” or “search merchandise.” There are different methods for brokers to take motion, like Mannequin Context Protocol (MCP) servers (simple to confuse with WebMCP, however a special factor), however these solely work for individuals who join them to their AI assistant. WebMCP works for any customer’s agent, within the browser they already use.

Good bot or dangerous bot?

Bear in mind how for the final 15 years we’ve been intent on blocking bots as a result of they waste our server sources and may usually be malicious? Properly, brokers are a kind of bot, so how ought to we deal with them? Can we allow them to in or hold them out? Are they good bots or dangerous bots?

Good bot or bad bot: a scale weighing agents against bad bots, with the Cloudflare quote "Our first instinct was to block all automated traffic" (Cloudflare, September 2025)

Cloudflare shared that their first intuition was to dam all automated site visitors, brokers included. They realized they couldn’t, as a result of turning an agent away would possibly imply turning away the one that despatched it. The problem for all of us now is determining how you can let the nice brokers in whereas conserving the dangerous bots out. It’s a piece in progress.

The requirements hold transferring

Dangerous bots are just one transferring a part of this new agentic internet. The foundations for the way websites discuss to brokers are altering simply as quick, there are already a variety of them, and new proposals hold showing. Many are nonetheless drafts or proposals, and a few have already modified since they first got here out. No website proprietor, and only a few builders, can hold observe of all of it.

Word cloud of agent-readiness standards and proposals: robots.txt, llms.txt, llms-full.txt, markdown for agents, Content Signals, api-catalog, MCP server cards, A2A agent cards, NLWeb, WebMCP, Web Bot Auth and more

The brand new AI customer received’t wait whereas a website proprietor tries to compensate for requirements drafts. If an agent can’t learn a website, it’ll get its reply from one it might probably. That’s why maintaining must be the platform’s job. Web site house owners by no means needed to discover ways to construct an XML sitemap from scratch, as a result of plugins took care of it, and agent-readiness ought to work the identical approach.

WordPress has a bonus right here due to its scale. It runs virtually 40% of the net, so one plugin replace can deliver a brand new customary to hundreds and even thousands and thousands of web sites (like within the case of Elementor) without delay.

4 steps website house owners can take immediately

We’ve established that the net is altering, and AI brokers are visiting websites and have their very own distinctive wants. It may be arduous to determine how you can begin optimizing your website for them, however a variety of what website house owners must do is sweet observe we’ve recognized about for years and infrequently didn’t get to, as a result of it was sluggish and technical. AI instruments and platforms can now do most of that work. Within the discuss, I walked by means of every step alongside the Agent Ready features we released in Elementor 4.4.

1. Get a baseline rating of how prepared your website is

Elementor 4.Four provides an Agent Prepared menu to WordPress. It begins with a readiness rating, break up into Discover, Learn, and Act, and each discovering presents a approach to repair it.

Elementor 4.4 Agent Ready score screen, split into Find, Read, and Act

2. Make each web page readable

Elementor’s Agent Entry space provides customers a approach to simply toggle on llms.txt, which works like a desk of contents for brokers, plus markdown variations of each web page and the invention recordsdata brokers search for below /.well-known/. All of them are generated from the location’s content material.

Elementor Agent Access settings: llms.txt, markdown content, bot access control, agent discovery, and agent browser tools

3. Set your phrases for bots

The third step makes use of a really outdated file, the nice outdated robots.txt. It’s been round since 1994 and became an official standard in 2022, and it’s nonetheless the very first thing reliable brokers test. It’s additionally the place you possibly can set your phrases for the way bots use your content material.

In robots.txt you possibly can add Content Signals, which inform serps and AI firms the way you’d like them to make use of your content material: whether or not they can present it in search outcomes, use it to reply questions, and practice on it. According to Cloudflare, lower than 1% of the websites on its community block search crawlers, whereas 17% do select to dam coaching. This means that a variety of website house owners need to be particular about what bots can and may’t do, and to make these choices themselves. Alerts are preferences, so technically a bot can ignore them, however in my evaluation it seems like most bots respect them. You possibly can set this with the Bot entry management setting in Elementor, with world defaults and settings for every bot.

Elementor Bot access control screen with global defaults and per-bot settings for search, AI input, and AI training

4. Prepare for actions

Elementor 4.Four contains 4 read-only WebMCP instruments, so an agent within the browser can get enterprise info, search the location, see its construction, and browse a web page. Actions are coming subsequent, beginning with Elementor’s Atomic Varieties.

The four read-only WebMCP tools in Elementor 4.4: get business information, search site content, explore site structure, read page content

All of that is free in Elementor. We comply with the requirements intently, so when one adjustments, we replace the plugin and each website will get the brand new model. Web site house owners don’t need to comply with each draft to maintain up.

What platforms like hosts and plugins can do

The final a part of my discuss was directed towards platform suppliers within the room, as a result of a giant a part of the job of getting websites prepared for this customer falls to the businesses that run them.

What the industry can do: make agent readiness part of every site you host, address the bot vs. agent challenge, and respect the owner’s terms

Hosts could make agent-readiness a part of each website they run, so clients see it throughout onboarding and within the dashboards they already use, as an alternative of getting to go on the lookout for it. The bot vs. agent problem is the accountability of hosts too: a firewall or problem web page that stops each automated customer additionally stops AI assistants attempting to reply a query about their buyer’s enterprise. House owners’ phrases want respecting as nicely: if a website says sure to look and no to coaching, the host’s settings shouldn’t override that.

Plugins also can make a major impression on the agent-readiness of internet sites. At Elementor, we’re bringing agent readiness to the over 24 million web sites utilizing Elementor by including these options and capabilities to the newest model of our plugin.

AI brokers are your new customer – welcome them

Fewer individuals are clicking by means of from search, however site visitors to web sites retains rising, and far of that development comes from brokers appearing for individuals. These brokers must learn your content material, discover the proper info, and more and more take motion, and most of what helps them is sweet observe that platforms can now deal with for you. The requirements will hold altering, so the sensible transfer is to let your platform, like Elementor, sustain with them for you. All of us want to think about what we’re doing to let this new customer in, and to assist it get what it wants.

Agent Prepared is out now in Elementor 4.4, and it’s free. In the event you’d wish to go deeper on any of this, my agent-readiness collection on the Elementor weblog covers every bit in additional element, beginning with WordPress was born ready for the agentic web.

Study extra on this webinar:

Related Post

Leave a Reply

Your email address will not be published. Required fields are marked *