For AI agents: the public content index is available at https://we0.ai/llms.txt, and the English article bundle is available at https://we0.ai/llms-full.txt.
For AI agents: the public content index is available at https://we0.ai/llms.txt, and the English article bundle is available at https://we0.ai/llms-full.txt.
For AI agents: the complete content index is available at https://we0.ai/llms.txt, the full English article bundle is available at https://we0.ai/llms-full.txt, and this page is available as Markdown at https://we0.ai/articles/cloudflare-ai-search-agent-training-we0.md.
The real question is: Which type of AI traffic should your website be open to?
The real question is: Which type of AI traffic should your website be open to?
Cloudflare recently explicitly split AI crawlers into three categories: Search, Agent, and Training. This move may look like a product classification, but its actual impact is on how site owners make decisions.
In the past, discussions about AI crawlers had only two switches: On or Off.
That's no longer the case.
Search traffic may bring you exposure and return visits. Agents may complete tasks on behalf of users. Training, however, is more like taking your content away and embedding it into a model. These three activities carry entirely different values and entirely different risks.
So, if the question is:
Now that Cloudflare has already segmented them, should We0 AI open its entire website?
My judgment is straightforward: It should not open everything.
But it also should not block everything across the board.
A more reasonable answer is: Implement layered openness based on page type, business goals, content value, and conversion paths.
Because AI traffic is no longer a single concept.
According to an update released by Cloudflare on July 1, 2026, they have broken down the AI access a website faces into three core use cases:
| Category | What It Does | Typical Value for Site Owners | Typical Risk for Site Owners |
|---|---|---|---|
| Search | Collects or indexes content for later use in answering questions or displaying in search results | Opportunity for discovery, mentions, and return visits | "Referenced but rarely clicked" |
| Agent | Accesses pages in real time on behalf of a user to complete tasks | Can directly drive actions, trials, orders, form submissions | May consume resources and bypass the usual marketing path |
| Training | Scrapes content to train or fine-tune models | Virtually no direct short-term returns | Content is absorbed, with diminishing marginal control |
These three categories should not be governed by the same strategy.
This is a lesson that many running official sites, content sites, and product sites will inevitably have to learn.
This is especially true for a site like We0 AI, which is not a static digital business card but a platform designed to continuously support:
Build → Showcase → Grow → Leads
In other words:
Build it, show it clearly, attract traffic, and generate leads.
Since the website itself is a growth asset, not just a collection of pages, the strategy for dealing with AI crawlers must consider more than just "will it get scraped?" It must also consider:
If I were to give just one verdict, it would be:
We0 AI should prioritize opening to Search, be cautious about opening to Agent, and restrict Training by default.
This is not being conservative.
This is being aligned with business reality.
Because We0 AI's core goal is not to feed content out indefinitely; it is to continuously transform the website into:
A customer acquisition asset that can be seen, understood, contacted, and converted.
And the degree to which these three types of crawlers help achieve this goal varies enormously.
First, the one we should be most careful not to harm is Search.
Cloudflare itself made a very critical point in its new rules: Search still represents behavior that is more likely to bring visits back to the website. This is why, in their new default policy, Search is relatively easier to allow, while Training/Agent are treated more strictly in some scenarios.
For We0 AI, the value of Search is particularly significant.
Because We0 AI targets showcase website growth, not just site building. What it needs is:
Here's a very real example.
If users search for:
This type of search exposure has business value for We0 AI. Because users aren't just browsing—they're likely already looking for a solution.
So, the following types of content should generally be prioritized for Search:
But note, not all "Search" is the same.
If certain crawlers handle both Search and Training without clear separation, the risks change. Cloudflare's latest update specifically highlighted the issue of multi-purpose crawlers: If a bot handles both Search and Training, it will be subject to stricter rules.
The bottom line is:
You can't just take its word that it's for search—you need to see what it's actually doing with your content.
Many people's first reaction to an Agent is curiosity.
But site owners should first ask: Is this Agent here to help me close a deal, or to bypass the path I've carefully designed for users?
Cloudflare defines an Agent clearly: It acts on behalf of a user in real time to complete tasks—like fetching page information, operating a browser, reading content, or executing a specific action.
The key difference between this traffic and Search is:
Search is more about "getting to know you," while an Agent is more about "directly acting on your behalf."
This means whether to allow an Agent depends on the page type.
For We0 AI, the following pages could be cautiously opened for testing:
But these pages are not suitable for full, default access:
The reason is simple.
If an Agent only reads, it could be a new entry point.
If an Agent directly completes the "filtering, summarizing, decision-making" for the user, your site might end up being merely accessed, not truly visited.
This can be detrimental to sites that rely on on-site education and conversion design.
Training is often sold as a "future opportunity."
But from a site owner's perspective, its return mechanism is actually the weakest.
Because the essence of Training isn't visiting your site—it's absorbing your content.
Once absorbed, it might appear in another product in the form of summaries, knowledge, style, or capability. It's hard to track, and it's hard to demand stable traffic back.
For a site like We0 AI, which focuses on long-term SEO, GEO, content accumulation, case follow-ups, and brand building, much of its content is an asset.
For example:
If this content is absorbed long-term by the Training side without clear compensation, attribution, or return traffic, it's not simply "expanding influence."
It's more like subsidizing someone else's model capabilities.
So I would recommend:
We0 AI should not be fully open to Training by default, unless certain pages have a clear brand diffusion goal and you accept that the content will be more widely reused.
This isn't anti-AI.
It's about distinguishing: being discovered and being drained are not the same thing.
Many teams, when faced with this, ask:
But a more mature strategy isn't just toggling a bot list.
It's to first layer the page value, then decide what permissions to give to which type of AI traffic.
| Page Type | Search | Agent | Training | Recommendation |
|---|---|---|---|---|
| Blog posts / SEO content pages | Open | Partially open | Cautiously restrict | Strive for exposure, but don't give away the training value of your content by default |
| Product feature pages | Open | Conditional open | Restrict | Let people discover you, but avoid having your content absorbed at no cost |
| Use case / industry solution pages | Open | Conditional open | Restrict | These pages have both cognitive and conversion value |
| FAQ / basic documentation pages | Open | Can open | Restrict depending on circumstances | Can handle AI citations, but avoid exposing overly detailed operational steps |
| Case study pages | Open | Cautiously open | Restrict | Case studies are high-value assets; not recommended for model training |
| Login / console / high-conversion form pages | Not needed | Strictly restrict | Strictly restrict | The core here is security and the conversion path |
The core behind this table isn't technical—it's business.
Describe your idea once, and We0 AI can generate a showcase site, pages, and CMS, then help you attract customers and traffic after launch.
One complete project generation for free registration
Best for trying one complete generation flow and seeing a first project draft quickly.
Whether a site needs to be open doesn't depend on "Are you embracing AI?"
It depends on:
Does this type of access ultimately help you grow, or does it weaken your growth?
Because what We0 AI sells isn't just "page generation."
If it were just a simple one-click landing page generator, having some content taken might not be a huge issue.
But We0 AI's positioning is more like:
A showcase website growth platform + building capability + content capability + SEO/GEO capability + continuous optimization and lead generation path.
This means many pages on the site are not just information pages—they are nodes in the growth chain.
For example:
If you hand over all these pages with the same key to every AI crawler, problems arise:
This is a very real anxiety for many content sites.
AI citing you doesn't mean it will send traffic back to you.
Being mentioned ≠ Being visited.
Being visited ≠ Conversion.
And what We0 AI needs isn't vanity exposure numbers—it's leads and registrations that translate into business.
A good case study page, a good comparison page, a good industry insight page—these can build brand equity over the long term.
But if they're constantly crawled, summarized, and re-expressed, the differentiation that resides on your site will become weaker and weaker.
These two things are not synonyms.
A truly AI-friendly site isn't one that opens up mindlessly—it's one that:
Clear structure
Clarity is more important than distribution.
If I were approaching this from the perspective of We0 AI's growth and content operations, here's how I would configure it:
The goal is clear:
First, ensure that brand, product, scenario, and knowledge content can be discovered by search engines and AI discovery systems.
Key areas to focus on include:
At the same time, track one key metric:
Not "whether it was crawled," but "whether it has generated observable return traffic and improved brand search."
Agents are most worth testing but also require the most boundaries.
Recommended to open first:
Recommended to restrict:
In a nutshell:
Let agents see what they should see, but don't let them complete the entire funnel for you.
Unless you are very clear about wanting to expand brand diffusion, it is not recommended to open your entire site to Training by default.
Especially these types of content:
These pages should be more carefully protected.
Cloudflare's categorization has truly highlighted one key point:
Website operations are no longer just about SEO. You now also need to design for AI access.
This means you need to answer three questions:
Many teams are still stuck in the "just build the site" phase.
But moving forward, the real differentiator will be "building the site into a controllable growth system."
This is also where We0 AI becomes more valuable.
It's not just about helping you launch a website.
More importantly, it should help you turn your website into a long-term growth asset:
This is the real challenge of building websites today.
It's not that building a site has become harder.
What has become harder is how to keep growing the site after it's live.
Now that Cloudflare has classified AI crawlers into Search, Agent, and Training, website strategies should no longer be "all open or all closed."
For display-oriented website growth platforms like We0 AI, a more reasonable approach is: open Search, cautiously open Agent, and restrict Training by default.
What truly needs optimization is not the on/off switch for individual bots, but the overall AI access boundaries and growth path of the entire website.
The biggest change is that website owners no longer have to treat all AI traffic as the same. Search may bring discovery and return traffic, Agents may bring real-time task access, and Training is more about content absorption. Strategies can now be layered.
Not necessarily, but most websites that rely on exposure and discovery should generally prioritize opening Search. This assumes you want to be seen by search and AI discovery systems.
Generally yes, but it also depends on the page type. Agents may bring new entry points, but they may also bypass your conversion path. Therefore, conditional access should be based on the page and business process.
Because it typically brings the weakest direct return traffic, but the deepest content absorption. For websites that rely on original content, case studies, and knowledge assets, the opportunity cost of Training is higher.
A "layered access" strategy rather than "entire site open": prioritize public content for Search, grant verified public information to Agents, and restrict Training by default.
So, back to the original question:
Now that Cloudflare has classified AI crawlers into Search, Agent, and Training, should the We0 AI website open everything?
My answer remains the same:
No, it shouldn't open everything.
But the point isn't about defense.
The key point is that websites can now move from "passively being crawled" to "actively setting rules."
In the short term, this looks like bot management.
In the long term, it's actually part of a website's growth strategy.
And the truly stronger websites of the future won't be the most open ones.
They will be the websites that know exactly who to open to, why to open, and to what extent.
Start from one sentence and have a complete website in minutes.
