Cloudflare and OpenAI say their research pilot helped OpenAI find and index hundreds of millions of pages that its crawler had missed. None of that changes what your robots.txt allows. If OAI-SearchBot is blocked, the ChatGPT Search crawler still cannot read the page, and the pilot only decides which pages get looked at in the first place.
What the pilot actually did
Cloudflare and OpenAI ran the pilot on about 1% of Cloudflare’s network and published the results on 15 October 2026. Cloudflare’s network signals told OpenAI’s crawler which pages were worth fetching, including pages that few other sites link to. Neither company reported citation counts or traffic from the new pages.
The examples in the announcement are the useful part. Cloudflare names a washing machine user guide, the latest passport requirements and fees, and a restaurant’s menu and opening hours. Those pages “can be valuable even if few other sites link to them”, according to the post. That is a fair description of most local business sites and most product documentation, and it is exactly the kind of page that link-based ranking tends to ignore.
Cloudflare announced the pilot on 8 July 2026. The signals include how recently a page was updated, the quality of traffic a site receives, and what actually changed on a page. All of it comes from participating websites, and neither announcement says which sites those were.
The number in the headline is not traffic
Hundreds of millions of pages found is a discovery figure. It says nothing about how often those pages get quoted in an answer or clicked by a reader. The post admits the first effect may be negative for site owners: “Finding new pages can initially increase crawling as those pages are evaluated and indexed.” It gives no figures for crawl volume or server load.
So the honest reading is that this changes what enters OpenAI’s index, not what leaves it as a visit. Search Engine Journal reported the pilot on 20 October 2026 and made the same point about the missing citation data. If your analytics have not moved this week, that is the expected result.
Your two crawler settings, and what each one does
OpenAI splits search and training into two separate robots.txt directives. Almost every mistake I see in this area comes from treating them as one switch, so the table below is the one I would keep next to the file.
| Directive | What it controls | What happens if you block it |
|---|---|---|
| OAI-SearchBot | The crawler that gathers pages for ChatGPT’s search results | Your pages drop out of ChatGPT search, though they may still appear as navigational links |
| GPTBot | Content that may be used for model training | No effect on whether ChatGPT Search can find or cite you |
| Both allowed | Crawling for search and training together | OpenAI may use what one crawl collected for both purposes |
Cloudflare is careful about one thing in the announcement, and it is worth repeating word for word. “A signal about a page doesn’t grant permission to access it.” Permission still comes from your robots.txt, your WAF rules and your crawler settings. A network signal is a hint about value. It is not a key.
How to check your own site this week
The pilot is not something you opt into, and there is no dashboard that shows your slice of it. You can still see the effect in four places.
- Cloudflare’s AI Crawl Control. It is available on every Cloudflare plan and lists which AI services are reaching your site, with an allow or block option for each one.
- Your server logs. Grep for OAI-SearchBot and count hits per day. Compare the week before 15 October with the week after, and you will know whether your pages are part of the added crawl.
- Your robots.txt file. Confirm which of the two bots you allow. A block on OAI-SearchBot is the only setting that keeps you out of ChatGPT’s search index.
- Your referrer and UTM data. ChatGPT visits only show up if a referrer or a tag survives the click. Some ChatGPT links carried neither earlier in October, and a fix began rolling out on 15 October.
If you want a second opinion on why a page you wrote never gets quoted, the problem is usually in the page rather than the crawler, and I have written about why blog posts do not get cited in AI search. Blocked crawlers are a real cause. Thin answers are a more common one.
What I would do with the two directives
For a site that publishes content and wants to be found, I would allow OAI-SearchBot and decide GPTBot separately on commercial grounds. Blocking the search bot to protect content is like refusing to be listed in a directory that sends you customers, because the directory also reads your brochure.
For a site that sells the content itself, the calculation changes. A paywalled publisher has a real argument for blocking both and negotiating access, and Cloudflare’s own product exists for that. The mistake is doing it by accident, through a template robots.txt nobody has read since 2023.
What is still unsaid
Two things are missing from the announcement. Nobody will say which sites made up the 1% cohort, so no one can check the results against a known sample. And there is still no date for opening the signals to everyone. Cloudflare said in July that it intends broad access at some point in the remainder of 2026, and the October post repeats that without a date.
The work you can do today is small. Open robots.txt, find the two user agents, and decide what each one is allowed to do. That decision will still be yours after the pilot ends, and it is the only part of this story you control. The primary documents are Cloudflare’s post on the pilot and OpenAI’s crawler documentation.
