Our own audit reported a client site as unreachable by AI crawlers. It was not. Cloudflare's Bot Fight Mode was challenging our crawler, while 836 verified crawler requests passed untouched in the same 24 hours. Here is how to tell the difference on your own site.
On 6 October 2026 we traced a client website that our own AI visibility audit was reporting as unreachable by AI crawlers. The site was blocking nothing. Cloudflare's Bot Fight Mode was challenging our auditor, while the same zone passed 836 verified crawler requests untouched in the preceding 24 hours. The audit was measuring its own obstruction and publishing it as the client's problem.
What happened
Our AI visibility audit fetches a site's homepage, reads the HTML and scores what it finds. On this client it came back with nothing readable, so the report said the homepage could not be reached by AI crawlers.
That reading was wrong. The homepage was fine. Every request we sent, whatever user agent it carried, returned HTTP 403 with the header cf-mitigated: challenge and a page titled "Just a moment...". That is Cloudflare's managed challenge, served at the edge before the request reaches the origin.
Two details gave it away. Requests to /robots.txt from the same address returned 200 while the homepage and the XML sitemap did not. And a third-party scanner reading the same site from its own infrastructure saw everything without difficulty.
The audit had measured its own obstruction. It then published that as a finding about the client's website.
Why "Allow" does not mean allowed
The zone's AI bot policies were all set to Allow (do not block): Search, Agent and Training. Anyone checking whether that site welcomed AI crawlers would have looked at exactly that panel and concluded it did.
Bot Fight Mode sits underneath those policies in the same Cloudflare screen and operates independently of them. It challenges traffic it classifies as automated. Turning the AI bot policies to Allow does not exempt anything from it.
The distinction that matters is verification, not user agent. Cloudflare decides whether a request comes from a real crawler by checking the source IP and reverse DNS against each operator's published ranges. It does not take the user agent's word for it, and nor should it, because a user agent string is a line of text anyone can send.
So a request carrying ClaudeBot or GPTBot from a data centre IP is not a verified bot. It is an unverified client claiming to be one, and Bot Fight Mode treats it accordingly. Our auditor runs on cloud infrastructure, so that is exactly what it looked like.
This is the trap. The settings page reports your intent. Bot Fight Mode acts on something else entirely, and nothing on that page tells you the two can disagree.
The test
We ran the same request three times from the same egress IP, changing one setting between runs. Nothing else was altered on the zone or in our code.
| Time (UTC) | Bot Fight Mode | Response to ClaudeBot | Response to GPTBot |
| --- | --- | --- | --- |
| 11:49 | On | 403, cf-mitigated: challenge | 403, cf-mitigated: challenge |
| 12:34 | Off | 200, 214,659 bytes of HTML | 200, 216,970 bytes of HTML |
| 12:58 | On | 403, cf-mitigated: challenge | 403, cf-mitigated: challenge |
At 12:34 we also requested the site as Googlebot, bingbot and PerplexityBot. All three returned 200 with the full page, and the XML sitemap returned 200 as valid markup. At 12:58, with Bot Fight Mode back on, Googlebot and PerplexityBot were challenged alongside the rest.
Turning the setting off fixed it, turning it back on broke it again, and the address we called from never changed. That sequence establishes cause rather than coincidence.
Real crawlers were never affected
This is the part that matters most, and it is the part an audit built on a blocked fetch gets backwards.
We exported the zone's security events for the three verified AI bot categories, covering 05 October 12:43 UTC to 06 October 12:39 UTC. That window sits entirely inside the period Bot Fight Mode was switched on.
849 events. 836 skipped, 13 allowed, none challenged and none blocked.
The named crawlers in that set:
| Crawler | Requests | Action | | --- | --- | --- | | GoogleOther | 191 | skip | | Google-Agent | 141 | skip | | Applebot | 139 | skip | | Amazonbot | 40 | skip | | ClaudeBot | 25 | skip | | ChatGPT-User | 23 | skip | | Claude-User | 11 | skip | | DuckAssistBot | 10 | skip | | meta-externalagent | 7 | skip | | GPTBot | 6 | skip |
Those account for 593 of the 849. The remaining 256 carried ordinary browser user agent strings that Cloudflare had still classified as verified bots, which is common for assistants fetching a page on a person's behalf.
Why were they spared? Not because of a rule the site owner wrote. Cloudflare states that verified bots are excluded from default bot configurations across all plans, and Bot Fight Mode is one of those. Verification itself is the exemption.
This is worth being precise about, because the zone did also carry a custom rule reading Known Bots equals true with the action Skip, and all 836 skips matched it. It is tempting to credit that rule. You should not, and Cloudflare is blunt about why: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules". Skip, Bypass and Allow actions have no effect on it, because it runs outside the Ruleset Engine.
So the site was visible to AI crawlers the entire time. The only traffic Bot Fight Mode stopped was unverified, and that category included the tool hired to measure the site.
How to check your own site
If you are on Cloudflare, this takes about five minutes.
- Find Bot Fight Mode. Security, then Bots. On the free plan it is a single toggle and it is easy to forget it is on. It does not appear among your WAF custom rules, which is why "all my rules are set to allow" can be true and beside the point.
- Do not expect a custom rule to help. Cloudflare states plainly that Bot Fight Mode cannot be bypassed with WAF custom rules or Page Rules, because it runs outside the Ruleset Engine. A Skip rule for a tool you want to allow will not work while Bot Fight Mode is on.
- Look at what was actually challenged. Security, then Analytics, then Events. Filter Action equals Managed Challenge over the last 24 hours and add the Verified Bot Category column. Any verified crawler in that list is a visibility problem you can act on.
- Filter one category at a time. If you filter Verified Bot Category for AI Crawler and AI Search and AI Assistant together under Match all, you will always get zero results, because a request has one category. That empty screen looks like a clean bill of health and is not one.
- Check the AI bot policies separately. Search, Agent and Training each have their own setting. Set them deliberately. Just do not read them as proof that crawlers are getting through.
If you are not on Cloudflare, the same question applies to whatever sits in front of your origin. Ask what it does with automated traffic, and ask it separately from whatever your crawler settings say.
What this means if you are buying an AI visibility audit
Almost every AI visibility or GEO tool on the market fetches your pages from cloud infrastructure. That means almost every one of them can be classified as an unverified bot by your own protection, and can report a crawl failure that says nothing about whether real crawlers reach you.
The failure is quiet and it points the wrong way. A tool that cannot read your homepage has two honest options: say so, or say nothing. What it usually does instead is score what it managed to retrieve, which may be a challenge page, and present the result as a finding about your site. A score of zero for structured data reads identically whether you have no structured data or the tool never saw your HTML.
Four questions worth asking any provider:
- When your crawler cannot read a page, does the report say so plainly, or does it score it anyway?
- Does the tool identify itself honestly, or does it send a well-known crawler's user agent?
- Can you show me the HTTP status and headers behind a reported crawl failure?
- Have you checked whether my own bot protection challenged you, before telling me my site blocks AI?
We got this wrong before we got it right. Our audit reported a client's site as unreachable by AI crawlers and, in an earlier run, scored a challenge page as though it were the homepage. Three things follow from that, and they are changing how we work.
First, a score we cannot support does not get published. If we could not read the page, the report says so and leaves the number empty rather than filling it with a zero that reads like a finding.
Second, the question "can AI crawlers reach this site?" should not be answered by simulation at all. The site's own CDN and server logs already record which crawlers arrived and what happened to them. That is ground truth, and it beats any attempt to impersonate a crawler from the outside.
Third, a tool that wants to be let through should earn it rather than disguise itself. Cloudflare runs an application process for its bots and agents directory, and the bar is honest self-identification, obeying robots.txt and crawl directives, and reasonable request rates. That is the right direction of travel for any serious auditor, and it is the opposite of sending someone else's user agent and hoping.
Method and limitations
Method. Requests were issued from a serverless function with a fixed egress IP, using a 15 to 20 second timeout. The Accept and Accept-Language headers were held constant across every run and only the User-Agent varied. We recorded HTTP status, response byte count and selected response headers, and identified a challenge by the presence of the cf-mitigated header together with the page title "Just a moment...". Security events were exported as JSON from the Cloudflare dashboard, filtered by Verified Bot Category, then parsed and counted. The site is not named because its security configuration is the client's to disclose, not ours.
What this does not establish. Several things, and they matter if you want to cite this.
- One zone, one site, one day, on Cloudflare's free plan. This is a worked case, not a survey, and we make no claim about how common the configuration is.
- Cloudflare's security event export states that Cloudflare Workers subrequests are excluded. Our own challenged requests therefore do not appear in the event data we analysed. Their absence from those logs is a property of the export, not evidence they were not challenged, and we hold the
cf-mitigatedresponses separately. - Verified bot classification is Cloudflare's own determination. We did not independently confirm the identity of the crawlers in the event data.
- A separate HTTP 403 was returned at the origin to browser user agents arriving from data centre addresses, unaffected by Bot Fight Mode. We did not characterise it further, since it did not bear on the question.
- We did not test Super Bot Fight Mode, available on paid plans, which exposes controls the free version does not.
Reproducing it. Send the same request to your own site twice, once with a browser user agent and once with a crawler's, from a cloud host rather than your laptop, and compare the status codes and the cf-mitigated header. A laptop on a home connection will often pass where a data centre address does not, which is precisely why testing from your desk can mislead you.
Sources
Primary evidence is our own: HTTP responses recorded from a fixed egress IP at the times given, and the client's Cloudflare security event export for the stated window.
Cloudflare documentation referenced, both read on 6 October 2026:
- Bot Fight Mode, for the statement that it cannot be bypassed or skipped using WAF custom rules or Page Rules, and that Skip, Bypass and Allow actions have no effect on it.
- Verified bots, for the exclusion of verified bots from default bot configurations across all plans, and for the application process and its requirements.