06-reference/research

sc raydata co crawler audit

2026-07-16·research-brief·source: deep-research·by Ray Data Co (deep-research synthesis)
geoai-crawlerssanity-checksc-raydata-corobots-txtcloudflare-wafllms-txt

sc.raydata.co crawler audit — the GEO stack is built and the WAF 403s every crawler it was built for

The question

"Same crawler audit against sc.raydata.co — different Cloudflare Pages project, possibly different defaults, and the surface that should most aggressively want LLM citation."

Context: GEO Step 0 for the Sanity Check newsletter surface, mirroring the completed raydata.co check. Outstanding since the May 9 brief.

What we already know (from the vault)

What the web says

Empirical results observed 2026-07-16 ~05:04 UTC. All via curl against live endpoints.

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

User-agent: *
Allow: /

Sitemap: https://sc.raydata.co/sitemap-index.xml
User-agent HTTP Named in robots.txt?
control browser (Chrome) 200 —
Googlebot 200 no
Google-Extended 200 yes — Disallow: /
Applebot-Extended 200 yes — Disallow: /
GPTBot 403 yes
ClaudeBot 403 yes
CCBot 403 yes
Bytespider 403 yes
anthropic-ai 403 no
PerplexityBot 403 no
OAI-SearchBot 403 no
ChatGPT-User 403 no
Claude-User 403 no
Claude-SearchBot 403 no

The 403 body is 25 bytes of text/plain: Your request was blocked. Headers on the block: cache-control: private, max-age=0, no-store..., x-frame-options: SAMEORIGIN, server: cloudflare. No cf-mitigated header and no challenge headers are present.

Convergences and contradictions

Synthesis for RDCO

Sanity Check is blocking the exact crawlers it most wants citation from, and it is doing so at the WAF — one layer below the robots.txt everyone has been reading. This is the blunt version: someone built the full GEO stack on sc.raydata.co. There is a hand-written llms.txt that says "Quoting with attribution and a link is welcome and encouraged." There are canonical .md endpoints serving text/markdown for every issue. There is a sitemap-index, correctly referenced from robots.txt. The Publishing-for-Agents spec was not a proposal that died in the vault — it shipped. And every single one of those artifacts returns 403 to GPTBot, ChatGPT-User, Claude-User, Claude-SearchBot, OAI-SearchBot, and PerplexityBot. The invitation is written, published, and unreachable by its addressee. That is the whole finding.

The distinction that matters, and that the prior briefs blurred: robots.txt is not the binding constraint here — the WAF is. Every prior brief in this thread framed the problem as "Cloudflare's managed robots.txt disallows ten AI bots" and teed up a robots.txt policy decision. But robots.txt is a voluntary protocol, and the observable proof that it is not what's stopping anyone is sitting right in the matrix above: Google-Extended is Disallow: / in robots.txt and still gets a 200. It ignores the file. Conversely PerplexityBot, OAI-SearchBot, ChatGPT-User, Claude-User and Claude-SearchBot are named in no disallow group anywhere in that file and still get a 403. Rewriting robots.txt to "allow search but not training" — the May 9 brief's recommended path — would have changed nothing observable. The 403s would have continued. That recommendation would have been executed, verified against the wrong artifact, and declared a success while the surface stayed dark.

The critical asymmetry for a newsletter with zero ads and no paywall: the blocked set is precisely the citation path, not the training path. GPTBot and CCBot are training scrapers — blocking those is a defensible, deliberate stance, and ai-train=no says RDCO means it. But ChatGPT-User and Claude-User are user-initiated retrievals: a human asked a question, the assistant went to fetch this page for them, and RDCO returned "Your request was blocked." OAI-SearchBot, Claude-SearchBot and PerplexityBot are the index-builders behind AI search answers — the mechanism by which Sanity Check gets cited rather than consumed. Per the parent brief's own market data, PerplexityBot is the only AI crawler with a favorable crawl-to-referral ratio, and it is 403'd. Sanity Check is paying the full strategic cost of being invisible in AI answers while capturing none of the benefit of blocking training, because the training crawlers that actually respect robots.txt would have honored ai-train=no for free.

The fix, and its shape. This is one zone-level change on raydata.co, not two per-project changes — the DNS evidence settles that, and it means fixing Sanity Check fixes the umbrella site in the same action. The move is: disable the WAF-level AI-bot block (or scope it to the training UAs only), keep the managed robots.txt ai-train=no signal, and add an explicit allow for OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Two hard caveats before anyone touches a dashboard. First, I cannot see the dashboard from outside and did not try — the plain 25-byte Your request was blocked. with no cf-mitigated header is consistent with a WAF custom rule with a custom response, and is not the signature I would expect from the managed "Block AI bots" toggle. Whether this is ai_bots_protection, a Bot Fight Mode side-effect, or a hand-written custom rule is UNVERIFIED; the confirming check is Security → WAF → Custom rules plus Security → Bots on the raydata.co zone, which needs the broken 1Password credential restored first. Do not assume the parent brief's ai_bots_protection attribution is right — it was written before this evidence existed. Second, the September 15 2026 Cloudflare default shift lands on free-tier zones automatically; if raydata.co is free-tier, posture may move again on its own, and any fix applied now should be re-verified after that date with this exact UA matrix. Until the credential is restored, the honest verdict on why the 403 fires is unknown — but the 403 itself is not in doubt, and neither is what it costs.

Why this is in the vault

This is the empirical Step 0 that unblocks (or kills) the GEO bet for Sanity Check, and it materially reframes the decision the May 9 and May 18 briefs teed up: the pending founder decision was scoped as a robots.txt policy question, and this audit shows robots.txt is not the binding layer — so executing the previously-recommended option 2 would have produced zero observable change. It also converts the raydata.co-vs-sc.raydata.co question from "audit two surfaces" into "one zone-level fix," which changes the cost estimate on the fix itself.

Open follow-ups

Related

Sources

Empirical (primary, all run 2026-07-16 ~05:04–05:06 UTC):

Vault:

Web: