06-reference/research

raydata co crawler status 9day check

2026-05-18·research-brief·source: deep-research
geoai-crawlersraydata-cocloudflare-analyticsrobots-txtgptbotclaudebotperplexitybotdelta-check

raydata.co AI crawler status — 9-day delta check

The question

What do GPTBot, ClaudeBot, and PerplexityBot logs actually show for raydata.co — is the site being crawled, at what frequency, which pages are being indexed, and are there any robots.txt or Cloudflare WAF rules blocking AI crawlers?

Headline finding

Nothing has changed since the 2026-05-09 brief. raydata.co still serves Cloudflare's managed Disallow: / robots.txt blocking ten AI crawlers (GPTBot, ClaudeBot, Bytespider, CCBot, Amazonbot, Applebot-Extended, Google-Extended, meta-externalagent, CloudflareBrowserRenderingCrawler, plus the wildcard Content-Signal: search=yes,ai-train=no). The body returned to a GET /robots.txt with User-Agent: GPTBot on 2026-05-18 05:36 UTC is byte-identical to the version captured on 2026-05-09. The robots.txt was last modified 2026-04-24 23:00 UTC — i.e., not touched since well before the prior brief.

The May 9 brief stands as the canonical empirical record. No founder decision has been made on the policy question (keep blocked / opt in to search-time crawl but not training / opt in fully), and no configuration change has been applied. The GEO bet remains blocked at Step 0.

What we already know (from the vault)

What the web says (industry context, 2026)

Empirical re-check — 2026-05-18

Check Result
GET /robots.txt as User-Agent: GPTBot 200 OK, returns Cloudflare-managed body with Disallow: / for the same 10 AI bots as 2026-05-09
last-modified header on robots.txt Fri, 24 Apr 2026 23:00:43 GMT (unchanged since pre-May-9 brief)
Cloudflare bot_management API Could not query (1Password lookup for CloudFlare - claude-rdco failed; wrapper script intact, item access broken)
Content-Signal directive in body search=yes,ai-train=no (unchanged)

I attempted to refresh the live Cloudflare GraphQL Analytics numbers but the 1Password lookup for the Cloudflare API token failed (Ray Agent vault, CloudFlare - claude-rdco item). The May 9 brief's day-level hit table is the most recent empirical signal; given the config is unchanged, the qualitative pattern (zero GPTBot/PerplexityBot/Google-Extended, ClaudeBot probing robots.txt then bouncing, ChatGPT-User as the only content-reaching crawler) will hold.

Convergences and contradictions

Convergences — none new since May 9.

Contradictions / surprises — none. The site config is frozen. The decision point flagged in the May 9 brief has not been resolved.

Synthesis for RDCO

This brief exists because the user explicitly asked the deep-research run to dequeue this question, even though the canonical brief already exists at 2026-05-09-raydata-co-bot-crawler-logs.md. The empirical answer in 9 days is "no material change." The actionable lever is not more research — it is the founder decision the May 9 brief teed up:

  1. Keep current config (Cloudflare managed disallow + WAF ai_bots_protection blocking) → accept GEO bet is dead and act accordingly (kill the work queue items, redirect publishing energy elsewhere).
  2. Opt in to search but not training (disable ai_bots_protection, hand-write robots.txt that allows OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Claude-SearchBot, Googlebot, Bingbot; explicit disallow for GPTBot, CCBot, Bytespider, Amazonbot) → unblocks the GEO thesis cleanly without giving training data away for free. This is the May 9 brief's recommended path.
  3. Opt in fully → treat raydata.co content as deliberately given to the commons.

The May 9 brief also flagged a reversible, no-policy-required fix that's still outstanding: the /p/* → /articles/* 301 redirects. ChatGPT-User retrievals are landing on 404s because of the unmigrated paths. This is rescuing real user traffic, not enabling AI training, and should be applied regardless of which policy direction the founder picks. It belongs on the task board as a discrete, reversible engineering task — not gated on the bigger policy decision.

Net for the deep-research log: the May 9 brief is still load-bearing. This delta-check confirms it. The next time this question deserves a fresh brief is after the founder picks a policy and a config change is made — at which point a 30-day post-change audit will show whether opting in actually attracted crawlers.

Open follow-ups

Sources