chore(seo): open robots.txt to all major AI crawlers for GEO
E2E Test Suite / Cross-Browser Regression (webkit) (push) Failing after 10m48s
E2E Test Suite / Cross-Browser Regression (firefox) (push) Failing after 11m43s
E2E Test Suite / Cross-Browser Regression (chromium) (push) Failing after 12m35s
E2E Test Suite / Destructive & Chaos Tests (push) Failing after 13m27s
E2E Test Suite / Form Interaction Tests (push) Failing after 14m20s
E2E Test Suite / Security Header Tests (push) Failing after 14m1s
E2E Test Suite / Mobile Device Tests (push) Failing after 14m56s
E2E Test Suite / Smoke Tests (P0) (push) Failing after 21m57s
E2E Test Suite / Critical User Journeys (push) Has been skipped
E2E Test Suite / API Integration Tests (push) Has been skipped
Deploy to Production / Build & Verify (push) Failing after 17m16s
E2E Test Suite / Test Report Summary (push) Failing after 13m13s
Deploy to Production / Pre-Deploy Tests (push) Failing after 12m22s
Deploy to Production / Deploy to Fly.io (push) Failing after 13m13s
Deploy to Production / Deploy to VPS (PM2) (push) Failing after 14m5s
Deploy to Production / Deploy to Render (push) Failing after 14m5s
Deploy to Production / Deploy to Railway (push) Failing after 14m57s
Deploy to Production / Post-Deploy Verification (push) Failing after 13h5m7s
Deploy to Production / Notify on Failure (push) Failing after 15m4s

Repo robots.txt already matched the deployed file (verified via diff
against https://workroot.in/robots.txt). Added commonly-cited AI agents
that were still missing, each with Allow: /:
- OpenAI: OAI-SearchBot
- Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
- Perplexity: Perplexity-User
- Meta: Meta-ExternalAgent / meta-externalagent
- Apple: Applebot, Applebot-Extended
- Amazon: Amazonbot
- ByteDance: Bytespider
- Cohere: cohere-ai
- DuckDuckGo: DuckAssistBot
- You.com: YouBot
- Diffbot

Disallow list unchanged (/admin/, /private/, /_astro/) so no content
directory needed for citation is blocked. Sitemap line preserved.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
platform-mcp
2026-06-19 09:35:47 +05:30
co-authored by Claude Opus 4.8
parent c7e130ce4d
commit 6b07b83583
+52
View File
@@ -58,6 +58,9 @@ Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
# Anthropic/Claude
User-agent: anthropic-ai
Allow: /
@@ -65,10 +68,22 @@ Allow: /
User-agent: Claude-Web
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Perplexity AI
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Google Gemini/Bard
User-agent: Google-Extended
Allow: /
@@ -80,3 +95,40 @@ Allow: /
# Meta AI
User-agent: FacebookBot
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
User-agent: meta-externalagent
Allow: /
# Apple Intelligence
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /
# Amazon AI
User-agent: Amazonbot
Allow: /
# ByteDance / TikTok AI
User-agent: Bytespider
Allow: /
# Cohere AI
User-agent: cohere-ai
Allow: /
# DuckDuckGo AI (DuckAssist)
User-agent: DuckAssistBot
Allow: /
# You.com AI
User-agent: YouBot
Allow: /
# Diffbot (knowledge graph / AI training)
User-agent: Diffbot
Allow: /