chore(seo): open robots.txt to all major AI crawlers for GEO
E2E Test Suite / Cross-Browser Regression (webkit) (push) Failing after 10m48s
E2E Test Suite / Cross-Browser Regression (firefox) (push) Failing after 11m43s
E2E Test Suite / Cross-Browser Regression (chromium) (push) Failing after 12m35s
E2E Test Suite / Destructive & Chaos Tests (push) Failing after 13m27s
E2E Test Suite / Form Interaction Tests (push) Failing after 14m20s
E2E Test Suite / Security Header Tests (push) Failing after 14m1s
E2E Test Suite / Mobile Device Tests (push) Failing after 14m56s
E2E Test Suite / Smoke Tests (P0) (push) Failing after 21m57s
E2E Test Suite / Critical User Journeys (push) Has been skipped
E2E Test Suite / API Integration Tests (push) Has been skipped
Deploy to Production / Build & Verify (push) Failing after 17m16s
E2E Test Suite / Test Report Summary (push) Failing after 13m13s
Deploy to Production / Pre-Deploy Tests (push) Failing after 12m22s
Deploy to Production / Deploy to Fly.io (push) Failing after 13m13s
Deploy to Production / Deploy to VPS (PM2) (push) Failing after 14m5s
Deploy to Production / Deploy to Render (push) Failing after 14m5s
Deploy to Production / Deploy to Railway (push) Failing after 14m57s
Deploy to Production / Post-Deploy Verification (push) Failing after 13h5m7s
Deploy to Production / Notify on Failure (push) Failing after 15m4s
E2E Test Suite / Cross-Browser Regression (webkit) (push) Failing after 10m48s
E2E Test Suite / Cross-Browser Regression (firefox) (push) Failing after 11m43s
E2E Test Suite / Cross-Browser Regression (chromium) (push) Failing after 12m35s
E2E Test Suite / Destructive & Chaos Tests (push) Failing after 13m27s
E2E Test Suite / Form Interaction Tests (push) Failing after 14m20s
E2E Test Suite / Security Header Tests (push) Failing after 14m1s
E2E Test Suite / Mobile Device Tests (push) Failing after 14m56s
E2E Test Suite / Smoke Tests (P0) (push) Failing after 21m57s
E2E Test Suite / Critical User Journeys (push) Has been skipped
E2E Test Suite / API Integration Tests (push) Has been skipped
Deploy to Production / Build & Verify (push) Failing after 17m16s
E2E Test Suite / Test Report Summary (push) Failing after 13m13s
Deploy to Production / Pre-Deploy Tests (push) Failing after 12m22s
Deploy to Production / Deploy to Fly.io (push) Failing after 13m13s
Deploy to Production / Deploy to VPS (PM2) (push) Failing after 14m5s
Deploy to Production / Deploy to Render (push) Failing after 14m5s
Deploy to Production / Deploy to Railway (push) Failing after 14m57s
Deploy to Production / Post-Deploy Verification (push) Failing after 13h5m7s
Deploy to Production / Notify on Failure (push) Failing after 15m4s
Repo robots.txt already matched the deployed file (verified via diff against https://workroot.in/robots.txt). Added commonly-cited AI agents that were still missing, each with Allow: /: - OpenAI: OAI-SearchBot - Anthropic: ClaudeBot, Claude-SearchBot, Claude-User - Perplexity: Perplexity-User - Meta: Meta-ExternalAgent / meta-externalagent - Apple: Applebot, Applebot-Extended - Amazon: Amazonbot - ByteDance: Bytespider - Cohere: cohere-ai - DuckDuckGo: DuckAssistBot - You.com: YouBot - Diffbot Disallow list unchanged (/admin/, /private/, /_astro/) so no content directory needed for citation is blocked. Sitemap line preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
c7e130ce4d
commit
6b07b83583
@@ -58,6 +58,9 @@ Allow: /
|
||||
User-agent: GPTBot
|
||||
Allow: /
|
||||
|
||||
User-agent: OAI-SearchBot
|
||||
Allow: /
|
||||
|
||||
# Anthropic/Claude
|
||||
User-agent: anthropic-ai
|
||||
Allow: /
|
||||
@@ -65,10 +68,22 @@ Allow: /
|
||||
User-agent: Claude-Web
|
||||
Allow: /
|
||||
|
||||
User-agent: ClaudeBot
|
||||
Allow: /
|
||||
|
||||
User-agent: Claude-SearchBot
|
||||
Allow: /
|
||||
|
||||
User-agent: Claude-User
|
||||
Allow: /
|
||||
|
||||
# Perplexity AI
|
||||
User-agent: PerplexityBot
|
||||
Allow: /
|
||||
|
||||
User-agent: Perplexity-User
|
||||
Allow: /
|
||||
|
||||
# Google Gemini/Bard
|
||||
User-agent: Google-Extended
|
||||
Allow: /
|
||||
@@ -80,3 +95,40 @@ Allow: /
|
||||
# Meta AI
|
||||
User-agent: FacebookBot
|
||||
Allow: /
|
||||
|
||||
User-agent: Meta-ExternalAgent
|
||||
Allow: /
|
||||
|
||||
User-agent: meta-externalagent
|
||||
Allow: /
|
||||
|
||||
# Apple Intelligence
|
||||
User-agent: Applebot
|
||||
Allow: /
|
||||
|
||||
User-agent: Applebot-Extended
|
||||
Allow: /
|
||||
|
||||
# Amazon AI
|
||||
User-agent: Amazonbot
|
||||
Allow: /
|
||||
|
||||
# ByteDance / TikTok AI
|
||||
User-agent: Bytespider
|
||||
Allow: /
|
||||
|
||||
# Cohere AI
|
||||
User-agent: cohere-ai
|
||||
Allow: /
|
||||
|
||||
# DuckDuckGo AI (DuckAssist)
|
||||
User-agent: DuckAssistBot
|
||||
Allow: /
|
||||
|
||||
# You.com AI
|
||||
User-agent: YouBot
|
||||
Allow: /
|
||||
|
||||
# Diffbot (knowledge graph / AI training)
|
||||
User-agent: Diffbot
|
||||
Allow: /
|
||||
|
||||
Reference in New Issue
Block a user