# Sitemap & Robots.txt Setup Guide > **SEO Foundation**: Automated sitemap generation and search engine crawl directives for WorkRoot IT Solutions --- ## Overview | File | Purpose | Location | |------|---------|----------| | **sitemap.xml** | Dynamic XML sitemap | `src/pages/sitemap.xml.ts` | | **sitemap-index.xml** | Sitemap index (scalability) | `src/pages/sitemap-index.xml.ts` | | **robots.txt** | Crawl directives | `public/robots.txt` | **Site URL**: `https://workroot.in` --- ## Sitemap Implementation ### Current Coverage | Content Type | Included | Priority | Change Frequency | |--------------|----------|----------|------------------| | Homepage | ✅ | 1.0 | Weekly | | Services | ✅ | 0.9 | Monthly | | Portfolio Index | ✅ | 0.8 | Weekly | | About | ✅ | 0.8 | Monthly | | Blog Index | ✅ | 0.8 | Weekly | | Blog Posts | ✅ | 0.7 | Monthly | | Portfolio Items | ✅ | 0.7 | Monthly | | Contact | ✅ | 0.7 | Monthly | | Legal Pages | ✅ | 0.3 | Yearly | ### Features - ✅ **Dynamic generation** - Automatically includes all published blog posts - ✅ **Draft filtering** - Excludes draft content from sitemap - ✅ **Portfolio support** - Ready for portfolio content collection - ✅ **Last modified dates** - Uses `updatedDate` or `pubDate` from frontmatter - ✅ **Proper namespaces** - Includes news, image, video schemas for future use - ✅ **Caching headers** - 1-hour cache for performance - ✅ **XML formatting** - Valid sitemap protocol --- ## Robots.txt Configuration ### Key Directives ``` User-agent: * Allow: / # Block private areas Disallow: /admin/ Disallow: /private/ Disallow: /_astro/ # Sitemap location Sitemap: https://workroot.in/sitemap.xml ``` ### AI Crawler Support (GEO) The robots.txt explicitly allows **AI search engines** for Generative Engine Optimization: | Bot | Purpose | Allowed | |-----|---------|---------| | GPTBot | ChatGPT training/search | ✅ | | ChatGPT-User | ChatGPT browsing | ✅ | | anthropic-ai | Claude AI training | ✅ | | Claude-Web | Claude search | ✅ | | PerplexityBot | Perplexity AI | ✅ | | Google-Extended | Gemini/Bard | ✅ | | CCBot | Common Crawl (many AIs) | ✅ | | FacebookBot | Meta AI | ✅ | **Why?** AI search engines can cite your content in responses, increasing brand visibility. --- ## Search Engine Submission ### 1. Google Search Console **URL**: https://search.google.com/search-console #### Steps: 1. **Verify ownership**: - Add `google-site-verification` meta tag to `` - Or upload HTML file to `public/` - Or use DNS TXT record 2. **Submit sitemap**: ``` Property: https://workroot.in Sitemaps → Add new sitemap URL: https://workroot.in/sitemap.xml ``` 3. **Monitor**: - Coverage report (indexed pages) - Enhancement reports (Core Web Vitals) - Performance (search analytics) #### Verification Tag Add to `src/components/SEO.astro`: ```html ``` --- ### 2. Bing Webmaster Tools **URL**: https://www.bing.com/webmasters #### Steps: 1. **Import from Google** (easiest): - Use same Google Search Console account - One-click import 2. **Or verify manually**: - Add `` - Or upload XML file 3. **Submit sitemap**: ``` Sitemaps → Submit sitemap URL: https://workroot.in/sitemap.xml ``` --- ### 3. Yandex Webmaster **URL**: https://webmaster.yandex.com #### Steps: 1. **Add site**: `https://workroot.in` 2. **Verify**: Upload HTML file or add meta tag 3. **Submit sitemap**: ``` Indexing → Sitemap files https://workroot.in/sitemap.xml ``` --- ### 4. IndexNow (Instant Indexing) **What**: Real-time indexing API for Bing, Yandex, and others **URL**: https://www.indexnow.org #### Implementation: ```bash # Generate API key openssl rand -hex 32 > public/[KEY].txt # Submit URLs on publish/update curl -X POST "https://api.indexnow.org/indexnow" \ -H "Content-Type: application/json" \ -d '{ "host": "workroot.in", "key": "YOUR_API_KEY", "urlList": [ "https://workroot.in/blog/new-post/" ] }' ``` **When to use**: After publishing new blog posts or major updates --- ## Sitemap Automation ### Build-Time Generation Sitemap is automatically generated on every build: ```bash npm run build # Sitemap available at: dist/client/sitemap.xml ``` ### Post-Publish Hook (Recommended) Create `.github/workflows/sitemap-ping.yml`: ```yaml name: Ping Search Engines on: push: branches: [main] paths: - 'src/content/blog/**' - 'src/pages/**' jobs: ping: runs-on: ubuntu-latest steps: - name: Ping Google run: | curl "https://www.google.com/ping?sitemap=https://workroot.in/sitemap.xml" - name: Ping Bing run: | curl "https://www.bing.com/ping?sitemap=https://workroot.in/sitemap.xml" ``` --- ## Testing & Validation ### 1. XML Validation **Test locally**: ```bash # Check XML is valid curl http://localhost:10000/sitemap.xml | xmllint --noout - # View in browser open http://localhost:10000/sitemap.xml ``` **Online validators**: - https://www.xml-sitemaps.com/validate-xml-sitemap.html - https://validator.w3.org/feed/ ### 2. Google Sitemap Tester ``` Google Search Console → Sitemaps → Test sitemap ``` ### 3. Check Coverage **Verify all important pages are included**: ```bash curl https://workroot.in/sitemap.xml | grep -o '[^<]*' ``` **Expected count**: - Static pages: 9 - Blog posts: 3+ (grows with content) - Portfolio items: 0+ (when added) --- ## Troubleshooting | Issue | Solution | |-------|----------| | Sitemap returns 404 | Check `src/pages/sitemap.xml.ts` exists, rebuild | | Empty sitemap | Check blog posts aren't all `draft: true` | | Search Console errors | Use Sitemap Tester, check XML validity | | Pages not indexed | Check robots.txt doesn't block, verify HTTPS | | Slow indexing | Submit via IndexNow, ping search engines | --- ## Maintenance Checklist - [ ] **Monthly**: Check Search Console coverage report - [ ] **When publishing**: Ping search engines (manual or automated) - [ ] **Quarterly**: Validate sitemap XML structure - [ ] **When adding collections**: Update `sitemap.xml.ts` to include new content types - [ ] **Annually**: Review robots.txt directives, update AI bot list --- ## Adding New Content Types ### Example: Portfolio Collection 1. **Create collection** (`src/content/config.ts`): ```typescript const portfolio = defineCollection({ type: 'content', schema: z.object({ title: z.string(), description: z.string(), date: z.coerce.date(), updatedDate: z.coerce.date().optional(), // ... other fields }), }); export const collections = { blog, portfolio }; ``` 2. **Sitemap auto-includes** (already implemented in `sitemap.xml.ts`): ```typescript // Already done! ✅ let portfolioItems: any[] = []; try { portfolioItems = await getCollection('portfolio'); } catch (e) { // Collection doesn't exist yet } const portfolioPages: SitemapEntry[] = portfolioItems.map((item) => ({ url: `/portfolio/${item.slug}/`, lastmod: item.data.updatedDate?.toISOString().split('T')[0], changefreq: 'monthly', priority: 0.7, })); ``` 3. **Rebuild & verify**: ```bash npm run build curl https://workroot.in/sitemap.xml | grep portfolio ``` --- ## Performance Optimizations ### Current Implementation | Optimization | Status | |--------------|--------| | Cache-Control header | ✅ 1 hour | | Gzip compression | ✅ Via Nginx/CDN | | XML minification | ✅ No whitespace waste | | Conditional inclusion | ✅ Drafts excluded | | X-Robots-Tag | ✅ Sitemap not indexed | ### Advanced: Sitemap Index **When sitemap exceeds 50,000 URLs**: Use `sitemap-index.xml.ts` to split into multiple sitemaps: - `sitemap-blog.xml` - Blog posts - `sitemap-portfolio.xml` - Portfolio - `sitemap-static.xml` - Static pages Currently not needed (< 100 URLs), but implementation ready. --- ## AI Search Visibility (GEO) ### Why Allow AI Crawlers? | Benefit | Impact | |---------|--------| | **Brand mentions** | Cited in AI responses | | **Authority** | Recognized as source | | **Traffic** | Users click through from AI | | **Future-proof** | AI search is growing | ### Monitoring AI Citations **Track manually**: - Search "[your topic]" in ChatGPT, Claude, Perplexity - Check if workroot.in is cited **Tools** (emerging): - https://citationtracker.ai (concept) - Monitor referrer traffic from AI platforms --- ## Resources | Resource | URL | |----------|-----| | Sitemap Protocol | https://www.sitemaps.org/protocol.html | | Google Sitemap Guide | https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview | | Robots.txt Spec | https://www.robotstxt.org/ | | IndexNow Docs | https://www.indexnow.org/documentation | | AI Crawler List | https://darkvisitors.com/ | --- ## Summary ✅ **Automated sitemap** - Dynamically includes all blog posts ✅ **Portfolio-ready** - Supports future portfolio collection ✅ **AI-friendly robots.txt** - Allows all major AI crawlers ✅ **SEO best practices** - Proper priorities, change frequencies ✅ **Scalable architecture** - Sitemap index ready for growth **Next steps**: 1. Submit sitemap to Google Search Console 2. Submit sitemap to Bing Webmaster Tools 3. Set up IndexNow for instant indexing 4. Monitor coverage reports monthly --- **Generated**: 2026-03-21 **Domain**: workroot.in **Agent**: seo-specialist