First Init

2026-03-21 16:46:46 +05:30
commit d402256547
216 changed files with 48375 additions and 0 deletions
@@ -0,0 +1,401 @@
+# Sitemap & Robots.txt Setup Guide
+
+> **SEO Foundation**: Automated sitemap generation and search engine crawl directives for WorkRoot IT Solutions
+
+---
+
+## Overview
+
+| File | Purpose | Location |
+|------|---------|----------|
+| **sitemap.xml** | Dynamic XML sitemap | `src/pages/sitemap.xml.ts` |
+| **sitemap-index.xml** | Sitemap index (scalability) | `src/pages/sitemap-index.xml.ts` |
+| **robots.txt** | Crawl directives | `public/robots.txt` |
+
+**Site URL**: `https://workroot.in`
+
+---
+
+## Sitemap Implementation
+
+### Current Coverage
+
+| Content Type | Included | Priority | Change Frequency |
+|--------------|----------|----------|------------------|
+| Homepage | ✅ | 1.0 | Weekly |
+| Services | ✅ | 0.9 | Monthly |
+| Portfolio Index | ✅ | 0.8 | Weekly |
+| About | ✅ | 0.8 | Monthly |
+| Blog Index | ✅ | 0.8 | Weekly |
+| Blog Posts | ✅ | 0.7 | Monthly |
+| Portfolio Items | ✅ | 0.7 | Monthly |
+| Contact | ✅ | 0.7 | Monthly |
+| Legal Pages | ✅ | 0.3 | Yearly |
+
+### Features
+
+- ✅ **Dynamic generation** - Automatically includes all published blog posts
+- ✅ **Draft filtering** - Excludes draft content from sitemap
+- ✅ **Portfolio support** - Ready for portfolio content collection
+- ✅ **Last modified dates** - Uses `updatedDate` or `pubDate` from frontmatter
+- ✅ **Proper namespaces** - Includes news, image, video schemas for future use
+- ✅ **Caching headers** - 1-hour cache for performance
+- ✅ **XML formatting** - Valid sitemap protocol
+
+---
+
+## Robots.txt Configuration
+
+### Key Directives
+
+```
+User-agent: *
+Allow: /
+
+# Block private areas
+Disallow: /admin/
+Disallow: /private/
+Disallow: /_astro/
+
+# Sitemap location
+Sitemap: https://workroot.in/sitemap.xml
+```
+
+### AI Crawler Support (GEO)
+
+The robots.txt explicitly allows **AI search engines** for Generative Engine Optimization:
+
+| Bot | Purpose | Allowed |
+|-----|---------|---------|
+| GPTBot | ChatGPT training/search | ✅ |
+| ChatGPT-User | ChatGPT browsing | ✅ |
+| anthropic-ai | Claude AI training | ✅ |
+| Claude-Web | Claude search | ✅ |
+| PerplexityBot | Perplexity AI | ✅ |
+| Google-Extended | Gemini/Bard | ✅ |
+| CCBot | Common Crawl (many AIs) | ✅ |
+| FacebookBot | Meta AI | ✅ |
+
+**Why?** AI search engines can cite your content in responses, increasing brand visibility.
+
+---
+
+## Search Engine Submission
+
+### 1. Google Search Console
+
+**URL**: https://search.google.com/search-console
+
+#### Steps:
+1. **Verify ownership**:
+   - Add `google-site-verification` meta tag to `<head>`
+   - Or upload HTML file to `public/`
+   - Or use DNS TXT record
+
+2. **Submit sitemap**:
+   ```
+   Property: https://workroot.in
+   Sitemaps → Add new sitemap
+   URL: https://workroot.in/sitemap.xml
+   ```
+
+3. **Monitor**:
+   - Coverage report (indexed pages)
+   - Enhancement reports (Core Web Vitals)
+   - Performance (search analytics)
+
+#### Verification Tag
+Add to `src/components/SEO.astro`:
+```html
+<meta name="google-site-verification" content="YOUR_CODE_HERE" />
+```
+
+---
+
+### 2. Bing Webmaster Tools
+
+**URL**: https://www.bing.com/webmasters
+
+#### Steps:
+1. **Import from Google** (easiest):
+   - Use same Google Search Console account
+   - One-click import
+
+2. **Or verify manually**:
+   - Add `<meta name="msvalidate.01" content="..." />`
+   - Or upload XML file
+
+3. **Submit sitemap**:
+   ```
+   Sitemaps → Submit sitemap
+   URL: https://workroot.in/sitemap.xml
+   ```
+
+---
+
+### 3. Yandex Webmaster
+
+**URL**: https://webmaster.yandex.com
+
+#### Steps:
+1. **Add site**: `https://workroot.in`
+2. **Verify**: Upload HTML file or add meta tag
+3. **Submit sitemap**:
+   ```
+   Indexing → Sitemap files
+   https://workroot.in/sitemap.xml
+   ```
+
+---
+
+### 4. IndexNow (Instant Indexing)
+
+**What**: Real-time indexing API for Bing, Yandex, and others
+
+**URL**: https://www.indexnow.org
+
+#### Implementation:
+```bash
+# Generate API key
+openssl rand -hex 32 > public/[KEY].txt
+
+# Submit URLs on publish/update
+curl -X POST "https://api.indexnow.org/indexnow" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "host": "workroot.in",
+    "key": "YOUR_API_KEY",
+    "urlList": [
+      "https://workroot.in/blog/new-post/"
+    ]
+  }'
+```
+
+**When to use**: After publishing new blog posts or major updates
+
+---
+
+## Sitemap Automation
+
+### Build-Time Generation
+
+Sitemap is automatically generated on every build:
+
+```bash
+npm run build
+# Sitemap available at: dist/client/sitemap.xml
+```
+
+### Post-Publish Hook (Recommended)
+
+Create `.github/workflows/sitemap-ping.yml`:
+
+```yaml
+name: Ping Search Engines
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'src/content/blog/**'
+      - 'src/pages/**'
+
+jobs:
+  ping:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Ping Google
+        run: |
+          curl "https://www.google.com/ping?sitemap=https://workroot.in/sitemap.xml"
+
+      - name: Ping Bing
+        run: |
+          curl "https://www.bing.com/ping?sitemap=https://workroot.in/sitemap.xml"
+```
+
+---
+
+## Testing & Validation
+
+### 1. XML Validation
+
+**Test locally**:
+```bash
+# Check XML is valid
+curl http://localhost:10000/sitemap.xml | xmllint --noout -
+
+# View in browser
+open http://localhost:10000/sitemap.xml
+```
+
+**Online validators**:
+- https://www.xml-sitemaps.com/validate-xml-sitemap.html
+- https://validator.w3.org/feed/
+
+### 2. Google Sitemap Tester
+
+```
+Google Search Console → Sitemaps → Test sitemap
+```
+
+### 3. Check Coverage
+
+**Verify all important pages are included**:
+```bash
+curl https://workroot.in/sitemap.xml | grep -o '<loc>[^<]*</loc>'
+```
+
+**Expected count**:
+- Static pages: 9
+- Blog posts: 3+ (grows with content)
+- Portfolio items: 0+ (when added)
+
+---
+
+## Troubleshooting
+
+| Issue | Solution |
+|-------|----------|
+| Sitemap returns 404 | Check `src/pages/sitemap.xml.ts` exists, rebuild |
+| Empty sitemap | Check blog posts aren't all `draft: true` |
+| Search Console errors | Use Sitemap Tester, check XML validity |
+| Pages not indexed | Check robots.txt doesn't block, verify HTTPS |
+| Slow indexing | Submit via IndexNow, ping search engines |
+
+---
+
+## Maintenance Checklist
+
+- [ ] **Monthly**: Check Search Console coverage report
+- [ ] **When publishing**: Ping search engines (manual or automated)
+- [ ] **Quarterly**: Validate sitemap XML structure
+- [ ] **When adding collections**: Update `sitemap.xml.ts` to include new content types
+- [ ] **Annually**: Review robots.txt directives, update AI bot list
+
+---
+
+## Adding New Content Types
+
+### Example: Portfolio Collection
+
+1. **Create collection** (`src/content/config.ts`):
+```typescript
+const portfolio = defineCollection({
+  type: 'content',
+  schema: z.object({
+    title: z.string(),
+    description: z.string(),
+    date: z.coerce.date(),
+    updatedDate: z.coerce.date().optional(),
+    // ... other fields
+  }),
+});
+
+export const collections = { blog, portfolio };
+```
+
+2. **Sitemap auto-includes** (already implemented in `sitemap.xml.ts`):
+```typescript
+// Already done! ✅
+let portfolioItems: any[] = [];
+try {
+  portfolioItems = await getCollection('portfolio');
+} catch (e) {
+  // Collection doesn't exist yet
+}
+
+const portfolioPages: SitemapEntry[] = portfolioItems.map((item) => ({
+  url: `/portfolio/${item.slug}/`,
+  lastmod: item.data.updatedDate?.toISOString().split('T')[0],
+  changefreq: 'monthly',
+  priority: 0.7,
+}));
+```
+
+3. **Rebuild & verify**:
+```bash
+npm run build
+curl https://workroot.in/sitemap.xml | grep portfolio
+```
+
+---
+
+## Performance Optimizations
+
+### Current Implementation
+
+| Optimization | Status |
+|--------------|--------|
+| Cache-Control header | ✅ 1 hour |
+| Gzip compression | ✅ Via Nginx/CDN |
+| XML minification | ✅ No whitespace waste |
+| Conditional inclusion | ✅ Drafts excluded |
+| X-Robots-Tag | ✅ Sitemap not indexed |
+
+### Advanced: Sitemap Index
+
+**When sitemap exceeds 50,000 URLs**:
+
+Use `sitemap-index.xml.ts` to split into multiple sitemaps:
+- `sitemap-blog.xml` - Blog posts
+- `sitemap-portfolio.xml` - Portfolio
+- `sitemap-static.xml` - Static pages
+
+Currently not needed (< 100 URLs), but implementation ready.
+
+---
+
+## AI Search Visibility (GEO)
+
+### Why Allow AI Crawlers?
+
+| Benefit | Impact |
+|---------|--------|
+| **Brand mentions** | Cited in AI responses |
+| **Authority** | Recognized as source |
+| **Traffic** | Users click through from AI |
+| **Future-proof** | AI search is growing |
+
+### Monitoring AI Citations
+
+**Track manually**:
+- Search "[your topic]" in ChatGPT, Claude, Perplexity
+- Check if workroot.in is cited
+
+**Tools** (emerging):
+- https://citationtracker.ai (concept)
+- Monitor referrer traffic from AI platforms
+
+---
+
+## Resources
+
+| Resource | URL |
+|----------|-----|
+| Sitemap Protocol | https://www.sitemaps.org/protocol.html |
+| Google Sitemap Guide | https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview |
+| Robots.txt Spec | https://www.robotstxt.org/ |
+| IndexNow Docs | https://www.indexnow.org/documentation |
+| AI Crawler List | https://darkvisitors.com/ |
+
+---
+
+## Summary
+
+✅ **Automated sitemap** - Dynamically includes all blog posts
+✅ **Portfolio-ready** - Supports future portfolio collection
+✅ **AI-friendly robots.txt** - Allows all major AI crawlers
+✅ **SEO best practices** - Proper priorities, change frequencies
+✅ **Scalable architecture** - Sitemap index ready for growth
+
+**Next steps**:
+1. Submit sitemap to Google Search Console
+2. Submit sitemap to Bing Webmaster Tools
+3. Set up IndexNow for instant indexing
+4. Monitor coverage reports monthly
+
+---
+
+**Generated**: 2026-03-21
+**Domain**: workroot.in
+**Agent**: seo-specialist