# Sitemap & Robots.txt Setup Guide

> **SEO Foundation**: Automated sitemap generation and search engine crawl directives for WorkRoot IT Solutions

---

## Overview

| File | Purpose | Location |
|------|---------|----------|
| **sitemap.xml** | Dynamic XML sitemap | `src/pages/sitemap.xml.ts` |
| **sitemap-index.xml** | Sitemap index (scalability) | `src/pages/sitemap-index.xml.ts` |
| **robots.txt** | Crawl directives | `public/robots.txt` |

**Site URL**: `https://workroot.in`

---

## Sitemap Implementation

### Current Coverage

| Content Type | Included | Priority | Change Frequency |
|--------------|----------|----------|------------------|
| Homepage | ✅ | 1.0 | Weekly |
| Services | ✅ | 0.9 | Monthly |
| Portfolio Index | ✅ | 0.8 | Weekly |
| About | ✅ | 0.8 | Monthly |
| Blog Index | ✅ | 0.8 | Weekly |
| Blog Posts | ✅ | 0.7 | Monthly |
| Portfolio Items | ✅ | 0.7 | Monthly |
| Contact | ✅ | 0.7 | Monthly |
| Legal Pages | ✅ | 0.3 | Yearly |

### Features

- ✅ **Dynamic generation** - Automatically includes all published blog posts
- ✅ **Draft filtering** - Excludes draft content from sitemap
- ✅ **Portfolio support** - Ready for portfolio content collection
- ✅ **Last modified dates** - Uses `updatedDate` or `pubDate` from frontmatter
- ✅ **Proper namespaces** - Includes news, image, video schemas for future use
- ✅ **Caching headers** - 1-hour cache for performance
- ✅ **XML formatting** - Valid sitemap protocol

---

## Robots.txt Configuration

### Key Directives

```
User-agent: *
Allow: /

# Block private areas
Disallow: /admin/
Disallow: /private/
Disallow: /_astro/

# Sitemap location
Sitemap: https://workroot.in/sitemap.xml
```

### AI Crawler Support (GEO)

The robots.txt explicitly allows **AI search engines** for Generative Engine Optimization:

| Bot | Purpose | Allowed |
|-----|---------|---------|
| GPTBot | ChatGPT training/search | ✅ |
| ChatGPT-User | ChatGPT browsing | ✅ |
| anthropic-ai | Claude AI training | ✅ |
| Claude-Web | Claude search | ✅ |
| PerplexityBot | Perplexity AI | ✅ |
| Google-Extended | Gemini/Bard | ✅ |
| CCBot | Common Crawl (many AIs) | ✅ |
| FacebookBot | Meta AI | ✅ |

**Why?** AI search engines can cite your content in responses, increasing brand visibility.

---

## Search Engine Submission

### 1. Google Search Console

**URL**: https://search.google.com/search-console

#### Steps:
1. **Verify ownership**:
   - Add `google-site-verification` meta tag to `<head>`
   - Or upload HTML file to `public/`
   - Or use DNS TXT record

2. **Submit sitemap**:
   ```
   Property: https://workroot.in
   Sitemaps → Add new sitemap
   URL: https://workroot.in/sitemap.xml
   ```

3. **Monitor**:
   - Coverage report (indexed pages)
   - Enhancement reports (Core Web Vitals)
   - Performance (search analytics)

#### Verification Tag
Add to `src/components/SEO.astro`:
```html
<meta name="google-site-verification" content="YOUR_CODE_HERE" />
```

---

### 2. Bing Webmaster Tools

**URL**: https://www.bing.com/webmasters

#### Steps:
1. **Import from Google** (easiest):
   - Use same Google Search Console account
   - One-click import

2. **Or verify manually**:
   - Add `<meta name="msvalidate.01" content="..." />`
   - Or upload XML file

3. **Submit sitemap**:
   ```
   Sitemaps → Submit sitemap
   URL: https://workroot.in/sitemap.xml
   ```

---

### 3. Yandex Webmaster

**URL**: https://webmaster.yandex.com

#### Steps:
1. **Add site**: `https://workroot.in`
2. **Verify**: Upload HTML file or add meta tag
3. **Submit sitemap**:
   ```
   Indexing → Sitemap files
   https://workroot.in/sitemap.xml
   ```

---

### 4. IndexNow (Instant Indexing)

**What**: Real-time indexing API for Bing, Yandex, and others

**URL**: https://www.indexnow.org

#### Implementation:
```bash
# Generate API key
openssl rand -hex 32 > public/[KEY].txt

# Submit URLs on publish/update
curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "workroot.in",
    "key": "YOUR_API_KEY",
    "urlList": [
      "https://workroot.in/blog/new-post/"
    ]
  }'
```

**When to use**: After publishing new blog posts or major updates

---

## Sitemap Automation

### Build-Time Generation

Sitemap is automatically generated on every build:

```bash
npm run build
# Sitemap available at: dist/client/sitemap.xml
```

### Post-Publish Hook (Recommended)

Create `.github/workflows/sitemap-ping.yml`:

```yaml
name: Ping Search Engines

on:
  push:
    branches: [main]
    paths:
      - 'src/content/blog/**'
      - 'src/pages/**'

jobs:
  ping:
    runs-on: ubuntu-latest
    steps:
      - name: Ping Google
        run: |
          curl "https://www.google.com/ping?sitemap=https://workroot.in/sitemap.xml"

      - name: Ping Bing
        run: |
          curl "https://www.bing.com/ping?sitemap=https://workroot.in/sitemap.xml"
```

---

## Testing & Validation

### 1. XML Validation

**Test locally**:
```bash
# Check XML is valid
curl http://localhost:10000/sitemap.xml | xmllint --noout -

# View in browser
open http://localhost:10000/sitemap.xml
```

**Online validators**:
- https://www.xml-sitemaps.com/validate-xml-sitemap.html
- https://validator.w3.org/feed/

### 2. Google Sitemap Tester

```
Google Search Console → Sitemaps → Test sitemap
```

### 3. Check Coverage

**Verify all important pages are included**:
```bash
curl https://workroot.in/sitemap.xml | grep -o '<loc>[^<]*</loc>'
```

**Expected count**:
- Static pages: 9
- Blog posts: 3+ (grows with content)
- Portfolio items: 0+ (when added)

---

## Troubleshooting

| Issue | Solution |
|-------|----------|
| Sitemap returns 404 | Check `src/pages/sitemap.xml.ts` exists, rebuild |
| Empty sitemap | Check blog posts aren't all `draft: true` |
| Search Console errors | Use Sitemap Tester, check XML validity |
| Pages not indexed | Check robots.txt doesn't block, verify HTTPS |
| Slow indexing | Submit via IndexNow, ping search engines |

---

## Maintenance Checklist

- [ ] **Monthly**: Check Search Console coverage report
- [ ] **When publishing**: Ping search engines (manual or automated)
- [ ] **Quarterly**: Validate sitemap XML structure
- [ ] **When adding collections**: Update `sitemap.xml.ts` to include new content types
- [ ] **Annually**: Review robots.txt directives, update AI bot list

---

## Adding New Content Types

### Example: Portfolio Collection

1. **Create collection** (`src/content/config.ts`):
```typescript
const portfolio = defineCollection({
  type: 'content',
  schema: z.object({
    title: z.string(),
    description: z.string(),
    date: z.coerce.date(),
    updatedDate: z.coerce.date().optional(),
    // ... other fields
  }),
});

export const collections = { blog, portfolio };
```

2. **Sitemap auto-includes** (already implemented in `sitemap.xml.ts`):
```typescript
// Already done! ✅
let portfolioItems: any[] = [];
try {
  portfolioItems = await getCollection('portfolio');
} catch (e) {
  // Collection doesn't exist yet
}

const portfolioPages: SitemapEntry[] = portfolioItems.map((item) => ({
  url: `/portfolio/${item.slug}/`,
  lastmod: item.data.updatedDate?.toISOString().split('T')[0],
  changefreq: 'monthly',
  priority: 0.7,
}));
```

3. **Rebuild & verify**:
```bash
npm run build
curl https://workroot.in/sitemap.xml | grep portfolio
```

---

## Performance Optimizations

### Current Implementation

| Optimization | Status |
|--------------|--------|
| Cache-Control header | ✅ 1 hour |
| Gzip compression | ✅ Via Nginx/CDN |
| XML minification | ✅ No whitespace waste |
| Conditional inclusion | ✅ Drafts excluded |
| X-Robots-Tag | ✅ Sitemap not indexed |

### Advanced: Sitemap Index

**When sitemap exceeds 50,000 URLs**:

Use `sitemap-index.xml.ts` to split into multiple sitemaps:
- `sitemap-blog.xml` - Blog posts
- `sitemap-portfolio.xml` - Portfolio
- `sitemap-static.xml` - Static pages

Currently not needed (< 100 URLs), but implementation ready.

---

## AI Search Visibility (GEO)

### Why Allow AI Crawlers?

| Benefit | Impact |
|---------|--------|
| **Brand mentions** | Cited in AI responses |
| **Authority** | Recognized as source |
| **Traffic** | Users click through from AI |
| **Future-proof** | AI search is growing |

### Monitoring AI Citations

**Track manually**:
- Search "[your topic]" in ChatGPT, Claude, Perplexity
- Check if workroot.in is cited

**Tools** (emerging):
- https://citationtracker.ai (concept)
- Monitor referrer traffic from AI platforms

---

## Resources

| Resource | URL |
|----------|-----|
| Sitemap Protocol | https://www.sitemaps.org/protocol.html |
| Google Sitemap Guide | https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview |
| Robots.txt Spec | https://www.robotstxt.org/ |
| IndexNow Docs | https://www.indexnow.org/documentation |
| AI Crawler List | https://darkvisitors.com/ |

---

## Summary

✅ **Automated sitemap** - Dynamically includes all blog posts
✅ **Portfolio-ready** - Supports future portfolio collection
✅ **AI-friendly robots.txt** - Allows all major AI crawlers
✅ **SEO best practices** - Proper priorities, change frequencies
✅ **Scalable architecture** - Sitemap index ready for growth

**Next steps**:
1. Submit sitemap to Google Search Console
2. Submit sitemap to Bing Webmaster Tools
3. Set up IndexNow for instant indexing
4. Monitor coverage reports monthly

---

**Generated**: 2026-03-21
**Domain**: workroot.in
**Agent**: seo-specialist
