Clintchiz
d402256547
Deploy to Production / Build & Verify (push) Failing after 5m56s
Ping Search Engines / Notify Search Engines (push) Successful in 2s
Deploy to Production / Pre-Deploy Tests (push) Has been skipped
Deploy to Production / Deploy to Railway (push) Has been skipped
Deploy to Production / Deploy to Render (push) Has been skipped
Deploy to Production / Deploy to VPS (PM2) (push) Has been skipped
Deploy to Production / Deploy to Fly.io (push) Has been skipped
Deploy to Production / Post-Deploy Verification (push) Has been skipped
Deploy to Production / Notify on Failure (push) Successful in 2s
E2E Test Suite / Critical User Journeys (push) Has been skipped
E2E Test Suite / API Integration Tests (push) Has been skipped
E2E Test Suite / Smoke Tests (P0) (push) Failing after 11m26s
E2E Test Suite / Form Interaction Tests (push) Failing after 11m42s
E2E Test Suite / Destructive & Chaos Tests (push) Failing after 12m2s
E2E Test Suite / Cross-Browser Regression (chromium) (push) Failing after 16m14s
E2E Test Suite / Cross-Browser Regression (webkit) (push) Failing after 17m45s
E2E Test Suite / Cross-Browser Regression (firefox) (push) Failing after 25m23s
E2E Test Suite / Security Header Tests (push) Failing after 7m55s
E2E Test Suite / Test Report Summary (push) Failing after 20s
E2E Test Suite / Mobile Device Tests (push) Failing after 2h49m9s
Uptime Monitor / Health & Response Time (push) Failing after 2s
Uptime Monitor / SSL Certificate (push) Successful in 2s
Uptime Monitor / Send Alerts (push) Failing after 3s
Uptime Monitor / Record Uptime Success (push) Has been skipped
402 lines
9.3 KiB
Markdown
402 lines
9.3 KiB
Markdown
# Sitemap & Robots.txt Setup Guide
|
|
|
|
> **SEO Foundation**: Automated sitemap generation and search engine crawl directives for WorkRoot IT Solutions
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
| File | Purpose | Location |
|
|
|------|---------|----------|
|
|
| **sitemap.xml** | Dynamic XML sitemap | `src/pages/sitemap.xml.ts` |
|
|
| **sitemap-index.xml** | Sitemap index (scalability) | `src/pages/sitemap-index.xml.ts` |
|
|
| **robots.txt** | Crawl directives | `public/robots.txt` |
|
|
|
|
**Site URL**: `https://workroot.in`
|
|
|
|
---
|
|
|
|
## Sitemap Implementation
|
|
|
|
### Current Coverage
|
|
|
|
| Content Type | Included | Priority | Change Frequency |
|
|
|--------------|----------|----------|------------------|
|
|
| Homepage | ✅ | 1.0 | Weekly |
|
|
| Services | ✅ | 0.9 | Monthly |
|
|
| Portfolio Index | ✅ | 0.8 | Weekly |
|
|
| About | ✅ | 0.8 | Monthly |
|
|
| Blog Index | ✅ | 0.8 | Weekly |
|
|
| Blog Posts | ✅ | 0.7 | Monthly |
|
|
| Portfolio Items | ✅ | 0.7 | Monthly |
|
|
| Contact | ✅ | 0.7 | Monthly |
|
|
| Legal Pages | ✅ | 0.3 | Yearly |
|
|
|
|
### Features
|
|
|
|
- ✅ **Dynamic generation** - Automatically includes all published blog posts
|
|
- ✅ **Draft filtering** - Excludes draft content from sitemap
|
|
- ✅ **Portfolio support** - Ready for portfolio content collection
|
|
- ✅ **Last modified dates** - Uses `updatedDate` or `pubDate` from frontmatter
|
|
- ✅ **Proper namespaces** - Includes news, image, video schemas for future use
|
|
- ✅ **Caching headers** - 1-hour cache for performance
|
|
- ✅ **XML formatting** - Valid sitemap protocol
|
|
|
|
---
|
|
|
|
## Robots.txt Configuration
|
|
|
|
### Key Directives
|
|
|
|
```
|
|
User-agent: *
|
|
Allow: /
|
|
|
|
# Block private areas
|
|
Disallow: /admin/
|
|
Disallow: /private/
|
|
Disallow: /_astro/
|
|
|
|
# Sitemap location
|
|
Sitemap: https://workroot.in/sitemap.xml
|
|
```
|
|
|
|
### AI Crawler Support (GEO)
|
|
|
|
The robots.txt explicitly allows **AI search engines** for Generative Engine Optimization:
|
|
|
|
| Bot | Purpose | Allowed |
|
|
|-----|---------|---------|
|
|
| GPTBot | ChatGPT training/search | ✅ |
|
|
| ChatGPT-User | ChatGPT browsing | ✅ |
|
|
| anthropic-ai | Claude AI training | ✅ |
|
|
| Claude-Web | Claude search | ✅ |
|
|
| PerplexityBot | Perplexity AI | ✅ |
|
|
| Google-Extended | Gemini/Bard | ✅ |
|
|
| CCBot | Common Crawl (many AIs) | ✅ |
|
|
| FacebookBot | Meta AI | ✅ |
|
|
|
|
**Why?** AI search engines can cite your content in responses, increasing brand visibility.
|
|
|
|
---
|
|
|
|
## Search Engine Submission
|
|
|
|
### 1. Google Search Console
|
|
|
|
**URL**: https://search.google.com/search-console
|
|
|
|
#### Steps:
|
|
1. **Verify ownership**:
|
|
- Add `google-site-verification` meta tag to `<head>`
|
|
- Or upload HTML file to `public/`
|
|
- Or use DNS TXT record
|
|
|
|
2. **Submit sitemap**:
|
|
```
|
|
Property: https://workroot.in
|
|
Sitemaps → Add new sitemap
|
|
URL: https://workroot.in/sitemap.xml
|
|
```
|
|
|
|
3. **Monitor**:
|
|
- Coverage report (indexed pages)
|
|
- Enhancement reports (Core Web Vitals)
|
|
- Performance (search analytics)
|
|
|
|
#### Verification Tag
|
|
Add to `src/components/SEO.astro`:
|
|
```html
|
|
<meta name="google-site-verification" content="YOUR_CODE_HERE" />
|
|
```
|
|
|
|
---
|
|
|
|
### 2. Bing Webmaster Tools
|
|
|
|
**URL**: https://www.bing.com/webmasters
|
|
|
|
#### Steps:
|
|
1. **Import from Google** (easiest):
|
|
- Use same Google Search Console account
|
|
- One-click import
|
|
|
|
2. **Or verify manually**:
|
|
- Add `<meta name="msvalidate.01" content="..." />`
|
|
- Or upload XML file
|
|
|
|
3. **Submit sitemap**:
|
|
```
|
|
Sitemaps → Submit sitemap
|
|
URL: https://workroot.in/sitemap.xml
|
|
```
|
|
|
|
---
|
|
|
|
### 3. Yandex Webmaster
|
|
|
|
**URL**: https://webmaster.yandex.com
|
|
|
|
#### Steps:
|
|
1. **Add site**: `https://workroot.in`
|
|
2. **Verify**: Upload HTML file or add meta tag
|
|
3. **Submit sitemap**:
|
|
```
|
|
Indexing → Sitemap files
|
|
https://workroot.in/sitemap.xml
|
|
```
|
|
|
|
---
|
|
|
|
### 4. IndexNow (Instant Indexing)
|
|
|
|
**What**: Real-time indexing API for Bing, Yandex, and others
|
|
|
|
**URL**: https://www.indexnow.org
|
|
|
|
#### Implementation:
|
|
```bash
|
|
# Generate API key
|
|
openssl rand -hex 32 > public/[KEY].txt
|
|
|
|
# Submit URLs on publish/update
|
|
curl -X POST "https://api.indexnow.org/indexnow" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"host": "workroot.in",
|
|
"key": "YOUR_API_KEY",
|
|
"urlList": [
|
|
"https://workroot.in/blog/new-post/"
|
|
]
|
|
}'
|
|
```
|
|
|
|
**When to use**: After publishing new blog posts or major updates
|
|
|
|
---
|
|
|
|
## Sitemap Automation
|
|
|
|
### Build-Time Generation
|
|
|
|
Sitemap is automatically generated on every build:
|
|
|
|
```bash
|
|
npm run build
|
|
# Sitemap available at: dist/client/sitemap.xml
|
|
```
|
|
|
|
### Post-Publish Hook (Recommended)
|
|
|
|
Create `.github/workflows/sitemap-ping.yml`:
|
|
|
|
```yaml
|
|
name: Ping Search Engines
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
paths:
|
|
- 'src/content/blog/**'
|
|
- 'src/pages/**'
|
|
|
|
jobs:
|
|
ping:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Ping Google
|
|
run: |
|
|
curl "https://www.google.com/ping?sitemap=https://workroot.in/sitemap.xml"
|
|
|
|
- name: Ping Bing
|
|
run: |
|
|
curl "https://www.bing.com/ping?sitemap=https://workroot.in/sitemap.xml"
|
|
```
|
|
|
|
---
|
|
|
|
## Testing & Validation
|
|
|
|
### 1. XML Validation
|
|
|
|
**Test locally**:
|
|
```bash
|
|
# Check XML is valid
|
|
curl http://localhost:10000/sitemap.xml | xmllint --noout -
|
|
|
|
# View in browser
|
|
open http://localhost:10000/sitemap.xml
|
|
```
|
|
|
|
**Online validators**:
|
|
- https://www.xml-sitemaps.com/validate-xml-sitemap.html
|
|
- https://validator.w3.org/feed/
|
|
|
|
### 2. Google Sitemap Tester
|
|
|
|
```
|
|
Google Search Console → Sitemaps → Test sitemap
|
|
```
|
|
|
|
### 3. Check Coverage
|
|
|
|
**Verify all important pages are included**:
|
|
```bash
|
|
curl https://workroot.in/sitemap.xml | grep -o '<loc>[^<]*</loc>'
|
|
```
|
|
|
|
**Expected count**:
|
|
- Static pages: 9
|
|
- Blog posts: 3+ (grows with content)
|
|
- Portfolio items: 0+ (when added)
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
| Issue | Solution |
|
|
|-------|----------|
|
|
| Sitemap returns 404 | Check `src/pages/sitemap.xml.ts` exists, rebuild |
|
|
| Empty sitemap | Check blog posts aren't all `draft: true` |
|
|
| Search Console errors | Use Sitemap Tester, check XML validity |
|
|
| Pages not indexed | Check robots.txt doesn't block, verify HTTPS |
|
|
| Slow indexing | Submit via IndexNow, ping search engines |
|
|
|
|
---
|
|
|
|
## Maintenance Checklist
|
|
|
|
- [ ] **Monthly**: Check Search Console coverage report
|
|
- [ ] **When publishing**: Ping search engines (manual or automated)
|
|
- [ ] **Quarterly**: Validate sitemap XML structure
|
|
- [ ] **When adding collections**: Update `sitemap.xml.ts` to include new content types
|
|
- [ ] **Annually**: Review robots.txt directives, update AI bot list
|
|
|
|
---
|
|
|
|
## Adding New Content Types
|
|
|
|
### Example: Portfolio Collection
|
|
|
|
1. **Create collection** (`src/content/config.ts`):
|
|
```typescript
|
|
const portfolio = defineCollection({
|
|
type: 'content',
|
|
schema: z.object({
|
|
title: z.string(),
|
|
description: z.string(),
|
|
date: z.coerce.date(),
|
|
updatedDate: z.coerce.date().optional(),
|
|
// ... other fields
|
|
}),
|
|
});
|
|
|
|
export const collections = { blog, portfolio };
|
|
```
|
|
|
|
2. **Sitemap auto-includes** (already implemented in `sitemap.xml.ts`):
|
|
```typescript
|
|
// Already done! ✅
|
|
let portfolioItems: any[] = [];
|
|
try {
|
|
portfolioItems = await getCollection('portfolio');
|
|
} catch (e) {
|
|
// Collection doesn't exist yet
|
|
}
|
|
|
|
const portfolioPages: SitemapEntry[] = portfolioItems.map((item) => ({
|
|
url: `/portfolio/${item.slug}/`,
|
|
lastmod: item.data.updatedDate?.toISOString().split('T')[0],
|
|
changefreq: 'monthly',
|
|
priority: 0.7,
|
|
}));
|
|
```
|
|
|
|
3. **Rebuild & verify**:
|
|
```bash
|
|
npm run build
|
|
curl https://workroot.in/sitemap.xml | grep portfolio
|
|
```
|
|
|
|
---
|
|
|
|
## Performance Optimizations
|
|
|
|
### Current Implementation
|
|
|
|
| Optimization | Status |
|
|
|--------------|--------|
|
|
| Cache-Control header | ✅ 1 hour |
|
|
| Gzip compression | ✅ Via Nginx/CDN |
|
|
| XML minification | ✅ No whitespace waste |
|
|
| Conditional inclusion | ✅ Drafts excluded |
|
|
| X-Robots-Tag | ✅ Sitemap not indexed |
|
|
|
|
### Advanced: Sitemap Index
|
|
|
|
**When sitemap exceeds 50,000 URLs**:
|
|
|
|
Use `sitemap-index.xml.ts` to split into multiple sitemaps:
|
|
- `sitemap-blog.xml` - Blog posts
|
|
- `sitemap-portfolio.xml` - Portfolio
|
|
- `sitemap-static.xml` - Static pages
|
|
|
|
Currently not needed (< 100 URLs), but implementation ready.
|
|
|
|
---
|
|
|
|
## AI Search Visibility (GEO)
|
|
|
|
### Why Allow AI Crawlers?
|
|
|
|
| Benefit | Impact |
|
|
|---------|--------|
|
|
| **Brand mentions** | Cited in AI responses |
|
|
| **Authority** | Recognized as source |
|
|
| **Traffic** | Users click through from AI |
|
|
| **Future-proof** | AI search is growing |
|
|
|
|
### Monitoring AI Citations
|
|
|
|
**Track manually**:
|
|
- Search "[your topic]" in ChatGPT, Claude, Perplexity
|
|
- Check if workroot.in is cited
|
|
|
|
**Tools** (emerging):
|
|
- https://citationtracker.ai (concept)
|
|
- Monitor referrer traffic from AI platforms
|
|
|
|
---
|
|
|
|
## Resources
|
|
|
|
| Resource | URL |
|
|
|----------|-----|
|
|
| Sitemap Protocol | https://www.sitemaps.org/protocol.html |
|
|
| Google Sitemap Guide | https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview |
|
|
| Robots.txt Spec | https://www.robotstxt.org/ |
|
|
| IndexNow Docs | https://www.indexnow.org/documentation |
|
|
| AI Crawler List | https://darkvisitors.com/ |
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
✅ **Automated sitemap** - Dynamically includes all blog posts
|
|
✅ **Portfolio-ready** - Supports future portfolio collection
|
|
✅ **AI-friendly robots.txt** - Allows all major AI crawlers
|
|
✅ **SEO best practices** - Proper priorities, change frequencies
|
|
✅ **Scalable architecture** - Sitemap index ready for growth
|
|
|
|
**Next steps**:
|
|
1. Submit sitemap to Google Search Console
|
|
2. Submit sitemap to Bing Webmaster Tools
|
|
3. Set up IndexNow for instant indexing
|
|
4. Monitor coverage reports monthly
|
|
|
|
---
|
|
|
|
**Generated**: 2026-03-21
|
|
**Domain**: workroot.in
|
|
**Agent**: seo-specialist
|