CompanySite/.agents/devops-engineer/MONITORING_SETUP.md

# Monitoring & Uptime Alerts — WorkRoot Website

> Production monitoring setup for workroot.in

---

## Overview

| Layer | Tool | Coverage |
|-------|------|----------|
| **Active checks** | GitHub Actions (every 5 min) | Uptime, response time, pages, SSL |
| **External uptime** | UptimeRobot (free tier) | HTTP 200, keyword, SSL cert |
| **Health endpoint** | `/api/health.json` | Server status, memory, uptime |
| **Metrics endpoint** | `/api/metrics.json` | Request counts, response times, error rate |

---

## Files Created / Modified

| File | Purpose |
|------|---------|
| `src/pages/api/health.json.ts` | **Enhanced** — now includes uptime, memory usage, version |
| `src/pages/api/metrics.json.ts` | **New** — request metrics, response time percentiles, error rate |
| `.github/workflows/uptime-monitor.yml` | **New** — runs every 5 min via GitHub Actions cron |
| `scripts/setup-uptimerobot.sh` | **New** — automates UptimeRobot monitor creation |

---

## Health Endpoint

**URL:** `https://workroot.in/api/health.json`

**Response:**
```json
{
  "status": "ok",
  "timestamp": "2026-03-21T10:00:00.000Z",
  "uptime": 86400,
  "version": "1.0.0",
  "mode": "ssr",
  "adapter": "node-standalone",
  "domain": "workroot.in",
  "memory": {
    "heapUsedMB": 45,
    "heapTotalMB": 64,
    "rssMB": 82
  },
  "checks": {
    "server": "ok"
  }
}
```

Used by: CI/CD pipeline, uptime monitors, load balancers.

---

## Metrics Endpoint

**URL:** `https://workroot.in/api/metrics.json`

**Authentication:** Optional. Set `METRICS_TOKEN` env var to require `Authorization: Bearer <token>`.

**Response:**
```json
{
  "timestamp": "2026-03-21T10:00:00.000Z",
  "uptime": { "seconds": 86400, "human": "1d 0h 0m 0s" },
  "requests": { "total": 1250, "errors": 3, "errorRate": "0.24%" },
  "responseTime": { "avgMs": 145, "p50Ms": 120, "p95Ms": 380, "p99Ms": 750, "samples": 100 },
  "memory": { "heapUsedMB": 45, "heapTotalMB": 64, "externalMB": 2, "rssMB": 82 },
  "process": { "pid": 1234, "nodeVersion": "v20.0.0", "platform": "linux" }
}
```

---

## GitHub Actions Uptime Monitor

**File:** `.github/workflows/uptime-monitor.yml`

**Schedule:** Every 5 minutes (`*/5 * * * *`)

### What It Checks

| Check | Threshold | Alert |
|-------|-----------|-------|
| Health endpoint HTTP 200 | Must be 200 | Failure → alert job runs |
| Health status field | Must be `"ok"` | Failure → alert job runs |
| Response time | < 3000ms | Slow → alert job runs |
| Critical pages (/, /services, /portfolio, /contact, /about) | HTTP 200 | Failure → job fails |
| Sitemap + robots.txt | HTTP 200 | Failure → job fails |
| SSL certificate | > 14 days remaining | Failure → alert job runs |

### Alert Channels

Currently configured in the workflow as commented examples. To enable:

#### Slack Alerts
1. Create a Slack Incoming Webhook
2. Add secret: `SLACK_WEBHOOK_URL` in GitHub → Settings → Secrets → Actions
3. Uncomment the Slack notification block in `.github/workflows/uptime-monitor.yml`

#### Generic Webhook (email, PagerDuty, etc.)
1. Add secret: `ALERT_WEBHOOK_URL`
2. Uncomment the webhook notification block in `.github/workflows/uptime-monitor.yml`

### Viewing Results

- GitHub → Actions → "Uptime Monitor" tab shows every run
- Failed runs = site is down or degraded
- Each run summary shows response times and SSL days remaining

---

## UptimeRobot Setup (External Monitoring)

UptimeRobot provides monitoring from external IPs, independent of GitHub Actions.

### Quick Setup

```bash
# Set your API key (from UptimeRobot dashboard → My Settings → API Settings)
export UPTIMEROBOT_API_KEY="ur_xxxxxxxxxxxxxxxx"

# Optional: set alert email
export ALERT_EMAIL="alerts@workroot.in"

# Run setup script
bash scripts/setup-uptimerobot.sh
```

### Manual Setup (Free Tier)

1. Sign up at **https://uptimerobot.com** (free)
2. Create monitors:

| Monitor Name | URL | Type | Interval |
|-------------|-----|------|----------|
| WorkRoot Health | `https://workroot.in/api/health.json` | HTTP(s) | 5 min |
| WorkRoot Homepage | `https://workroot.in/` | HTTP(s) | 5 min |
| WorkRoot Health Keyword | `https://workroot.in/api/health.json` | Keyword | 5 min |
| WorkRoot Services | `https://workroot.in/services` | HTTP(s) | 5 min |
| WorkRoot Contact | `https://workroot.in/contact` | HTTP(s) | 5 min |

3. For keyword monitor: keyword = `"status":"ok"`, type = "Exists"
4. Enable **SSL monitoring** on each HTTPS monitor:
   - Edit monitor → Advanced → SSL monitoring: ON
   - Alert threshold: 14 days before expiry
5. Set **response time alert**: Edit → Alert when response time > 3000ms
6. Configure **alert contacts**: Alert Contacts → Add Email/Slack/Webhook

### Status Page

Create a public status page:
- UptimeRobot Dashboard → Status Pages → Create New
- Add all monitors
- Set URL: `status.workroot.in` (add CNAME DNS record)

---

## Alert Thresholds Reference

| Metric | Warning | Critical |
|--------|---------|----------|
| Response time | > 2000ms | > 3000ms |
| SSL expiry | < 30 days | < 14 days |
| Memory (heap) | > 80% | > 95% |
| Error rate | > 1% | > 5% |
| Downtime | 1 failed check | 2+ consecutive |

---

## Incident Response Runbook

### Site Down (HTTP non-200 or timeout)

```
1. Check GitHub Actions → Uptime Monitor for recent failures
2. Check UptimeRobot → Incidents for start time and location
3. SSH to server: ssh deploy@<VPS_HOST>
4. pm2 status                              # Is process running?
5. pm2 logs workroot-website --lines 50    # Check for crash errors
6. curl http://localhost:10000/api/health.json  # Direct check
7. If crashed: pm2 restart workroot-website
8. If persistent: trigger rollback (see CI/CD pipeline docs)
```

### Slow Response (> 3s)

```
1. Check /api/metrics.json for memory and error rate
2. pm2 monit                               # Real-time CPU/memory
3. Check for memory leak: heapUsedMB trending up?
4. Check Nginx logs: sudo tail -f /var/log/nginx/access.log
5. If memory issue: pm2 restart workroot-website (graceful)
6. Consider scaling: increase PM2 cluster instances
```

### SSL Certificate Expiring

```
1. SSH to VPS
2. Check cert: echo | openssl s_client -connect workroot.in:443 2>/dev/null | openssl x509 -noout -dates
3. Renew with Certbot: sudo certbot renew --nginx
4. Verify renewal: sudo certbot certificates
5. Reload Nginx: sudo nginx -s reload
```

### High Error Rate

```
1. Check /api/metrics.json → requests.errorRate
2. pm2 logs workroot-website --err --lines 100
3. Check Sentry dashboard for exception details
4. Identify error pattern (specific endpoint? all routes?)
5. Deploy hotfix or rollback if regression
```

---

## Environment Variables for Monitoring

Add to `.env` (production) or hosting platform secrets:

```env
# Optional: protect the /api/metrics.json endpoint
METRICS_TOKEN=your-secure-random-token-here
```

---

## Dashboard Quick Links

| Resource | URL |
|----------|-----|
| Health endpoint | https://workroot.in/api/health.json |
| Metrics endpoint | https://workroot.in/api/metrics.json |
| GitHub Actions | https://github.com/<org>/<repo>/actions/workflows/uptime-monitor.yml |
| UptimeRobot | https://uptimerobot.com/dashboard |
| UptimeRobot Status Page | https://status.workroot.in *(after setup)* |

---

*Created by: devops-engineer agent | Date: 2026-03-21*