CompanySite/.agents/devops-engineer/CI_CD_PIPELINE.md

# CI/CD Pipeline — WorkRoot Website

> Production deployment pipeline documentation for the WorkRoot website.
> Framework: Astro SSR + Express | Platform: GitHub Actions

---

## Overview

The CI/CD system consists of two complementary pipelines:

| Workflow | File | Purpose |
|----------|------|---------|
| **E2E Test Suite** | `.github/workflows/e2e-tests.yml` | Runs on every push/PR — 9 test jobs |
| **Deploy to Production** | `.github/workflows/deploy.yml` | Deploys on merge to `main` |

---

## Pipeline Architecture

```
Push to main
     │
     ├─── E2E Test Suite (parallel) ──────────────────────────────────┐
     │    ├── Smoke Tests (P0)                                         │
     │    ├── Critical Paths                                           │
     │    └── API Integration                                          │
     │                                                                 │
     └─── Deploy Pipeline ────────────────────────────────────────────┘
          │
          ▼
    [Job 1] Build & Verify
          │  ✓ npm ci
          │  ✓ tsc --noEmit
          │  ✓ npm run build
          │  ✓ Verify dist/ structure
          │  ✓ Upload build artifact
          │
          ▼
    [Job 2] Pre-Deploy Tests
          │  ✓ Download build artifact
          │  ✓ Start server locally
          │  ✓ Run smoke tests (Chromium)
          │  ✓ Run API integration tests
          │
          ▼
    [Job 3] Deploy (one of:)
          ├── Railway (DEPLOY_TARGET=railway)
          ├── Render  (DEPLOY_TARGET=render)
          ├── VPS/PM2 (DEPLOY_TARGET=vps)
          └── Fly.io  (DEPLOY_TARGET=fly)
          │
          ▼
    [Job 4] Post-Deploy Verification
          │  ✓ Smoke tests against live production
          │  ✓ Critical endpoint checks
          │  ✓ Upload results (14-day retention)
          │
          ▼
    [Job 5] Notify on Failure (if any stage failed)
```

---

## Deployment Targets

### Option A: Railway (Recommended for simplicity)

Railway auto-deploys from GitHub. CI/CD adds a verification layer.

**Required secrets:**
| Secret | Value |
|--------|-------|
| `RAILWAY_TOKEN` | From Railway dashboard → Settings → Tokens |

**Required variables:**
| Variable | Value |
|----------|-------|
| `DEPLOY_TARGET` | `railway` |

**Setup:**
1. Create a Railway project, connect the GitHub repo
2. Set `START_COMMAND`: `npm run start:prod`
3. Set `PORT`: `10000`
4. Add the `RAILWAY_TOKEN` secret to GitHub
5. Set `DEPLOY_TARGET=railway` in GitHub repo variables

---

### Option B: Render

Render uses deploy hooks triggered by the CI pipeline.

**Required secrets:**
| Secret | Value |
|--------|-------|
| `RENDER_DEPLOY_HOOK_URL` | From Render dashboard → Service → Deploy Hook URL |

**Required variables:**
| Variable | Value |
|----------|-------|
| `DEPLOY_TARGET` | `render` |

**Render service configuration:**
- Environment: Node
- Build command: `npm ci && npm run build`
- Start command: `npm run start:prod`
- Health check path: `/api/health.json`

---

### Option C: VPS with PM2 (Full Control)

SSH-based deployment to any Linux VPS. Includes automatic rollback.

**Required secrets:**
| Secret | Description |
|--------|-------------|
| `VPS_HOST` | VPS IP or hostname |
| `VPS_USER` | SSH user (e.g., `ubuntu`, `deploy`) |
| `VPS_SSH_PRIVATE_KEY` | Private key (contents of `~/.ssh/id_rsa`) |
| `VPS_HOST_KEY` | SSH host key fingerprint |

**Required variables:**
| Variable | Value |
|----------|-------|
| `DEPLOY_TARGET` | `vps` |

**VPS setup (one-time):**
```bash
# On your VPS:
# 1. Create deployment directory
mkdir -p /var/www/workroot
chown -R deploy:www-data /var/www/workroot

# 2. Install Node.js 20 (via nvm or nodesource)
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs

# 3. Install PM2 globally
npm install -g pm2
pm2 startup  # Follow the output instructions

# 4. Create logs directory
mkdir -p /var/www/workroot/logs

# 5. Set up Nginx reverse proxy (port 80/443 → 10000)
# See Nginx config below
```

**Nginx configuration:**
```nginx
server {
    listen 80;
    server_name workroot.in www.workroot.in;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name workroot.in www.workroot.in;

    ssl_certificate /etc/letsencrypt/live/workroot.in/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/workroot.in/privkey.pem;

    location / {
        proxy_pass http://localhost:10000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
    }
}
```

**Deployment behavior:**
1. Backs up current deployment before replacing
2. Extracts new files to `/var/www/workroot/`
3. Installs production dependencies only
4. Uses `pm2 reload` for zero-downtime restart
5. Runs health check (12 attempts × 10s = 2 minutes max)
6. **Auto-rollback** if health check fails

---

### Option D: Fly.io

Container-based deployment with edge distribution.

**Required secrets:**
| Secret | Value |
|--------|-------|
| `FLY_API_TOKEN` | From `flyctl auth token` |

**Required variables:**
| Variable | Value |
|----------|-------|
| `DEPLOY_TARGET` | `fly` |

**Setup:**
```bash
# Install flyctl
curl -L https://fly.io/install.sh | sh

# Authenticate
flyctl auth login

# Launch app (first time only)
flyctl launch --name workroot-website

# Set production secrets
flyctl secrets set NODE_ENV=production PORT=10000
```

**fly.toml** (create in project root if using Fly.io):
```toml
app = "workroot-website"
primary_region = "sin"  # Singapore - closest to India

[build]
  [build.args]
    NODE_VERSION = "20"

[env]
  PORT = "10000"
  HOST = "0.0.0.0"
  NODE_ENV = "production"

[http_service]
  internal_port = 10000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 1

  [http_service.concurrency]
    type = "connections"
    hard_limit = 25
    soft_limit = 20

[[vm]]
  cpu_kind = "shared"
  cpus = 1
  memory_mb = 512
```

---

## GitHub Repository Setup

### Required Secrets (Settings → Secrets → Actions)

Configure secrets for your chosen deployment target:

```
# For ALL targets:
# (none required at the base level)

# For Railway:
RAILWAY_TOKEN=<token>

# For Render:
RENDER_DEPLOY_HOOK_URL=https://api.render.com/deploy/...

# For VPS:
VPS_HOST=123.456.789.0
VPS_USER=deploy
VPS_SSH_PRIVATE_KEY=-----BEGIN OPENSSH PRIVATE KEY-----...
VPS_HOST_KEY=123.456.789.0 ssh-rsa AAAA...

# Optional notifications:
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
```

### Required Variables (Settings → Variables → Actions)

```
DEPLOY_TARGET=railway  # or: render, vps, fly
```

### Environment Protection Rules

Configure via Settings → Environments:

1. Create environment named `production`
2. Enable "Required reviewers" for manual approval before deploy
3. Set allowed branches to `main` only

---

## Trigger Conditions

| Event | Build | Tests | Deploy |
|-------|-------|-------|--------|
| Push to `main` | ✓ | ✓ | ✓ |
| Manual dispatch | ✓ | ✓ (unless skip_tests=true) | ✓ |
| Push to `develop` | via e2e-tests.yml | ✓ | ✗ |
| Pull Request | via e2e-tests.yml | ✓ | ✗ |

---

## Environment Variables in Production

All env vars must be configured in your hosting platform, not in the workflow.

| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `NODE_ENV` | Yes | — | Must be `production` |
| `HOST` | Yes | — | `0.0.0.0` |
| `PORT` | Yes | — | `10000` |
| `CONTACT_EMAIL` | Yes (legacy) | — | Legacy admin recipient for contact form; kept for backward compatibility with older code paths. Mirror of `MAIL_ADMIN_TO`. |
| `MAIL_ADMIN_TO` | Yes | — | Address that receives "Tell us about your project" and contact form submissions. |
| `SMTP_HOST` | Yes | — | SMTP server hostname (e.g. `smtp.hostinger.com`). |
| `SMTP_PORT` | Yes | `587` | SMTP server port. Use `587` for STARTTLS, `465` for implicit TLS. |
| `SMTP_USER` | Yes | — | SMTP auth username (full email address). |
| `SMTP_PASS` | Yes | — | SMTP auth password — MUST be stored as a platform secret, never plaintext. |
| `SMTP_FROM` | Yes | — | `From:` address used by outbound mail (should match `SMTP_USER` or an allowed alias). |
| `SMTP_FROM_NAME` | Yes | — | Human-readable display name shown alongside `SMTP_FROM`. |
| `SMTP_SECURE` | Yes | `false` | `true` only for implicit-TLS ports (465). For port 587 keep `false` and rely on STARTTLS. |
| `SMTP_STARTTLS` | Yes | `true` | Negotiate STARTTLS on the SMTP connection. |
| `SMTP_REQUIRE_TLS` | Yes | `true` | Refuse to send if the server does not advertise STARTTLS — prevents silent plaintext fallback. |
| `SMTP_AUTH` | Yes | `true` | Authenticate with `SMTP_USER` / `SMTP_PASS`. |
| `SMTP_CONNECTION_TIMEOUT_MS` | No | `10000` | Socket connect timeout in milliseconds. |
| `SMTP_GREETING_TIMEOUT_MS` | No | `10000` | Wait time for the server greeting (EHLO/HELO) in milliseconds. |
| `SMTP_SOCKET_TIMEOUT_MS` | No | `10000` | Idle socket timeout in milliseconds (covers read/write while transferring the message). |
| `SENTRY_DSN` | No | — | Sentry error tracking |
| `LOG_LEVEL` | No | `info` | Logging verbosity |

> Production SMTP is provisioned on Hostinger and configured in the **Default** environment via the platform's env-var store (`SMTP_PASS` flagged as secret). After changing any `SMTP_*` value you MUST restart the `frontend` service so the new variables are picked up by the Astro SSR process.

---

## Rollback Procedures

### Railway / Render / Fly.io
Use the hosting platform dashboard to redeploy a previous commit or use the rollback button.

### VPS (PM2)
The deployment script auto-rolls back if health checks fail. For manual rollback:

```bash
# SSH to VPS
ssh deploy@your-vps-ip

# Check what backups exist
ls -la /var/www/ | grep workroot-backup

# Rollback to most recent backup
BACKUP=$(ls -t /var/www/ | grep workroot-backup | head -1)
echo "Rolling back to: $BACKUP"

# Stop, restore, restart
pm2 stop workroot-website
cp -r /var/www/$BACKUP/. /var/www/workroot/
cd /var/www/workroot && npm ci --omit=dev
pm2 start ecosystem.config.cjs --env production
pm2 save

# Verify
curl http://localhost:10000/api/health.json
```

### Emergency Deploy (Skip Tests)
Use `workflow_dispatch` with `skip_tests: true` only in genuine emergencies.
Document the reason in the workflow run description.

---

## Monitoring After Deploy

### Immediate (0–5 minutes)
- [ ] Workflow shows all jobs green
- [ ] Health endpoint: `https://workroot.in/api/health.json` returns `{"status":"ok"}`
- [ ] Homepage loads correctly
- [ ] Contact form accessible

### Short-term (15–60 minutes)
- [ ] No spike in error logs
- [ ] PM2 showing stable process count (VPS only)
- [ ] Nightly E2E suite runs clean

### Verification Commands (VPS)
```bash
# Check PM2 status
pm2 status
pm2 logs workroot-website --lines 50

# Check Nginx logs
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

# Health check
curl -s http://localhost:10000/api/health.json | python3 -m json.tool
```

---

## Pipeline Files Reference

| File | Purpose |
|------|---------|
| `.github/workflows/deploy.yml` | Main deployment pipeline |
| `.github/workflows/e2e-tests.yml` | Test suite (smoke, critical, API, chaos, cross-browser, mobile, security) |
| `.github/workflows/sitemap-ping.yml` | Pings Google/Bing after content changes |
| `ecosystem.config.cjs` | PM2 cluster configuration |
| `server.mjs` | Express wrapper for Astro SSR |
| `.env.example` | Environment variable template |

---

## Adding a New Deployment Target

1. Add a new job block in `deploy.yml` after the existing deploy jobs
2. Add the condition: `vars.DEPLOY_TARGET == 'your-target'`
3. Add required secrets to this document
4. Update the `post-deploy-verify` job's `needs` array to include the new job
5. Test with a manual `workflow_dispatch` trigger

---

*Last updated: 2026-03-21 | Created by: devops-engineer agent*