SRE, SLA/SLO/SLI và Incident Management

🎯 Mục tiêu: Viết SLO document chuẩn cho một web API, tính error budget theo phút và quyết định policy deploy dựa trên budget còn lại.

🧰 Công cụ / nền tảng: WSL2/Ubuntu hoặc Git Bash, VS Code, bc (basic calculator), curl.

📦 Chuẩn bị: Cài WSL2 + Ubuntu. Kiểm tra: bc --version (có sẵn trên Ubuntu). Tạo thư mục làm việc.

mkdir -p ~/sre-lab && cd ~/sre-lab

▶️ Các bước:

Bước 1 — Viết SLO Document

cat > slo-payment-api.md << 'EOF'
# SLO Document — Payment API v1

## Service Overview
- **Service:** payment-api (checkout endpoint)
- **Owner:** [email protected]
- **Review cadence:** Monthly

## SLIs (Service Level Indicators)
| SLI Name        | Query / Metric                                         | Good event definition          |
|-----------------|-------------------------------------------------------|-------------------------------|
| Availability    | HTTP 2xx responses / total responses                  | status_code < 500             |
| Latency p99     | 99th percentile response time                         | latency_ms < 500              |
| Error Rate      | HTTP 5xx responses / total responses                  | error_rate < 0.1%             |

## SLOs (Service Level Objectives)
| SLO                  | Target    | Window       |
|----------------------|-----------|--------------|
| Availability         | >= 99.9%  | 30-day rolling|
| Latency p99          | <= 500ms  | 7-day rolling |
| Error Rate           | < 0.1%    | 24h rolling   |

## SLA (external commitment)
- Availability >= 99.5% per month (buffer: 0.4%)
- Violation → 10% service credit

## Error Budget Policy
- Budget > 50% remaining: normal deploy cadence
- Budget 10–50% remaining: require extra review + canary
- Budget < 10% remaining: freeze feature deployments
- Budget = 0: full freeze, SRE focuses on reliability only
EOF
echo "SLO document created."

Bước 2 — Tính Error Budget

# Các biến cơ bản
SLO=99.9          # percent
WINDOW_DAYS=30

# Tổng số phút trong window
TOTAL_MINUTES=$(echo "$WINDOW_DAYS * 24 * 60" | bc)
echo "Total minutes in window: $TOTAL_MINUTES"
# Output: Total minutes in window: 43200

# Error budget (phút được phép downtime)
ERROR_BUDGET=$(echo "scale=2; $TOTAL_MINUTES * (100 - $SLO) / 100" | bc)
echo "Error budget (minutes): $ERROR_BUDGET"
# Output: Error budget (minutes): 43.20

# Giả sử tháng này đã có 2 incidents: 15 phút + 10 phút
CONSUMED=25
REMAINING=$(echo "scale=2; $ERROR_BUDGET - $CONSUMED" | bc)
PCT_REMAINING=$(echo "scale=1; $REMAINING * 100 / $ERROR_BUDGET" | bc)

echo "Budget consumed: ${CONSUMED} min"
echo "Budget remaining: ${REMAINING} min (${PCT_REMAINING}%)"

# Policy decision
if (( $(echo "$PCT_REMAINING > 50" | bc -l) )); then
  echo "DECISION: Green - Normal deploy cadence OK"
elif (( $(echo "$PCT_REMAINING > 10" | bc -l) )); then
  echo "DECISION: Yellow - Extra review required before deploy"
else
  echo "DECISION: RED - Feature deploys FROZEN"
fi

Bước 3 — Simulate SLI từ log

# Tạo access log giả
cat > access.log << 'EOF'
200 45ms /api/pay
200 120ms /api/pay
500 2ms /api/pay
200 89ms /api/pay
503 1ms /api/pay
200 340ms /api/pay
200 67ms /api/pay
200 510ms /api/pay
200 78ms /api/pay
200 200ms /api/pay
EOF

TOTAL=$(wc -l < access.log)
SUCCESS=$(grep -c "^2" access.log)
ERRORS=$(grep -cE "^5" access.log)

AVAILABILITY=$(echo "scale=4; $SUCCESS * 100 / $TOTAL" | bc)
ERROR_RATE=$(echo "scale=4; $ERRORS * 100 / $TOTAL" | bc)

echo "Total requests: $TOTAL"
echo "Successful (2xx): $SUCCESS"
echo "Errors (5xx): $ERRORS"
echo "Availability: ${AVAILABILITY}%"
echo "Error rate: ${ERROR_RATE}%"

# SLO check
if (( $(echo "$AVAILABILITY >= 99.9" | bc -l) )); then
  echo "SLO STATUS: PASSING"
else
  echo "SLO STATUS: BREACHED - Burning error budget!"
fi

✅ Kết quả mong đợi: File slo-payment-api.md có đủ 5 mục. Script tính đúng error budget = 43.20 phút. SLO check output rõ PASSING/BREACHED. Policy decision hiển thị Green/Yellow/Red dựa trên budget còn lại.

🧹 Cleanup: cd ~ && rm -rf ~/sre-lab (hoặc giữ lại cho LAB-112).

🎯 Mục tiêu: Viết alert rule multi-window burn rate cho Prometheus, validate bằng promtool, chạy thử với Prometheus container.

🧰 Công cụ / nền tảng: Docker Desktop (WSL2 backend), promtool (bundled trong Prometheus Docker image), curl.

📦 Chuẩn bị: Docker Desktop đang chạy. Kiểm tra: docker version.

mkdir -p ~/prom-lab/{rules,config} && cd ~/prom-lab

▶️ Các bước:

Bước 1 — Viết alert rules file

cat > rules/slo-alerts.yml << 'EOF'
groups:
  - name: slo_burn_rate_alerts
    rules:
      # Fast burn: 2% error rate trong 1h = burn rate 14.4x (budget cạn trong ~2h)
      # Kết hợp với 5m window để giảm false positive
      - alert: PaymentAPIHighBurnRate
        expr: |
          (
            rate(http_requests_total{job="payment-api",status=~"5.."}[1h])
            /
            rate(http_requests_total{job="payment-api"}[1h])
          ) > (14.4 * 0.001)
          and
          (
            rate(http_requests_total{job="payment-api",status=~"5.."}[5m])
            /
            rate(http_requests_total{job="payment-api"}[5m])
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
          slo: payment_api_availability
        annotations:
          summary: "Payment API đang burn error budget nhanh (SEV2)"
          description: >
            Burn rate {{ $value | humanize }}x trong 1h.
            Error budget sẽ cạn trong ~{{ printf "%.0f" (div 1 $value) }}h nếu tiếp tục.
          runbook_url: "https://wiki.internal/runbooks/payment-api-high-error-rate"

      # Slow burn: 5% error rate trong 6h = burn rate 36x (budget cạn trong ~20h)
      - alert: PaymentAPISlowBurnRate
        expr: |
          (
            rate(http_requests_total{job="payment-api",status=~"5.."}[6h])
            /
            rate(http_requests_total{job="payment-api"}[6h])
          ) > (6 * 0.001)
          and
          (
            rate(http_requests_total{job="payment-api",status=~"5.."}[30m])
            /
            rate(http_requests_total{job="payment-api"}[30m])
          ) > (6 * 0.001)
        for: 15m
        labels:
          severity: warning
          slo: payment_api_availability
        annotations:
          summary: "Payment API đang burn error budget chậm (SEV3)"
          description: "Burn rate {{ $value | humanize }}x trong 6h. Cần điều tra."

      # Latency SLO: p99 > 500ms
      - alert: PaymentAPIHighLatency
        expr: |
          histogram_quantile(0.99,
            rate(http_request_duration_ms_bucket{job="payment-api"}[5m])
          ) > 500
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Payment API p99 latency vượt SLO (500ms)"
          description: "p99 = {{ $value | humanize }}ms"
EOF
echo "Alert rules written."

Bước 2 — Validate rules bằng promtool

# Validate syntax với promtool trong Docker
docker run --rm \
  -v $(pwd)/rules:/rules \
  prom/prometheus:latest \
  promtool check rules /rules/slo-alerts.yml

Bước 3 — Chạy Prometheus với rules

# Tạo prometheus.yml
cat > config/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "/rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
EOF

# Chạy Prometheus
docker run -d --name prom-slo \
  -p 9090:9090 \
  -v $(pwd)/config/prometheus.yml:/etc/prometheus/prometheus.yml \
  -v $(pwd)/rules:/rules \
  prom/prometheus:latest

echo "Prometheus running at http://localhost:9090"

# Kiểm tra rules loaded
sleep 5
curl -s http://localhost:9090/api/v1/rules | \
  python3 -c "import sys,json; r=json.load(sys.stdin); \
  [print(rule['name']) for g in r['data']['groups'] for rule in g['rules']]"

✅ Kết quả mong đợi: promtool check rules in SUCCESS: 1 rules found. Prometheus UI tại http://localhost:9090/rules hiển thị 3 alert rules ở trạng thái inactive. Không có syntax error.

🧹 Cleanup: docker stop prom-slo && docker rm prom-slo && cd ~ && rm -rf ~/prom-lab

🎯 Mục tiêu: Thực hành toàn bộ vòng đời incident: deploy service, inject lỗi, detect bằng health check, triage, mitigate bằng rollback, resolve và document.

🧰 Công cụ / nền tảng: Docker, bash, curl, jq.

📦 Chuẩn bị: Docker Desktop chạy. sudo apt-get install -y jq (WSL2).

mkdir -p ~/incident-lab && cd ~/incident-lab

▶️ Các bước:

Bước 1 — Deploy "production" service (v1 — healthy)

# Chạy nginx v1 làm service giả
docker run -d --name payment-api-v1 \
  -p 8080:80 \
  --label version=v1 \
  nginx:alpine
echo "v1 deployed"

# Health check
curl -s -o /dev/null -w "HTTP %{http_code} - latency: %{time_total}s\n" \
  http://localhost:8080/

Bước 2 — Deploy v2 với lỗi (inject incident)

# Tạo image "buggy" trả về 500
cat > Dockerfile.buggy << 'EOF'
FROM nginx:alpine
RUN echo 'server { listen 80; location / { return 500 "Internal Server Error"; } }' \
  > /etc/nginx/conf.d/default.conf
EOF

docker build -t payment-api:v2-buggy -f Dockerfile.buggy .

# Simulate "deploy v2" — stop v1, start v2
docker stop payment-api-v1
docker run -d --name payment-api-v2 \
  -p 8080:80 \
  --label version=v2-buggy \
  payment-api:v2-buggy
echo "v2 deployed (with bug!)"

Bước 3 — Detect: automated health check script

cat > health-check.sh << 'EOF'
#!/bin/bash
URL="http://localhost:8080/"
THRESHOLD_CODE=200
LOG="incident.log"
CHECKS=5

echo "=== Starting health check monitoring ==="
for i in $(seq 1 $CHECKS); do
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" $URL)
  LATENCY=$(curl -s -o /dev/null -w "%{time_total}" $URL)
  TS=$(date +"%Y-%m-%dT%H:%M:%S")
  echo "$TS HTTP $HTTP_CODE latency=${LATENCY}s" | tee -a $LOG
  if [ "$HTTP_CODE" != "$THRESHOLD_CODE" ]; then
    echo "🚨 ALERT FIRED: HTTP $HTTP_CODE at $TS — SLO BREACH DETECTED"
    echo "   → Trigger PagerDuty / Slack notification"
    echo "   → Severity: SEV2 (payment API down)"
  fi
  sleep 1
done
EOF
chmod +x health-check.sh
./health-check.sh

Bước 4 — Mitigate: rollback to v1

echo "=== INCIDENT RESPONSE: Rolling back to v1 ==="
ROLLBACK_START=$(date +%s)

docker stop payment-api-v2
docker run -d --name payment-api-v1-restore \
  -p 8080:80 \
  --label version=v1 \
  nginx:alpine

ROLLBACK_END=$(date +%s)
TTM=$((ROLLBACK_END - ROLLBACK_START))
echo "Rollback completed in ${TTM}s (MTTR: ${TTM}s)"

# Verify recovery
sleep 2
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/)
echo "Post-rollback health check: HTTP $HTTP_CODE"
if [ "$HTTP_CODE" = "200" ]; then
  echo "✅ SERVICE RESTORED — Incident resolved"
  echo "   → Update status page, notify stakeholders"
fi

Bước 5 — Document incident ticket (CLI)

cat > incident-$(date +%Y%m%d).md << 'EOF'
# Incident Report — INC-2026-001

**Status:** Resolved
**Severity:** SEV2
**Service:** payment-api
**Date:** $(date +%Y-%m-%d)
**IC:** [email protected]
**Duration:** ~3 minutes

## Timeline
- T+00:00 — v2 deployed to production
- T+00:10 — Health check fired: HTTP 500 (availability SLO breach)
- T+00:15 — On-call acknowledged, war room opened
- T+01:00 — Root cause identified: nginx config bug in v2
- T+02:30 — Rollback to v1 initiated
- T+03:00 — Service restored, SLO back to normal

## Root Cause
Nginx config in v2 image always returned HTTP 500 due to misconfigured `return` directive.
Missing pre-deploy smoke test in CI/CD pipeline.

## Impact
- Duration: ~3 minutes
- Affected: All payment API requests
- Revenue impact: ~$5,000 (estimated)

## Action Items
| # | Action                              | Owner         | Deadline    |
|---|-------------------------------------|---------------|-------------|
| 1 | Add smoke test to CI pipeline       | dev-team      | 2026-05-30  |
| 2 | Add canary deployment for v2+       | platform-team | 2026-06-06  |
| 3 | Review deployment checklist         | tech-lead     | 2026-05-27  |
EOF
echo "Incident ticket created: incident-$(date +%Y%m%d).md"

✅ Kết quả mong đợi: Health check script phát hiện HTTP 500 đúng. Rollback khôi phục HTTP 200 trong <30s. File incident report đủ timeline + action items. cat incident.log hiển thị rõ thời điểm breach.

🧹 Cleanup: docker stop payment-api-v1-restore && docker rm payment-api-v1 payment-api-v2 payment-api-v1-restore 2>/dev/null; docker rmi payment-api:v2-buggy; cd ~ && rm -rf ~/incident-lab

🎯 Mục tiêu: Áp dụng Five Whys để tìm root cause, viết postmortem chuẩn Google SRE với action items có người phụ trách và deadline.

🧰 Công cụ / nền tảng: VS Code, Git, terminal.

📦 Chuẩn bị: Git repo. Dùng incident từ LAB-113 làm bối cảnh.

mkdir -p ~/postmortem-lab && cd ~/postmortem-lab
git init
git config user.name "SRE On-Call" && git config user.email "[email protected]"

▶️ Các bước:

Bước 1 — Tạo postmortem template và điền nội dung

cat > postmortem-INC-2026-001.md << 'POSTMORTEM'
# Postmortem — INC-2026-001: Payment API Outage
**Date:** 2026-05-23 | **Duration:** 3 min | **Severity:** SEV2 | **Author:** platform-on-call

> ⚠️ This is a BLAMELESS postmortem. The goal is to identify system weaknesses,
> not to assign individual blame.

---

## Summary
Deployment of payment-api v2 caused all payment requests to return HTTP 500
for ~3 minutes due to a misconfigured nginx reverse proxy. Service was restored
by rolling back to v1.

---

## Impact
- **Downtime:** 3 minutes (180 seconds)
- **Affected users:** ~12,000 (based on DAU × 3min/1440min)
- **Failed transactions:** ~400 requests
- **Revenue impact:** ~$5,000 (avg $12.50/transaction)
- **SLO impact:** Consumed 3 min of 43.2 min monthly error budget (6.9%)

---

## Timeline (UTC+7)
| Time     | Event                                                    |
|----------|----------------------------------------------------------|
| 14:00:00 | v2 deploy triggered via CI/CD pipeline                  |
| 14:00:10 | Health check fires: HTTP 500 detected                   |
| 14:00:15 | PagerDuty alert → on-call engineer wakes                |
| 14:00:45 | War room opened in Slack #incident-2026-001             |
| 14:01:00 | Root cause identified: nginx config bug                 |
| 14:02:30 | Rollback to v1 initiated                                |
| 14:03:00 | Service restored, HTTP 200 confirmed                    |
| 14:03:15 | Status page updated, stakeholders notified              |

---

## Root Cause Analysis (Five Whys)

**Problem:** Payment API returned HTTP 500 for all requests after v2 deploy.

1. **Why?** Nginx config in v2 image had `return 500` directive.
2. **Why?** Developer tested with `nginx -t` (syntax check) but not end-to-end.
3. **Why?** CI pipeline had no smoke test stage after build.
4. **Why?** Smoke test was never added when the service was first created.
5. **Why?** Service onboarding checklist did not require smoke tests. ← ROOT CAUSE

**Root Cause:** Missing smoke test requirement in service onboarding checklist
allowed a misconfigured image to reach production.

---

## What Went Well
- Alert fired within 10 seconds of deploy (fast MTTD)
- Rollback completed in <90 seconds (fast MTTR)
- War room communication was clear and focused
- Status page was updated promptly

## What Went Wrong
- No pre-deploy smoke test in CI/CD
- No canary/staged rollout for this service
- Deployment proceeded without manual review (auto-merge)

---

## Action Items
| # | Action                                        | Type          | Owner         | P  | Deadline   |
|---|-----------------------------------------------|---------------|---------------|----|------------|
| 1 | Add smoke test (curl healthz) to CI pipeline  | Prevention    | dev-team      | P0 | 2026-05-30 |
| 2 | Implement canary deployment (10% traffic)     | Prevention    | platform-team | P1 | 2026-06-06 |
| 3 | Update service onboarding checklist           | Process       | tech-lead     | P1 | 2026-05-27 |
| 4 | Add auto-rollback on >10% error rate for 2m | Detection     | sre-team      | P2 | 2026-06-13 |

---

## Lessons Learned
- "nginx -t passes" ≠ "service works correctly". Always test HTTP response.
- Every production service needs a smoke test stage in CI before deploy gate.
- Error budget gives us a shared language: 6.9% of budget burned in one incident.
POSTMORTEM

echo "Postmortem written. Word count: $(wc -w < postmortem-INC-2026-001.md)"

Bước 2 — Commit và review

git add postmortem-INC-2026-001.md
git commit -m "postmortem: INC-2026-001 payment-api outage 2026-05-23"
git log --oneline

# Kiểm tra action items
grep -E "^\| [0-9]" postmortem-INC-2026-001.md | \
  awk -F'|' '{print "Action:"$3" Owner:"$5" Deadline:"$7}'

✅ Kết quả mong đợi: Postmortem có đủ 6 mục (Summary, Impact, Timeline, Five Whys, Action Items, Lessons). 4 action items với owner + deadline rõ ràng. Commit thành công. Grep action items hiển thị đúng format.

🧹 Cleanup: cd ~ && rm -rf ~/postmortem-lab hoặc push lên GitHub làm portfolio.

🎯 Mục tiêu: Xây dựng runbook on-call đầy đủ cho payment API với script kiểm tra tự động, quy trình escalation và hướng dẫn xử lý 5 sự cố thường gặp.

🧰 Công cụ / nền tảng: VS Code, Git, bash, curl, Docker (tuỳ chọn).

📦 Chuẩn bị: Git repo sẵn có. Terminal WSL2/Ubuntu.

mkdir -p ~/runbook-lab && cd ~/runbook-lab && git init

▶️ Các bước:

Bước 1 — Tạo diagnostic script

cat > diagnose-payment-api.sh << 'EOF'
#!/bin/bash
# ============================================================
# Runbook Diagnostic Script — payment-api
# Run this FIRST when you receive a payment-api alert
# ============================================================
set -euo pipefail

API_HOST="${API_HOST:-http://localhost:8080}"
DB_HOST="${DB_HOST:-localhost}"
DB_PORT="${DB_PORT:-5432}"

echo "======================================================"
echo "  PAYMENT-API DIAGNOSTIC REPORT — $(date)"
echo "======================================================"

# 1. Health endpoint
echo ""
echo "--- [1/5] Health Check ---"
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "${API_HOST}/healthz" || echo "TIMEOUT")
LATENCY=$(curl -s -o /dev/null -w "%{time_total}" --max-time 5 "${API_HOST}/healthz" 2>/dev/null || echo "N/A")
echo "HTTP Status: $HTTP_CODE | Latency: ${LATENCY}s"
[ "$HTTP_CODE" != "200" ] && echo "  ⚠️  SERVICE UNHEALTHY" || echo "  ✅ Service healthy"

# 2. Database connectivity
echo ""
echo "--- [2/5] Database Connectivity ---"
if command -v pg_isready &>/dev/null; then
  pg_isready -h "$DB_HOST" -p "$DB_PORT" -q && \
    echo "  ✅ PostgreSQL reachable at ${DB_HOST}:${DB_PORT}" || \
    echo "  ❌ PostgreSQL NOT reachable — check DB_HOST/DB_PORT"
else
  nc -zw3 "$DB_HOST" "$DB_PORT" 2>/dev/null && \
    echo "  ✅ Port ${DB_PORT} open on ${DB_HOST}" || \
    echo "  ❌ Port ${DB_PORT} NOT reachable on ${DB_HOST}"
fi

# 3. Recent error rate from container logs
echo ""
echo "--- [3/5] Recent Error Rate (last 100 log lines) ---"
if docker ps --format '{{.Names}}' 2>/dev/null | grep -q "payment"; then
  CONTAINER=$(docker ps --format '{{.Names}}' | grep payment | head -1)
  TOTAL=$(docker logs "$CONTAINER" 2>&1 | tail -100 | wc -l)
  ERRORS=$(docker logs "$CONTAINER" 2>&1 | tail -100 | grep -cE "ERROR|500|panic" || true)
  echo "  Last 100 lines: $TOTAL | Errors: $ERRORS"
  (( ERRORS > 5 )) && echo "  ⚠️  High error count in logs" || echo "  ✅ Error count within normal range"
else
  echo "  (No Docker container named 'payment*' found — skip)"
fi

# 4. Disk space
echo ""
echo "--- [4/5] Disk Space ---"
df -h / | awk 'NR==2 {
  used=$5+0;
  if (used > 90) print "  ❌ CRITICAL: Disk " used "% full";
  else if (used > 75) print "  ⚠️  WARNING: Disk " used "% full";
  else print "  ✅ Disk " used "% used — OK";
}'

# 5. Memory pressure
echo ""
echo "--- [5/5] Memory ---"
FREE_MB=$(free -m | awk '/^Mem/ {print $4}')
echo "  Free memory: ${FREE_MB} MB"
(( FREE_MB < 200 )) && echo "  ⚠️  Low memory — possible OOM risk" || echo "  ✅ Memory OK"

echo ""
echo "======================================================"
echo "  Diagnostic complete. See runbook for next steps."
echo "======================================================"
EOF
chmod +x diagnose-payment-api.sh

Bước 2 — Viết runbook document

cat > runbook-payment-api.md << 'EOF'
# Runbook — payment-api On-Call Guide
**Owner:** platform-team | **Last updated:** 2026-05-23 | **Version:** 1.2

## Quick Reference
- **Service URL:** https://api.company.com/payment
- **Health check:** `curl https://api.company.com/payment/healthz`
- **Dashboard:** https://grafana.internal/d/payment-api
- **Logs:** `kubectl logs -n prod -l app=payment-api --tail=200 -f`
- **On-call rotation:** PagerDuty → @platform-oncall
- **Escalation:** L1 (on-call) → L2 (platform-team-lead) → L3 (CTO)

## Step 1: First Response Checklist
Run diagnostic script FIRST:
```bash
./diagnose-payment-api.sh
```
Then check Grafana dashboard for SLI overview.

---

## Symptom Guide

### 🔴 S1: Service completely down (HTTP 503/timeout)
**Check:**
```bash
curl -v https://api.company.com/payment/healthz
kubectl get pods -n prod -l app=payment-api
kubectl describe pod -n prod -l app=payment-api | grep -A5 Events
```
**Fix options:**
1. Recent deploy → rollback: `kubectl rollout undo deploy/payment-api -n prod`
2. Pod crashloop → check logs: `kubectl logs -n prod -l app=payment-api --previous`
3. All pods evicted → check node resources: `kubectl top nodes`

---

### 🟠 S2: High error rate >1% (HTTP 500s)
**Check:**
```bash
# Last 5 min error rate
kubectl logs -n prod -l app=payment-api --since=5m | grep -c "ERROR\|500\|panic"
# DB connection errors?
kubectl logs -n prod -l app=payment-api --since=5m | grep -i "connection refused\|timeout"
```
**Fix options:**
1. DB connection pool exhausted → restart app: `kubectl rollout restart deploy/payment-api -n prod`
2. Downstream service down → check dependencies, enable circuit breaker
3. Memory leak → check pod memory: `kubectl top pods -n prod`

---

### 🟡 S3: High latency p99 >500ms
**Check:**
```bash
# Check if DB is slow
kubectl exec -n prod deploy/payment-api -- \
  psql $DATABASE_URL -c "SELECT pid,query,state,query_start FROM pg_stat_activity WHERE state='active';"
# Check if external API is slow
curl -w "DNS: %{time_namelookup}s | Connect: %{time_connect}s | Total: %{time_total}s\n" \
  -s -o /dev/null https://external-payment-gateway.com/health
```
**Fix options:**
1. Slow DB queries → check pg_stat_activity, kill long-running queries
2. External gateway slow → enable fallback/timeout, alert vendor

---

### 🟡 S4: Memory leak / OOM
**Check:**
```bash
kubectl top pods -n prod -l app=payment-api
kubectl get events -n prod --sort-by='.lastTimestamp' | grep -i oom
```
**Fix:** Restart pod gracefully: `kubectl delete pod -n prod -l app=payment-api`

---

### ⚪ S5: Certificate expired
**Check:**
```bash
echo | openssl s_client -connect api.company.com:443 2>/dev/null | \
  openssl x509 -noout -dates
```
**Fix:** Trigger cert-manager renewal: `kubectl annotate cert payment-api-tls cert-manager.io/renew=true -n prod`

---

## Escalation Matrix
| Condition                      | Action                              |
|-------------------------------|-------------------------------------|
| Can't resolve in 15 min        | Page L2: platform-team-lead         |
| Revenue impact > $10,000       | Page L3: CTO immediately            |
| Data loss suspected            | SEV1: all hands + legal notification|
| SLO budget burned > 50%        | Freeze deploys, notify product mgr  |

## Post-Incident
1. Update incident ticket with resolution
2. Update status page: https://status.company.com
3. Schedule postmortem within 48h (see postmortem template)
EOF
echo "Runbook created."

Bước 3 — Test diagnostic script và commit

# Chạy diagnostic (sẽ fail DB/Docker check nếu không có service thật — đó là OK)
./diagnose-payment-api.sh

# Commit runbook
git add .
git commit -m "ops: add payment-api on-call runbook v1.2"
git log --oneline

# Verify runbook structure
echo "=== Runbook sections ===" && grep "^##" runbook-payment-api.md
echo "=== Symptom guides ===" && grep "^### " runbook-payment-api.md

✅ Kết quả mong đợi: diagnose-payment-api.sh chạy không lỗi bash syntax, output 5 section rõ ràng. Runbook có Quick Reference + 5 symptom guides + escalation matrix. git log hiển thị 1 commit. Section headers đúng cấu trúc.

🧹 Cleanup: cd ~ && rm -rf ~/runbook-lab (hoặc push GitHub làm portfolio SRE).