Triển khai & Vận hành phần mềm

1. Lý thuyết cốt lõi / Core theory

1.1. Secure Deployment Strategies (Blue-Green / Canary / Rolling)

Blue-Green deployment: duy trì hai môi trường production identical (Blue = current, Green = new). Deploy lên Green → test → switch traffic. Security benefit: rollback ngay lập tức nếu phát hiện security issue; Green environment có thể được scanned trước khi nhận traffic. Risk: cần maintain đồng bộ secrets/config giữa hai env; cost gấp đôi infrastructure. Canary deployment: route % nhỏ traffic (5-10%) sang version mới, monitor metrics, dần tăng tỷ lệ. Security benefit: phát hiện security regression ở scale nhỏ trước khi full rollout; dễ detect anomalies như tăng error rate hoặc suspicious calls. Feature flags: enable/disable features runtime mà không redeploy — security benefit: có thể disable vulnerable feature ngay lập tức mà không cần hotfix + redeploy cycle.

1.2. Container Runtime Security (Falco / AppArmor / seccomp)

Falco (CNCF): runtime security tool cho containers và Kubernetes — monitor system calls và Kubernetes audit logs, alert khi detect suspicious behavior (container spawning shell, reading sensitive files, privilege escalation). Rules dạng: "Alert khi process bên trong container gọi execve với binary lạ". AppArmor: Linux Security Module — restrict what programs can do via profiles (allow/deny specific file access, network, capabilities). Docker: docker run --security-opt apparmor=docker-default. seccomp (Secure Computing Mode): whitelist system calls mà container được phép gọi — loại bỏ attack surface bằng cách block syscalls không cần thiết. Read-only filesystem: docker run --read-only — container không thể write ra filesystem (trừ explicitly mounted volumes), ngăn attacker persist malware.

1.3. Secrets Rotation trong Production (Zero-Downtime Rotation)

Secrets rotation là critical practice: database passwords, API keys, TLS certificates phải rotate định kỳ hoặc khi suspect compromise. Zero-downtime rotation pattern: (1) Create new secret version; (2) Update application to accept both old AND new; (3) Gradually migrate connections to new secret; (4) Revoke old secret. Tools: HashiCorp Vault dynamic secrets — generate unique, short-lived credentials per application instance (no shared passwords), auto-expire; AWS Secrets Manager tự động rotate RDS passwords theo schedule với Lambda function; Kubernetes Secrets + External Secrets Operator sync từ Vault/AWS SM. Certificate rotation: cert-manager (Kubernetes) tự động renew Let's Encrypt certificates trước khi expire.

1.4. Security Monitoring & APM (Application Security Monitoring)

APM + Security: Application Performance Monitoring tools (Datadog, New Relic, Dynatrace) thêm security signals: detect sudden spike trong 4xx/5xx errors (attack indicator), anomalous API call patterns, geographic anomalies (user từ VN bỗng dùng account từ Russia), slow queries (SQL injection probe), large data exports. Security-specific signals: failed authentication rate (brute force), account enumeration (same IP nhiều username), privilege escalation attempts (user truy cập admin endpoints), data exfiltration (unusually large response size). RASP (Runtime Application Self-Protection) instrument vào application runtime, detect và block attacks in real-time: SQLi, XSS, path traversal — không cần WAF rule update.

1.5. Application Incident Response & Decommissioning (IR + Secure EOL)

Application breach IR: Contain (isolate affected service, disable compromised accounts, enable enhanced logging); Eradicate (identify và remove backdoors/webshells, patch vulnerability, rotate ALL secrets in affected app); Recover (redeploy từ known-good IaC, verify integrity); Post-incident (root cause analysis, update threat model, add detection rules). Serverless security operations: Lambda/Azure Functions — shorter attack surface (no persistent server), nhưng function permissions (IAM roles) cần least privilege; audit CloudTrail/Activity Log cho function invocations. Application decommissioning: khi retire app, revoke tất cả credentials và API keys; delete data theo retention policy; remove DNS entries; archive source code securely; notify third parties integrating với app.

1.6. MTTA/MTTR & SOAR KPIs (Security Operations KPIs)

Hai chỉ số KPI quan trọng nhất để đo hiệu quả vận hành bảo mật và ROI của SOAR (Security Orchestration, Automation & Response):

• MTTA (Mean Time to Acknowledge): thời gian trung bình từ khi alert được tạo đến hành động đầu tiên (tự động hoặc con người). Mục tiêu: < 5 phút. Công thức: MTTA = Σ(ack_time − alert_time) / n
• MTTR (Mean Time to Respond): thời gian từ khi phát hiện đến khi remediation hoàn tất. Mục tiêu: < 4 giờ với SOAR automation.

2. Bài thực hành / Hands-on labs

Lab 1 — Secrets Rotation Simulation & IaC Drift Detection (PowerShell)

OS: Windows 11 · Tool: PowerShell 7.

# Lab: Simulate secrets rotation workflow và IaC drift check

Write-Host "=== Secrets Rotation Simulation ===" -ForegroundColor Cyan

# Simulate current secrets state
$secrets = @{
    "DB_PASSWORD"    = @{ Value="OldPass123!"; Created="2026-02-01"; AgeInDays=113 }
    "API_KEY_STRIPE" = @{ Value="sk_live_xxxx"; Created="2026-04-15"; AgeInDays=39  }
    "JWT_SECRET"     = @{ Value="MyJWTSecret";  Created="2025-11-01"; AgeInDays=204 }
}

# Rotation policy: rotate if older than 90 days
$rotationThresholdDays = 90
Write-Host "`nSecret Rotation Status:"
$needsRotation = @()
foreach ($name in $secrets.Keys) {
    $s = $secrets[$name]
    $status = if ($s.AgeInDays -gt $rotationThresholdDays) { "NEEDS ROTATION"; $needsRotation += $name; "OVERDUE" } else { "OK" }
    $color  = if ($s.AgeInDays -gt $rotationThresholdDays) { "Red" } else { "Green" }
    Write-Host "  [$status] $name — Age: $($s.AgeInDays) days (threshold: $rotationThresholdDays)" -ForegroundColor $color
}

if ($needsRotation.Count -gt 0) {
    Write-Host "`n[ACTION REQUIRED] Rotate these secrets immediately:" -ForegroundColor Red
    $needsRotation | ForEach-Object { Write-Host "  - $_" -ForegroundColor Red }
}

# Simulate IaC drift detection
Write-Host "`n=== IaC Drift Detection (Terraform Plan simulation) ===" -ForegroundColor Cyan
$desiredState   = @{ "SecurityGroup.Port443" = "OPEN"; "SecurityGroup.Port22"  = "CLOSED"; "S3Bucket.PublicAccess" = "BLOCKED" }
$currentState   = @{ "SecurityGroup.Port443" = "OPEN"; "SecurityGroup.Port22"  = "OPEN";   "S3Bucket.PublicAccess" = "BLOCKED" }

Write-Host "`nDrift Analysis (IaC desired vs actual production):"
$driftFound = $false
foreach ($resource in $desiredState.Keys) {
    if ($desiredState[$resource] -ne $currentState[$resource]) {
        Write-Host "  [DRIFT] $resource`: desired=$($desiredState[$resource]), actual=$($currentState[$resource])" -ForegroundColor Red
        $driftFound = $true
    } else {
        Write-Host "  [OK]    $resource`: $($desiredState[$resource])" -ForegroundColor Green
    }
}
if ($driftFound) { Write-Host "`n[ALERT] Configuration drift detected — run terraform apply to remediate!" -ForegroundColor Red }

Lab 2 — Container Runtime Security với Falco (Bash)

OS: Ubuntu 22.04 · Tool: Bash + Docker + Falco.

# Lab: Container Security Hardening + Falco monitoring

echo "=== Container Security Hardening Demo ==="

# 1. Run container với security profiles
echo "--- Running hardened container ---"
docker run --rm -d \
    --name secure-app \
    --read-only \
    --tmpfs /tmp:size=50m \
    --security-opt no-new-privileges \
    --cap-drop ALL \
    --cap-add NET_BIND_SERVICE \
    --memory 256m \
    --cpus 0.5 \
    nginx:alpine 2>/dev/null && echo "Secure container started" || echo "Docker not available"

# 2. Verify security settings
echo "--- Security config verification ---"
docker inspect secure-app 2>/dev/null | python3 -c "
import json, sys
try:
    data = json.load(sys.stdin)[0]
    hc = data['HostConfig']
    print(f'ReadonlyRootfs: {hc.get(\"ReadonlyRootfs\", False)}')
    print(f'NoNewPrivileges: {hc.get(\"SecurityOpt\", [])}')
    print(f'CapDrop: {hc.get(\"CapDrop\", [])}')
    print(f'Memory: {hc.get(\"Memory\", 0) // 1024 // 1024} MB')
except: print('Container not running or Docker unavailable')
" 2>/dev/null

docker stop secure-app 2>/dev/null

# 3. Falco rules sample (educational — shows detection logic)
echo ""
echo "=== Sample Falco Security Rules ==="
cat << 'FALCO'
# Rule 1: Detect shell spawned in container (potential breach indicator)
- rule: Shell Spawned in Container
  desc: A shell was spawned inside a container — investigate immediately
  condition: spawned_process and container and shell_procs
  output: "Shell spawned in container (user=%user.name container=%container.name
           cmd=%proc.cmdline)"
  priority: WARNING

# Rule 2: Sensitive file read (credential theft attempt)
- rule: Read Sensitive File Untrusted
  desc: An untrusted process read a sensitive file like /etc/shadow
  condition: open_read and sensitive_files and not trusted_containers
  output: "Sensitive file opened for reading (file=%fd.name container=%container.name)"
  priority: ERROR

# Rule 3: Outbound connection from container to unexpected IP
- rule: Unexpected Outbound Network Connection
  desc: Container making outbound connection to non-whitelisted destination
  condition: outbound and container and not allowed_external_ips
  output: "Unexpected outbound connection (dest=%fd.rip container=%container.name)"
  priority: WARNING
FALCO

# Check if Falco is installed
if command -v falco &>/dev/null; then
    echo "--- Falco status ---"
    systemctl is-active falco 2>/dev/null || falco --version
else
    echo "Install Falco: curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg"
fi

3. Tình huống doanh nghiệp / Real-world scenario

4. Tự kiểm tra / Knowledge check

1. Blue-green deployment giảm thiểu security risk như thế nào so với in-place deployment?
2. IaC configuration drift là gì và tại sao là security risk? Công cụ nào phát hiện drift?
3. Falco hoạt động dựa trên cơ chế gì? Cho ví dụ một Falco rule và event nó detect.
4. Zero-downtime secrets rotation cần bao nhiêu bước? Tại sao không thể chỉ swap secret trực tiếp?
5. Khi phát hiện container breach, hành động đầu tiên là gì và tại sao quan trọng?