High Availability (vSphere HA)

1. HA Admission Control Policy — Design Cho Production

Admission Control đảm bảo cluster luôn có đủ capacity để restart VMs sau khi một số host fail. Chọn sai policy → VMs không restart được sau failover.

### PowerCLI — Cấu hình HA Admission Control (Cluster Resource %)
Connect-VIServer vcsa-01.lab.local

$cluster = Get-Cluster "CL-HN-Prod-01"

# Cấu hình: cho phép 1 host failure, reserve 25% CPU + RAM
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.DasConfig = New-Object VMware.Vim.ClusterDasConfigInfo
$spec.DasConfig.Enabled = $true
$spec.DasConfig.AdmissionControlEnabled = $true
$spec.DasConfig.AdmissionControlPolicy = New-Object VMware.Vim.ClusterFailoverResourcesAdmissionControlPolicy
$spec.DasConfig.AdmissionControlPolicy.CpuFailoverResourcesPercent    = 25
$spec.DasConfig.AdmissionControlPolicy.MemoryFailoverResourcesPercent = 25
$spec.DasConfig.AdmissionControlPolicy.FailoverLevel = 1  # Chịu được 1 host fail
$spec.DasConfig.DefaultVmSettings = New-Object VMware.Vim.ClusterDasVmSettings
$spec.DasConfig.DefaultVmSettings.RestartPriority = "high"
$spec.DasConfig.DefaultVmSettings.IsolationResponse = "none"

($cluster | Get-View).ReconfigureComputeResource_Task($spec, $true)

2. Heartbeat Datastore Configuration — Tránh False Positive Isolation

Datastore heartbeat là cơ chế backup để phân biệt "host thực sự bị isolated khỏi management network" với "host đã chết hoàn toàn".

### Cấu hình Heartbeat Datastores (vSphere Client + PowerCLI)
# vSphere Client:
# Cluster → Configure → vSphere Availability → Heartbeat Datastores
# → Use datastores only from the specified list → Add
# → Chọn 2 datastores trên 2 storage controller khác nhau

### PowerCLI — Đặt heartbeat datastores
$cluster = Get-Cluster "CL-HN-Prod-01"
$hbDS1   = Get-Datastore "VMFS-SAN-FC-LUN01"
$hbDS2   = Get-Datastore "NFS-NetApp-Vol01"

$clusterView = $cluster | Get-View
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.DasConfig = New-Object VMware.Vim.ClusterDasConfigInfo
$spec.DasConfig.HBDatastoreCandidatePolicy = "userSelectedDs"
$spec.DasConfig.HeartbeatDatastore = @($hbDS1.Id, $hbDS2.Id)
$clusterView.ReconfigureComputeResource_Task($spec, $true)

### Xác nhận heartbeat status
$cluster | Get-View | Select-Object -ExpandProperty Summary |
    Select-Object -ExpandProperty DasData |
    Select-Object HeartbeatDatastoreInfo

3. APD / PDL Response Policy — VM Component Protection (VMCP)

VMCP phản ứng khi storage path bị mất — phân biệt APD (All Paths Down — tạm thời) vs PDL (Permanent Device Loss — mất hẳn).

### PowerCLI — Cấu hình VMCP (APD + PDL response)
$cluster = Get-Cluster "CL-HN-Prod-01"
$view    = $cluster | Get-View
$spec    = New-Object VMware.Vim.ClusterConfigSpecEx
$spec.DasConfig = New-Object VMware.Vim.ClusterDasConfigInfo

# PDL: Power off VMs ngay khi detect PDL
$spec.DasConfig.VmComponentProtecting = "enabled"
$spec.DasConfig.DefaultVmSettings = New-Object VMware.Vim.ClusterDasVmSettings
$spec.DasConfig.DefaultVmSettings.VmComponentProtectionSettings = `
    New-Object VMware.Vim.ClusterVmComponentProtectionSettings
$spec.DasConfig.DefaultVmSettings.VmComponentProtectionSettings.VmStorageProtectionForPDL = `
    "restartAggressive"     # Power off VM → restart on healthy host

# APD: Restart VMs sau 140 giây (default) nếu APD không phục hồi
$spec.DasConfig.DefaultVmSettings.VmComponentProtectionSettings.VmStorageProtectionForAPD = `
    "restartConservative"
$spec.DasConfig.DefaultVmSettings.VmComponentProtectionSettings.VmTerminateDelayForAPDSec = 140

$view.ReconfigureComputeResource_Task($spec, $true)
Write-Host "VMCP configured: PDL=restartAggressive, APD=restartConservative(140s)"

4. Network Isolation Response — Tránh Split-Brain

Khi host mất management network, vSphere HA cần quyết định: shutdown VMs trên host isolated hay để chạy? Chọn sai → VMs chạy song song trên 2 host (split-brain).

5. HA Event Monitoring & Alerting Setup

Setup alerts tự động cho các HA events quan trọng — IT team được thông báo ngay khi có host failure hoặc VM restart do HA.

### vSphere Client — Tạo Alarm cho HA Events
# Cluster → Configure → Alarm Definitions → Add

## Alarm 1: Host bị HA isolate
Name:       "HA Host Isolation Detected"
Target:     Host
Trigger:    Event — com.vmware.vc.ha.HostIsolatedEvent
Action:     Send Email → [email protected]
            Run Script → /scripts/notify-oncall.sh

## Alarm 2: VM được HA restart
Name:       "HA VM Restart"
Target:     Virtual Machine
Trigger:    Event — com.vmware.vc.ha.VmRestartedByHAEvent
Action:     Send Email + SNMP Trap đến monitoring (Zabbix/PRTG)

## Alarm 3: HA insufficient failover capacity
Name:       "HA Admission Control Violated"
Target:     Cluster
Trigger:    Metric — vSphere HA.vSphere HA host status = Insufficient capacity
Threshold:  = true (any violation)
Action:     Send Email + PagerDuty webhook (critical)

### PowerCLI — Lấy lịch sử HA Events (audit log)
Connect-VIServer vcsa-01.lab.local

$start  = (Get-Date).AddDays(-7)
$haEvents = @(
    "com.vmware.vc.ha.VmRestartedByHAEvent",
    "com.vmware.vc.ha.HostIsolatedEvent",
    "com.vmware.vc.ha.VmFailoverFailed"
)

$events = Get-VIEvent -Start $start -MaxSamples 1000 |
          Where-Object { $haEvents -contains $_.EventTypeId }

$events | Select-Object CreatedTime, FullFormattedMessage, ObjectName |
          Format-Table -AutoSize |
          Tee-Object -FilePath "ha-events-$(Get-Date -f yyyyMMdd).txt"

Write-Host "Tổng HA events trong 7 ngày: $($events.Count)"

6. vSphere HA Production Checklist

Checklist kiểm tra hoàn chỉnh sau khi enable hoặc modify HA configuration trên production cluster.