Monitoring, Alarms & Performance Tuning

1. VMware Aria Operations Deployment Sizing — Small / Medium / Large

Chọn đúng deployment size ngay từ đầu — undersized Aria Operations gây chậm dashboard, mất metrics; oversized lãng phí tài nguyên. (Nodes = ESXi hosts + VMs được monitor)

2. Custom Dashboard cho Capacity Management

Dashboard capacity management cho phép dự báo khi nào cần mua thêm tài nguyên — tránh tình trạng hết tài nguyên bất ngờ trong production.

### Tạo Capacity Management Dashboard trong vROps

# 1. Home → Dashboards → Create Dashboard
# Name: "Capacity Management — Executive View"
# Layout: 3 columns

# WIDGET 1: Cluster Remaining Capacity (Scoreboard)
# Metrics:
#   - Cluster|CPU|Capacity Remaining (%)
#   - Cluster|Memory|Capacity Remaining (%)
#   - Datastore|Disk Space|Capacity Remaining (%)
# Color: Green >40%, Yellow 20-40%, Red <20%

# WIDGET 2: Time to Exhaustion (Metric Chart trend)
# Metrics:
#   - Cluster|CPU|Time Remaining (days)
#   - Cluster|Memory|Time Remaining (days)
# Threshold line: 60 days (procurement lead time)

# WIDGET 3: Top 10 Oversized VMs (Table)
# From: vROps Reclaim → Oversized VMs
# Columns: VM Name, Current vCPU, Recommended vCPU, Wasted CPU (GHz)
# Sort by: Wasted CPU descending
# → Direct link to "Right-size" action

# WIDGET 4: Capacity Forecast (12 months)
# vROps built-in: Capacity → Potential Headroom
# Shows: current + projected usage with confidence bands

3. Alert Definition & Notification Plugins

vROps alert system vượt trội hơn vCenter alarms nhờ predictive analysis và root cause analysis tự động — giảm MTTR đáng kể.

### Tạo Custom Alert Definition trong vROps

# 1. Home → Alerts → Alert Definitions → Add
# Name: "VM CPU Ready Critical — Production"
# Base Object Type: Virtual Machine

# Symptom Set:
#   Symptom 1: CPU|CPU Ready (ms) > 500 for 5 minutes  [CRITICAL]
#   Symptom 2: CPU|Demand (MHz) > 90% of limit for 5m  [WARNING]
#   Condition: Symptom 1 OR Symptom 2

# Recommendation:
#   "Check host CPU utilization. Consider:
#    1. Reduce vCPU count of this VM
#    2. Migrate VM to less-loaded host (DRS manual)
#    3. Add host to cluster if sustained"

# Notification Plugin — Gửi alert đến Slack/Teams:
# Administration → Notifications → Outbound Settings
# → Add: Webhook notification plugin
# URL: https://hooks.slack.com/services/xxx/yyy/zzz
# Payload template:
# {"text":"*[vROps Alert]* {{alertName}}\n*Object:* {{resourceName}}\n*Severity:* {{criticality}}\n*Time:* {{startTimeUTC}}"}

# Email notification cho Critical alerts:
# Outbound Settings → SMTP → smtp.hoatranlab.io.local:587
# → Notification rule: Alert level = CRITICAL → email [email protected]

4. PowerCLI Automation Scripts cho Common Tasks

PowerCLI (hiện tại v13.x) scripts hóa các tác vụ lặp đi lặp lại — từ weekly health report đến bulk VM provisioning và compliance auditing.

### Cài đặt PowerCLI (v13.x — yêu cầu PowerShell 5.1+ hoặc PowerShell 7+)
Install-Module VMware.PowerCLI -AllowClobber -Scope CurrentUser
Get-Module VMware.PowerCLI -ListAvailable | Select Version

### Kết nối vCenter
Connect-VIServer -Server vcsa-01.lab.local -User [email protected] -Password 'VMware1!'

### Script 1: Weekly VM Health Report (gửi email tự động)
Connect-VIServer vcsa-01.lab.local -User [email protected]

$report = Get-VM | Select-Object Name,
    @{N='PowerState';E={$_.PowerState}},
    @{N='CPU_Ready_%';E={[math]::Round((Get-Stat -Entity $_ -Stat "cpu.ready.summation" -MaxSamples 1 -Realtime).Value / 200, 2)}},
    @{N='Mem_Balloon_MB';E={[math]::Round((Get-Stat -Entity $_ -Stat "mem.vmmemctl.average" -MaxSamples 1 -Realtime).Value / 1024, 0)}},
    @{N='Snapshot_Count';E={(Get-Snapshot -VM $_).Count}},
    @{N='Tools_Status';E={$_.ExtensionData.Guest.ToolsStatus}}

# Alert: VMs có snapshot >3 ngày
$oldSnapshots = Get-VM | Get-Snapshot | Where-Object {$_.Created -lt (Get-Date).AddDays(-3)}
$oldSnapshots | Format-Table VM, Name, Created, SizeGB -AutoSize

# Export & Email
$report | Export-Csv "C:\Reports\VM-Health-$(Get-Date -f yyyyMMdd).csv"
Send-MailMessage -To "[email protected]" -From "[email protected]" `
    -Subject "Weekly VM Health Report $(Get-Date -f yyyy-MM-dd)" `
    -Attachments "C:\Reports\VM-Health-$(Get-Date -f yyyyMMdd).csv" `
    -SmtpServer "smtp.hoatranlab.io.local"

### Script 2: Bulk VM Provisioning từ Template
$vmList = Import-Csv "C:\Provision\vm-list.csv"  # Columns: VMName,Template,Cluster,Datastore,Folder
$vmList | ForEach-Object {
    New-VM -Name $_.VMName -Template (Get-Template $_.Template) `
           -VMHost (Get-Cluster $_.Cluster | Get-VMHost | Get-Random) `
           -Datastore (Get-Datastore $_.Datastore) `
           -Location (Get-Folder $_.Folder) `
           -RunAsync
    Write-Host "Provisioning $($_.VMName)..."
}

5. Terraform vSphere Provider cho Infrastructure as Code

Terraform vSphere provider cho phép define và version-control toàn bộ VM infrastructure — reproducible, auditable, và tích hợp CI/CD pipeline.

### Terraform vSphere — Deploy VM Production (main.tf)

terraform {
  required_providers {
    vsphere = { source = "hashicorp/vsphere", version = "~> 2.5" }
  }
}

provider "vsphere" {
  user                 = var.vsphere_user
  password             = var.vsphere_password
  vsphere_server       = "vcsa-01.lab.local"
  allow_unverified_ssl = false
}

data "vsphere_datacenter"     "dc"      { name = "DC-HaNoi-01" }
data "vsphere_compute_cluster" "cluster" { name = "CL-HN-Prod-01"; datacenter_id = data.vsphere_datacenter.dc.id }
data "vsphere_datastore"       "ds"      { name = "ds-prod-ssd-01"; datacenter_id = data.vsphere_datacenter.dc.id }
data "vsphere_network"         "net"     { name = "PG-VLAN100-Production"; datacenter_id = data.vsphere_datacenter.dc.id }
data "vsphere_virtual_machine" "tmpl"    { name = "TMPL-RHEL9-Base"; datacenter_id = data.vsphere_datacenter.dc.id }

resource "vsphere_virtual_machine" "app_server" {
  count            = var.instance_count  # scale: terraform apply -var="instance_count=3"
  name             = "LNXAPP-${var.app_name}-PRD-0${count.index + 1}"
  resource_pool_id = data.vsphere_compute_cluster.cluster.resource_pool_id
  datastore_id     = data.vsphere_datastore.ds.id
  folder           = "/DC-HaNoi-01/vm/Production/${var.app_name}"

  num_cpus = 4; memory = 8192; guest_id = data.vsphere_virtual_machine.tmpl.guest_id

  network_interface { network_id = data.vsphere_network.net.id }
  disk { label = "disk0"; size = 100; thin_provisioned = true }

  clone {
    template_uuid = data.vsphere_virtual_machine.tmpl.id
    customize {
      linux_options { host_name = "lnxapp-${lower(var.app_name)}-prd-0${count.index + 1}"; domain = "hoatranlab.io.local" }
      network_interface { ipv4_address = "10.100.${var.vlan_octet}.${10 + count.index}"; ipv4_netmask = 24 }
      ipv4_gateway = "10.100.${var.vlan_octet}.1"
    }
  }
  tags = [vsphere_tag.env_prod.id, vsphere_tag.app.id]
}

6. Ansible VMware Modules cho Configuration Management

Ansible community.vmware collection có 100+ modules cho việc manage vSphere — idempotent, declarative và tích hợp tốt với AWX/Tower.

### Cài đặt Ansible VMware Collection
# Yêu cầu: Python 3.8+, Ansible 2.12+
ansible-galaxy collection install community.vmware
# Verify
ansible-galaxy collection list | grep vmware

### Ansible Playbook — ESXi Host Hardening (playbook-esxi-hardening.yml)
---
- name: ESXi Security Hardening
  hosts: localhost
  gather_facts: false
  vars:
    vcenter_hostname: vcsa-01.lab.local
    vcenter_username: "{{ vault_vcenter_user }}"     # Lưu trong Ansible Vault
    vcenter_password: "{{ vault_vcenter_password }}"
    datacenter: DC-HaNoi-01

  tasks:
    - name: Disable SSH on all ESXi hosts
      community.vmware.vmware_host_service_manager:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        validate_certs: true
        cluster_name: CL-HN-Prod-01
        state: absent
        service_name: TSM-SSH
      loop: "{{ groups['esxi_hosts'] }}"

    - name: Configure NTP servers
      community.vmware.vmware_host_ntp:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        validate_certs: true
        cluster_name: CL-HN-Prod-01
        state: present
        ntp_servers: ["10.100.1.10", "10.100.1.11"]

    - name: Enable Lockdown Mode (Normal)
      community.vmware.vmware_host_lockdown:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        validate_certs: true
        cluster_name: CL-HN-Prod-01
        state: normal  # normal | strict | disabled

    - name: Configure Syslog forwarding
      community.vmware.vmware_host_config_manager:
        hostname: "{{ vcenter_hostname }}"
        username: "{{ vcenter_username }}"
        password: "{{ vcenter_password }}"
        validate_certs: true
        esxi_hostname: "{{ item }}"
        options:
          'Syslog.global.logHost': "ssl://siem.hoatranlab.io.local:6514"
      loop: "{{ esxi_hosts }}"