Up Is Not the Same as Healthy: The CPU Stress Lab

A virtual machine pegged at 100 percent CPU is still running. It answers pings. The Azure portal shows it green. And it may be completely useless to the people trying to reach the application on it. The gap between “the box is up” and “the box is healthy” is where a lot of incidents live, and the CPU lab exists to make that gap visible on demand.

What the lab does

It drives target VMs past their CPU thresholds using Azure VM Run Command, invoked by a Logic App through a managed identity. Run Command pushes a small stress script onto each VM (PowerShell for Windows, shell for Linux), so there is no agent to install and no long-lived configuration system in the path. When you want it to stop, a second call cleans the workload up.

The inject contract

This lab uses action rather than the scenario field the application labs use, a small inconsistency that is worth knowing if you script against several labs at once. The body names the resource group and the VMs to hit.

{
  "action": "start",
  "resourceGroup": "wisdm-poc-platform",
  "vmNames": ["wisdm-poc1-linux"]
}

Payloads exist for a single VM, all demo VMs, and Windows-only runs, so you can scope the blast radius to match what you are trying to show. Setting action to stop ends the stress and lets the hosts fall back to idle.

From POST to alert

sequenceDiagram participant Op as Operator participant LA as Logic App participant VM as Target VM participant M as Azure Monitor participant AG as Action Group Op->>LA: POST action start LA->>VM: Run Command starts stress VM-->>M: CPU utilization climbs M->>M: Scheduled query crosses 80 percent M->>AG: log-vm-high-cpu-enriched fires Op->>LA: POST action stop LA->>VM: Run Command cleans up VM-->>M: CPU returns to idle, alert resolves

The signal that proves it

The lab adds an enriched scheduled-query alert in Log Analytics that reads from InsightsMetrics and fires when average CPU crosses 80 percent over a five-minute bin.

InsightsMetrics
| where TimeGenerated > ago(5m)
| where Origin == "vm.azm.ms"
| where Namespace == "Processor" and Name == "UtilizationPercentage"
| where _ResourceId has "/resourceGroups/wisdm-poc-platform/providers/Microsoft.Compute/virtualMachines/"
| summarize AvgCPU = avg(Val) by bin(TimeGenerated, 5m), Computer, _ResourceId
| where AvgCPU > 80
| project TimeGenerated, Computer, AvgCPU, _ResourceId

The query returns the computer name and the resource ID, which is what makes the alert “enriched.” A plain threshold alert tells you something is hot. This one tells you which machine, so the person who gets paged does not start the incident by hunting for the affected host.

Recovery

Stopping is a single action: stop call against the same VM list. CPU falls back to baseline, the scheduled query stops returning rows, and the alert resolves on its own. Baseline here means the hosts are back to normal idle utilization and no enriched CPU rows appear in the workspace for the lab VMs.

The lesson

Saturation is a liveness problem disguised as a capacity problem. The VM never went down, so any check that only asks “is it running” stays green the entire time. The useful question is “can it still do work,” and answering that requires a metric that measures load, not presence. Build the alert around the thing that actually degrades, and enrich it with enough identity (which host, which resource) that the page is actionable on arrival.

Next: Running but Unreachable, the network isolation lab.