Proving It from Outside the Cluster: The AKS Lab

This is the lab that ties the whole series together, because it forces the hardest version of the central question. In Kubernetes, the control plane is an enthusiastic optimist. It will tell you pods are running, deployments are available, and everything is green, while users cannot reach the service at all. The AKS lab injects three realistic failures into a live cluster and then proves the symptom from outside, using an independent observer that the control plane cannot talk out of reporting the truth.

What the lab stands up

The cluster is long-lived, meant to stay deployed so operators can run drills whenever they want rather than waiting on a fresh provision. It is built on Azure CNI overlay with the Cilium dataplane, and it separates a system node pool from a dedicated lab user pool, so faults land on lab nodes without disturbing cluster services.

Inside a single namespace, a tiny demo-api deployment runs two replicas behind an internal load balancer, protected by a pod disruption budget and spread across nodes by anti-affinity. The app itself is deliberately trivial, a single-file Python server with a FAILURE_MODE environment variable and a /healthz endpoint, because the cluster is the system under test, not the application.

The piece that makes the lab honest is the networking. A delegated subnet hosts private Azure Container Instances behind a NAT gateway. Those ACI runners are the independent observers. They sit inside the VNet and probe the internal service the way a real caller would.

The three failures

Each scenario breaks the cluster in a way that maps to a real production incident.

node-drain drains one schedulable node from the lab pool. With anti-affinity spreading the two replicas and a disruption budget holding minAvailable: 1, this tests whether the workload reschedules and stays available, or whether the budget and a single drained node interact in a way you did not predict.

network-isolate applies a deny-all ingress network policy to demo-api. The pods stay running and healthy. The control plane stays content. Nothing can reach them. This is the “up but unreachable” failure from the network lab, now in its purest Kubernetes form, and it is the one the control plane is least willing to admit to.

crashloop patches the deployment to FAILURE_MODE=crashloop, which makes the container exit on startup, and you watch CrashLoopBackOff ripple through the pods.

The signals that prove it

Three scheduled-query alerts back the three failures, and the queries are specific.

A ready-replica query against KubePodInventory fires when fewer than two demo-api pods are actually running:

KubePodInventory
| where TimeGenerated > ago(5m)
| where ClusterName =~ '<cluster-name>'
| where Namespace =~ 'aks-failure-lab'
| where ContainerName has '/demo-api'
| summarize ReadyPods = dcountif(PodUid, PodStatus =~ 'Running' and ContainerStatus =~ 'running')
| where ReadyPods < 2

A crashloop query watches for crash states with a rising restart count:

KubePodInventory
| where TimeGenerated > ago(5m)
| where ClusterName =~ '<cluster-name>'
| where Namespace =~ 'aks-failure-lab'
| where ContainerName has '/demo-api'
| where ContainerStatusReason in ('CrashLoopBackOff', 'Error')
| summarize RestartCount = max(ContainerRestartCount)
| where RestartCount > 0

And the one that makes the lab worth building, a synthetic-runner query against the ACI container logs:

ContainerInstanceLog_CL
| where TimeGenerated > ago(5m)
| where ContainerGroup_s startswith 'wisdm-aks-runner-'
| where Message has '"ok": false'
| summarize FailureCount = count() by ContainerGroup_s
| where FailureCount > 3

That third query is the point of the whole series in one place. When network-isolate cuts ingress, the pods still report healthy to the control plane, so a pod-status alert can stay quiet. The ACI runner, probing from inside the VNet along the real traffic path, starts failing its checks, and that failure is what fires. You are not asking Kubernetes whether Kubernetes is fine. You are asking an independent observer whether the service is reachable. That is the difference between monitoring that catches real outages and monitoring that catches only the ones the platform is willing to confess to.

Recovery

recover uncordons the lab nodes, removes the deny policy, and restores the healthy deployment environment in a single call. Baseline is two ready replicas, ingress open, the ACI runners passing their checks, and no crashloop states in the inventory.

The lesson, and the series in one line

Build the independent observer. The control plane is a participant in the system you are trying to monitor, and a participant should never be your only witness. The ACI runner exists precisely because pod status can be green while the service is dark, and the only way to catch that gap is to watch from outside, along the path a user actually takes.

That is the thread running through all eleven labs. The CPU lab watches load, not liveness. The network lab pairs a symptom with a cause. The dependency lab watches the caller, not the callee. The queue lab watches depth, where there is no error to catch. Every one of them is the same move: do not trust the thing you are monitoring to report on its own health. Inject the failure, then prove from the outside that you would have known. The injection was always the easy half.

This concludes the Azure Failure Labs series. The full set: CPU, network, App Service, dependency, auth and secret, database, memory and crash, quota and throttling, TLS, queue, and AKS, each with a Logic App controller, an inject contract, a recover path, and standardized alerting against a shared workspace and action group.