Most teams hold two beliefs about production that they have never tested. The first is that their application degrades gracefully when something it depends on fails. The second is that their monitoring fires when that happens. Both beliefs are free, and both are usually wrong in some specific, surprising way that you only discover during a real incident at a bad hour.
The fix is not a one-off chaos session. A single kubectl delete pod during a brown-bag lunch proves nothing you can repeat. What you want is a harness: the same contract for every failure type, so anyone can run a drill on demand and compare results from one run to the next. This series is about that harness. It covers eleven failure labs on Azure, each one a single HTTP POST away from breaking, and each one with a matching button to put things back.
The one idea worth keeping
Injecting failure is the easy half. The hard half is proving you would have noticed. Every lab in this series pairs a fault with an independent signal that moves when the fault is live. If you take nothing else from the series, take that.
The shape every lab shares
The point of a harness is sameness. Eleven different failure modes, one contract.
- A Logic App controller is the single entry point. You resolve its signed callback URL, POST a payload, and it executes the fault through a managed identity. No standing credentials live in the repo.
- Every controller supports at least one inject scenario and a
recover. Recovery is not optional. A fault you cannot cleanly reverse is not a drill, it is an outage you caused on purpose. - The fault surfaces in Azure Monitor plus a Log Analytics scheduled-query alert in one shared workspace, and every alert fires into one shared action group. That last part matters, because it means each drill exercises the same notification path a real alert would travel.
Here is the contract as a diagram.
The drill, every time
The validation flow is identical across all eleven labs. The steps never change, which is what makes the results comparable.
- Confirm the healthy baseline.
- POST the failure payload.
- Confirm the symptom appears.
- Confirm the alert signal starts moving.
- POST
recover. - Confirm health, metrics, and state return to baseline.
Step 4 is the whole reason the harness exists. Anyone can make an app return errors. The question that matters is whether your monitoring would have caught it, and the only honest way to answer is to watch the alert fire.
Here is that flow as a sequence.
The eleven labs, grouped
The labs fall into three families. The series walks them in this order.
Infrastructure. A CPU lab drives VMs past their utilization thresholds through Run Command. A network lab deletes a network security group allow-rule to sever an HTTP path while the VM stays powered on.
Application. This is the densest family: deterministic 500s, unhandled exceptions, and injected latency on an App Service; a broken downstream dependency; Key Vault secret and identity failures; bad database credentials and slow queries; memory pressure and crash loops; self-imposed 429 throttling; and a TLS lab that swaps a dependency from a trusted certificate to an invalid one.
Integration. A queue lab grows backlog and dead-letter counts on Service Bus. An AKS lab injects node drains, ingress isolation, and crash loops into a live cluster, then proves the symptom from outside using synthetic runners.
Why build it this way
For a team, the value is plain. You stop guessing whether your runbooks, alerts, and graceful-degradation assumptions hold, because you can rehearse them on a Tuesday afternoon instead of finding the gaps at three in the morning. Because every alert fires into the real action group, “the alert worked” actually means something.
For an engineer, building the harness is a forcing function. You cannot write a credible network-isolation drill until you understand the difference between a host being down and a host being unreachable. You cannot write the AKS lab until you can prove a service is broken even while the control plane reports it healthy. That understanding is what separates someone who has read about failures from someone who has caused and recovered from them on purpose.
The next post starts where the harness is easiest to see in action: CPU saturation, where “the box is up” and “the box is healthy” are two very different claims.
Next: Up Is Not the Same as Healthy, the CPU stress lab.