There is a particular kind of demo that falls flat: you trigger a failure, the dashboard shows one sad red dot, and then nothing, because a single failing request does not move a chart built for production traffic. The App Service lab solves that by making failure both deterministic and busy. It can push an app into 500s, exceptions, latency, or a full stop, and it generates enough sustained traffic that the telemetry actually moves while the fault is live.
What the lab does
It deploys a small Python web app instrumented with Azure Monitor OpenTelemetry, plus a Logic App controller that flips the app between failure modes. The app keeps /healthz returning 200 even during injected failures, on purpose, so App Service health checks do not quietly recycle the instance and erase the scenario you are trying to show. It also exposes an admin load endpoint so the controller can drive continuous traffic during a drill.
The inject contract
The payload names the scenario and shapes the load. The load fields are what keep the charts populated instead of catching one lonely request.
{
"scenario": "http500",
"durationSeconds": 300,
"burstRequests": 120,
"loadDurationSeconds": 300,
"loadConcurrency": 8,
"loadIntervalMs": 100
}The scenarios are http500, exception, latency, stop, and recover. For http500 and exception, the controller fires an initial burst and then keeps continuous foreground load running, so Live Metrics stays busy. For latency, it starts traffic right after enabling the delay, so the response-time graph moves without anyone manually clicking around.
From POST to alert
The signal that proves it
A scheduled-query alert counts 5xx responses from the app’s HTTP logs and fires past a small threshold.
AppServiceHTTPLogs
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<webapp-resource-id>"
| where ScStatus >= 500
| summarize FailureCount = count() by _ResourceId
| where FailureCount > 5A second activity-log alert catches the stop scenario specifically, because a stopped app produces a different shape of failure than one returning 500s, and you want to tell them apart at a glance.
The latency scenario does not raise 5xx at all. It pushes AverageResponseTime up instead, and the sustained load is what keeps the duration chart readable while the delay is active.
Recovery
recover returns the app to normal request behavior and lets traffic settle. Baseline is a 200 on the app root, /healthz still green (it never left), 5xx counts back to zero, and response time back to its normal band.
The lesson
A failure demo is only as convincing as the telemetry behind it, and production telemetry is built to ignore single events. If you want a drill that teaches something, you have to generate enough load that the signal rises above noise, and you have to keep your health endpoint honest so the platform does not paper over the failure for you. The design choice to hold /healthz at 200 during a fault looks counterintuitive until you have watched App Service silently recycle an instance mid-demo and wipe out the exact symptom you were trying to show a room full of people.
Next: When the Thing You Call Breaks, the dependency lab.