The Queue That Kept Growing: The Service Bus Lab

Queues fail silently. Unlike a web app that throws a visible 500, a backed-up queue produces no errors at all. Messages arrive, nobody processes them, and the depth climbs while every component reports itself perfectly healthy. The first sign of trouble is often a customer asking why something they did an hour ago still has not happened. The queue lab makes that slow, invisible failure visible by treating queue depth and dead-letter count as first-class signals.

What the lab does

It runs a dedicated Service Bus queue and a worker app that consumes from it. The Logic App controller can pause the consumer so backlog builds, enqueue a batch of valid messages to grow that backlog faster, or enqueue poison messages that fail processing and land in the dead-letter queue. Each one stresses a different part of the messaging system.

The inject contract

{ "scenario": "enqueue-backlog", "count": 50 }

The scenarios are pause-consumer (the worker stops draining), enqueue-backlog (add valid messages, with a count), enqueue-poison (add messages that fail and dead-letter, with a count), and recover. Pausing the consumer and enqueuing backlog together grow the active-message count. Enqueuing poison grows the dead-letter count. They are deliberately separate, because backlog and dead-letter are different problems with different fixes.

From POST to symptom

sequenceDiagram participant Op as Operator participant LA as Logic App participant W as Worker participant SB as Service Bus participant M as Azure Monitor Op->>LA: POST pause-consumer LA->>W: Stop draining Op->>LA: POST enqueue-backlog count 50 LA->>SB: Add 50 messages SB-->>M: ActiveMessages climbs Op->>LA: POST enqueue-poison count 25 LA->>SB: Add poison messages W->>SB: Process, fail repeatedly SB-->>M: DeadletteredMessages climbs Op->>LA: POST recover LA->>W: Resume, drain, purge DLQ

The signals that prove it

The native Azure Monitor alerts here watch the queue itself: active-message count and dead-lettered-message count. Those are the metrics that matter, because they move when nothing is erroring. The worker app also carries the standard 5xx scheduled query for the cases where processing failures surface as HTTP errors:

AppServiceHTTPLogs
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<queue-worker-resource-id>"
| where ScStatus >= 500
| summarize FailureCount = count() by _ResourceId
| where FailureCount > 5

The point of this lab is the queue-depth alerts, not the 5xx. A growing backlog is the rare failure where the most important signal is a number that goes up rather than an error that gets thrown.

Recovery

recover resumes the consumer, drains the backlog, and purges the dead-letter queue. Baseline is an active-message count near zero, an empty dead-letter queue, and a worker that is keeping up with arrivals.

The lesson

Some failures have no error to alert on, and queues are the clearest example. If your monitoring only watches for exceptions and 5xx, a backed-up queue is invisible until it is a customer complaint. The signal you need is depth: active messages climbing means consumers are not keeping up, dead-letter messages climbing means something is failing to process and giving up. Watch both, alert on both, and treat the absence of errors as exactly that, an absence, not proof that everything is fine.

Next: Proving It from Outside the Cluster, the AKS capstone.