Memory Pressure and the Crash Loop: The Stability Lab

Two failures live at opposite ends of the same spectrum. A memory leak is slow: the working set creeps up over minutes or hours until the platform forces a restart, and by then the graph looks like a gentle ramp toward a cliff. A crash loop is instant: the app dies on startup, restarts, dies again, and hammers out a storm of failed launches in seconds. The memory and crash lab gives you both, on a dedicated plan, so the blast radius stays contained.

What the lab does

It runs an isolated App Service on its own plan, which matters: memory and crash behavior can disturb neighbors on a shared plan, so this lab keeps its mess to itself. The Logic App controller pushes the app into memory pressure or a crash loop. Each one fails on a different timescale, and they show up in different telemetry.

The inject contract

{ "scenario": "memory", "durationSeconds": 300 }

The scenarios are memory (working-set pressure builds), crashloop (the app fails on startup and restarts repeatedly), and recover. The first is a slow burn. The second is a fast storm.

From POST to symptom

sequenceDiagram participant Op as Operator participant LA as Logic App participant APP as App Service participant M as Azure Monitor Op->>LA: POST memory LA->>APP: Enable memory pressure APP-->>M: Working set climbs over time M->>M: Memory query crosses threshold Op->>LA: POST crashloop LA->>APP: Force startup failure APP-->>M: Repeated restarts, 503 responses Op->>LA: POST recover LA->>APP: Healthy startup restored

The signals that prove it

This lab watches two things, because the two failures need two different alerts. The crash loop surfaces as 5xx (often 503 while the app is between restarts):

AppServiceHTTPLogs
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<memory-app-resource-id>"
| where ScStatus >= 500
| summarize FailureCount = count() by _ResourceId
| where FailureCount > 5

The memory leak surfaces as a rising working set, caught by a metric query that fires when peak average memory crosses a byte threshold:

AzureMetrics
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<memory-app-resource-id>"
| where MetricName == "AverageMemoryWorkingSet"
| summarize PeakAverageMemory = max(Average) by _ResourceId
| where PeakAverageMemory > 300000000

The memory alert is the one worth dwelling on, because it can fire before users notice anything. A leak caught at the working-set level is a leak you can act on during the slow ramp, rather than after the forced restart that finally makes it visible to everyone.

Recovery

recover returns the app to healthy startup and normal memory. Baseline is a clean launch, working set back in its normal band, and no 5xx from restart churn.

The lesson

Match the alert to the timescale of the failure. An error-rate alert is perfect for a crash loop, which produces failures by the second, and useless for a slow leak, which produces no errors at all until the very end. The leak only becomes an error-rate event at the moment of the forced restart, which is the worst possible time to first hear about it. Watch the resource metric that degrades gradually (working-set memory here) so you get the early ramp, and keep the error-rate alert for the failures that arrive all at once.

Next: Throttled by Design, the quota lab.