← Azure Failure Labs
07 / 12 Technical field note

The Database Said No: The PostgreSQL Lab

How bad credentials, slow queries, and blocked database paths surface in an Azure failure lab, and what telemetry proves the symptom.

AzurePostgreSQLChaos EngineeringMonitoring

Database failures come in flavors, and they do not all look alike. A wrong password fails instantly and loudly. A slow query fails gradually and quietly, dragging response times up until something times out. A blocked firewall path fails like a network problem even though the database is perfectly healthy. The database lab reproduces all three against a real PostgreSQL server, so you can see how differently each one moves your telemetry.

What the lab does

It runs an App Service backed by a dedicated PostgreSQL flexible server. The Logic App controller swaps in bad credentials, injects slow-query behavior, or blocks the firewall path to the database. The app keeps running throughout, so you are watching how an otherwise-healthy app behaves when its data layer turns hostile.

flowchart LR OP["Operator"] -->|"scenario"| LA["Logic App\ndatabase-controller"] LA -->|"bad creds / slow / firewall"| APP["Database app"] APP -->|"query"| PG["PostgreSQL\nflexible server"] APP --> MON["Azure Monitor"] PG --> MON MON --> AG["Shared action group"] OP -.->|"recover"| LA

The inject contract

{ "scenario": "slow-query", "durationSeconds": 300 }

The scenarios are bad-credentials (authentication fails), slow-query (queries run but crawl), firewall-block (the network path to PostgreSQL is cut), and recover. Three different causes, three different curves.

From POST to symptom

sequenceDiagram participant Op as Operator participant LA as Logic App participant APP as Database app participant PG as PostgreSQL participant M as Azure Monitor Op->>LA: POST scenario alt bad-credentials LA->>APP: Swap in wrong password APP->>PG: Connect PG-->>APP: Auth failure APP-->>M: 500 with auth error else slow-query LA->>APP: Enable slow query path APP->>PG: Long-running query PG-->>APP: Late response APP-->>M: Latency climbs else firewall-block LA->>PG: Block network path APP->>PG: Connect times out APP-->>M: Connectivity failure end Op->>LA: POST recover

The signal that proves it

The app’s 5xx scheduled query catches the credential and connectivity failures.

AppServiceHTTPLogs
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<database-app-resource-id>"
| where ScStatus >= 500
| summarize FailureCount = count() by _ResourceId
| where FailureCount > 5

The firewall-block scenario also produces a firewall-rule change in the activity log, so this lab pairs a symptom alert with a cause alert the same way the network lab does. And slow-query deliberately does not raise 5xx at all. It inflates response time, which is the signal you would chase in a real “the site feels slow” complaint where nothing is technically erroring.

Recovery

recover restores healthy credentials, normal query behavior, and the firewall path, returning the app to 200. Baseline is fast queries, a reachable database, and no firewall-change events for the server.

The lesson

One subsystem can fail in several ways, and a single alert will not teach your team to tell them apart. The credential failure and the connectivity failure both raise 5xx, but the slow query hides in latency where a 5xx alert never sees it. If you only watch error rate, you will miss the entire class of “working but unbearably slow” database problems. Watch error rate and latency together, and rehearse each failure mode so responders learn that “the database said no” has at least three distinct dialects.


Next: Memory Pressure and the Crash Loop.

Architecture depth + engineering leadership

Building platforms enterprises trust at scale.