Running but Unreachable: The Network Isolation Lab

The most-missed call in an outage is mistaking unreachable for down. A team spends the first twenty minutes restarting a service that was never the problem, because the box was up the whole time and the network in front of it was quietly dropping traffic. The network lab recreates that exact trap: it severs an HTTP path while leaving the VM powered on and perfectly healthy from its own point of view.

What the lab does

It stands up a dedicated Linux VM serving a simple health endpoint on port 8080, plus a small App Service probe that calls that endpoint on a loop. The Logic App controller deletes the inbound network security group rule that allows port 8080, so the probe starts timing out while the VM keeps running, keeps serving locally, and keeps reporting itself fine. Recovery recreates the rule.

Here is the topology as a portable SVG figure.

The inject contract

The payload is as small as it gets. There is one fault and one recovery.

{ "scenario": "isolate", "durationSeconds": 300 }

isolate deletes the inbound allow rule for port 8080. recover recreates it. The durationSeconds field stays in the contract for payload stability, but the fault applies immediately and waits for an explicit recover, which keeps the operator in control of when the path comes back.

From POST to alert

sequenceDiagram participant Op as Operator participant LA as Logic App participant NSG as NSG rule participant VM as Linux VM participant P as App Service probe participant M as Azure Monitor Op->>LA: POST isolate LA->>NSG: Delete allow rule :8080 P->>VM: Probe request times out P-->>M: Probe returns 5xx Note over VM: VM stays up, serves locally M->>M: Two signals: probe 5xx and NSG change Op->>LA: POST recover LA->>NSG: Recreate allow rule P->>VM: Probe succeeds again

The signals that prove it

This lab fires on two independent signals, which is the interesting part. One watches the symptom, the other watches the cause.

The symptom is the probe failing, caught by a scheduled query over App Service logs:

AppServiceHTTPLogs
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<network-probe-resource-id>"
| where ScStatus >= 500
| summarize FailureCount = count() by _ResourceId
| where FailureCount > 5

The cause is the NSG rule changing, caught by an activity-log query:

AzureActivity
| where TimeGenerated > ago(5m)
| where _ResourceId =~ "<nsg-rule-resource-id>"
| where CategoryValue =~ "Administrative"
| where ActivityStatusValue =~ "Succeeded"
| where OperationNameValue in~ ("MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/SECURITYRULES/DELETE", "MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/SECURITYRULES/WRITE")
| summarize EventCount = count() by _ResourceId
| where EventCount > 0

Having both is what shortens the incident. The symptom alert tells you something broke. The cause alert tells you a security rule changed at the same moment, which points the responder at the network instead of at the VM.

Recovery

recover recreates the inbound rule, the probe path heals, and the probe goes back to 200. Baseline is the probe returning success and no new NSG-change events in the activity log for the lab’s rule.

The lesson

When you can, alert on the cause as well as the symptom. A symptom alert (“the probe is failing”) sends a responder looking. A paired cause alert (“an NSG rule was deleted thirty seconds earlier”) sends them looking in the right place. The VM in this lab is a deliberate decoy. It is healthy, it is up, and every check that asks the VM about itself will lie to you by omission. The truth lives in the path between the caller and the host, which is exactly where you should be watching.

Next: Deterministic Application Failure, the App Service lab.