Building an Idempotent Azure Platform with Bicep and ARM

Infrastructure drift is one of the quietest risks in cloud operations.

It rarely starts as a major outage. It usually starts with a small manual change: a setting adjusted in the portal, a role assignment added during troubleshooting, a diagnostic setting missed during a deployment, or a service deployed slightly differently between environments. Over time, those exceptions become the real architecture.

For a platform operating across many Azure subscriptions, customers, environments, and services, that model does not scale. We needed a way to ensure that infrastructure was not only deployed correctly the first time, but continuously aligned to an approved architecture.

That is where Bicep and ARM became foundational to our platform.

We use infrastructure as code to define the architecture we approve, deploy it repeatedly, and make every environment converge toward that intended state. The goal is not simply automation. The goal is idempotency: every deployment should be safe to run again, and every run should move the environment closer to the approved baseline.

The Problem: Manual Infrastructure Does Not Stay Approved

In a small environment, manual deployment can feel manageable. A team can create a Function App, wire up storage, configure Key Vault, add monitoring, and move on.

At scale, that becomes fragile.

We had to support a platform architecture that included API Management, Azure Functions, Cosmos DB, PostgreSQL Flexible Server, Key Vault, Service Bus, Storage Accounts, Log Analytics, Application Insights, managed identities, and customer-specific role assignments.

Each service had its own configuration requirements. Each environment had different sizing, naming, networking, and security parameters. Each customer onboarding flow required identity and access components that had to be consistent, repeatable, and auditable.

The risk was not that we could not deploy the platform once. The risk was that we could not guarantee every deployment remained aligned over time.

We needed an approach where the architecture itself became executable.

Approved Architecture as Code

The core principle is simple:

If it is part of the approved architecture, it belongs in source control.

That means infrastructure is not treated as a one-time setup task. It is treated as a versioned platform artifact.

Our infrastructure repository is organized around a main Bicep orchestrator, reusable service modules, environment-specific parameter files, and ARM templates for specific customer identity workflows.

infra/
├── main.bicep                    # Main orchestrator template
├── modules/
│   ├── apim.bicep                # API Management
│   ├── cosmos.bicep              # Cosmos DB
│   ├── functions.bicep           # Azure Functions
│   ├── keyvault.bicep            # Key Vault
│   ├── keyvault-access.bicep     # Key Vault RBAC access
│   ├── monitoring.bicep          # Log Analytics & App Insights
│   ├── postgres.bicep            # PostgreSQL Flexible Server
│   ├── servicebus.bicep          # Service Bus
│   └── storage.bicep             # Storage Account
├── parameters/
│   ├── dev.bicepparam            # Development environment
│   ├── staging.bicepparam        # Staging environment
│   └── prod.bicepparam           # Production environment
└── arm/
    ├── customer_uami.json
    └── customer_uami_role_assignment.json

This structure gives us a clean separation of concerns.

The main.bicep file defines how the platform fits together. It is the orchestrator. It does not need to contain every low-level resource detail directly. Instead, it composes the architecture from modules.

The modules/ directory contains reusable infrastructure building blocks. Each module owns one platform capability: API Management, Cosmos DB, Azure Functions, PostgreSQL, Service Bus, monitoring, storage, Key Vault, and access configuration.

The parameters/ directory defines the environment-specific differences. Development, staging, and production use the same architecture, but with different values where appropriate.

The arm/ directory contains ARM templates used for customer-specific identity and role assignment operations, such as customer user-assigned managed identities and delegated role assignments.

This gives us a consistent deployment pattern while still allowing controlled variation across environments and customers.

Why Bicep for the Core Platform

Bicep is a strong fit for the main platform because it gives us a readable, modular, Azure-native way to define infrastructure.

Instead of maintaining large JSON templates by hand, we can express resources, parameters, outputs, dependencies, and modules in a cleaner format. That matters when the platform includes many connected services.

For example, the Function App depends on a Storage Account. The application needs access to Key Vault. API Management needs to route to backend services. Monitoring needs to be attached consistently. Service Bus needs queues, topics, authorization boundaries, and diagnostic settings. PostgreSQL needs configuration that is different between lower environments and production.

Bicep lets us define those relationships directly.

That means the deployment is not a checklist. It is a dependency graph.

The result is a platform that can be deployed consistently from source control rather than reconstructed from memory, screenshots, or tribal knowledge.

Why ARM Still Has a Place

Although Bicep is our preferred authoring language for the core platform, ARM templates still have value in specific cases.

In our case, ARM is used for customer user-assigned managed identity workflows and customer role assignment templates.

These templates support repeatable customer onboarding patterns where identity and authorization must be precise. When working across many subscriptions and delegated customer environments, role assignment consistency is critical. A missing or incorrect role assignment can break automation, monitoring, remediation, or data collection.

The ARM templates allow us to standardize those customer-specific operations and keep them versioned alongside the rest of the infrastructure code.

This is an important point: infrastructure as code does not have to be one tool only. The goal is not ideological purity. The goal is controlled, repeatable, auditable deployment.

Bicep handles the modular platform architecture. ARM handles targeted identity and role assignment scenarios where JSON templates are already reliable and explicit.

Idempotency as the Operating Model

The most important design goal is idempotency.

An idempotent deployment can be run multiple times without creating duplicate resources, breaking existing resources, or introducing unexpected changes. If the environment already matches the template, the deployment should result in no meaningful change. If the environment has drifted, the deployment should bring it back toward the desired state.

This changes the way infrastructure is managed.

Instead of asking, “Did someone deploy this correctly?” we ask, “Does the current environment still match the approved definition?”

That distinction matters.

With idempotent Bicep and ARM deployments, we can safely reapply the baseline. We can use deployment validation and what-if checks to understand impact before making changes. We can promote changes from development to staging to production with the same structure. We can review infrastructure changes through pull requests. We can track exactly when the approved architecture changed and why.

This creates a stronger control plane for platform engineering.

Environment Consistency Without Copy-Paste

One of the common mistakes in infrastructure automation is duplicating templates per environment.

That creates three versions of the truth: one for development, one for staging, and one for production. Over time, they drift. A fix lands in one environment and not another. A security setting is added to production but forgotten in staging. A diagnostic setting exists in development but not production.

Our approach avoids that by using the same Bicep modules and orchestrator across environments.

The difference lives in .bicepparam files.

Development can use lower-cost SKUs, different capacity, relaxed scaling, or sandbox naming. Production can use higher availability, stronger performance tiers, stricter retention, and production-grade configuration. But the architectural shape remains consistent.

That means the platform is promoted, not reinvented.

The same API Management module is used. The same Key Vault module is used. The same monitoring module is used. The same PostgreSQL module is used. The same Service Bus module is used.

This makes the deployment path predictable:

dev → staging → production

By the time a change reaches production, the architecture has already been exercised in lower environments.

Modular Design Keeps the Platform Maintainable

Each Bicep module represents a platform capability.

The API Management module defines the gateway layer. The Functions module defines the compute layer. The Cosmos and PostgreSQL modules define data services. The Service Bus module defines asynchronous messaging. The Storage module supports runtime and platform storage needs. The Monitoring module ensures observability is part of the baseline. The Key Vault and Key Vault access modules enforce secret and access patterns.

This modularity gives us several advantages.

First, ownership is clearer. A change to PostgreSQL configuration belongs in the PostgreSQL module. A change to diagnostic settings belongs in monitoring. A Key Vault RBAC change belongs in the access module.

Second, reviews are easier. Pull requests are smaller and more focused because changes are isolated to the relevant module.

Third, reuse becomes practical. The same module can be used across environments and potentially across platform instances.

Fourth, policy becomes enforceable. Approved architecture is no longer a diagram in a slide deck. It is implemented as deployable code.

Deployment as a Governance Control

Bicep and ARM are not just deployment tools. Used correctly, they become governance controls.

Every infrastructure change can go through the same path as application code:

1. A change is proposed.
2. The change is reviewed.
3. The deployment is validated.
4. A what-if comparison is performed.
5. The change is applied to the target environment.
6. The resulting state becomes the new approved baseline.

This makes the infrastructure lifecycle auditable.

When someone asks why a resource exists, how it is configured, or when a setting changed, the answer should be in source control. When someone asks what production is supposed to look like, the answer should not be “check the portal.” The answer should be the Bicep and ARM definitions.

That is a major operational improvement.

Preventing Drift from Becoming the Architecture

Drift is inevitable in real-world systems.

Emergency fixes happen. Troubleshooting happens. Teams test things. Customer-specific exceptions happen. Azure services evolve. Permissions change. Environments are touched by humans and automation.

The problem is not that drift happens. The problem is when drift becomes invisible.

Our infrastructure-as-code model gives us a way to detect and correct that drift. By continuously comparing deployed infrastructure against the approved templates, we can identify when an environment no longer matches the intended architecture.

Some drift may be legitimate and should become a pull request. Some drift is accidental and should be remediated. Some drift indicates a process gap.

Either way, the source-controlled architecture remains the standard.

That is the key principle: the portal shows what exists, but the repository defines what is approved.

Supporting Customer Onboarding at Scale

The customer identity templates are especially important in a multi-customer platform.

When onboarding customers, we need repeatable identity creation and role assignment. This includes customer user-assigned managed identities and the permissions required for the platform to monitor, automate, and remediate resources.

Doing this manually is risky. The process is detailed, security-sensitive, and easy to get slightly wrong.

By defining customer UAMI and role assignment templates in ARM, we make onboarding repeatable. Each customer follows the same tested deployment pattern. Access can be reviewed. Role assignments can be versioned. Exceptions can be controlled.

This is especially important when working across delegated subscriptions or customer-owned environments. The more distributed the platform becomes, the more important it is to have a consistent identity and authorization model.

The Bigger Outcome: Architecture That Can Reapply Itself

The real value of this approach is not just faster deployment.

The real value is that the approved architecture can reapply itself.

That gives us a stronger operational foundation. We can build, validate, and promote infrastructure changes through a controlled process. We can recover consistency when drift occurs. We can onboard customers with repeatable templates. We can reduce manual configuration. We can prove what the architecture is supposed to be.

This is how infrastructure as code becomes more than automation.

It becomes a mechanism for architectural control.

Lessons Learned

The first lesson is that modularity matters. Large monolithic templates become difficult to understand and review. Breaking the platform into focused modules makes the system easier to maintain.

The second lesson is that parameters should express controlled variation, not architectural differences. Development, staging, and production should not become separate architectures unless there is a deliberate reason.

The third lesson is that identity and access deserve first-class treatment. Managed identities, Key Vault access, and role assignments are not afterthoughts. They are core platform components.

The fourth lesson is that idempotency should be a design requirement from the beginning. Every template should be safe to run repeatedly.

The fifth lesson is that infrastructure code should be reviewed with the same seriousness as application code. A small infrastructure change can have a large operational impact.

Conclusion

Bicep and ARM allow us to define Azure infrastructure in a way that is repeatable, reviewable, and enforceable.

For our platform, that means API Management, Azure Functions, Cosmos DB, PostgreSQL, Key Vault, Service Bus, Storage, monitoring, managed identities, and customer role assignments are no longer manually assembled pieces. They are part of a governed architecture.

The result is an idempotent system that keeps environments aligned to the approved design.

At small scale, this improves consistency.

At large scale, it becomes essential.

When infrastructure is defined as code, the architecture is no longer just documented. It is executable.