Automation at MSP scale is different.
In a single-company IT environment, automation usually operates inside one business, one governance model, one identity boundary, one cloud maturity level, and one set of operational expectations. The environment may still be complex, but the assumptions are usually more consistent. You know the business. You know the stakeholders. You know the operating model. You usually have a clearer view of who owns what and how change is approved.
In a managed service provider model, those assumptions change quickly.
You are no longer engineering for one environment. You are engineering for many. Each customer has a different cloud journey, a different level of maturity, a different governance model, a different risk posture, and a different expectation for how change should happen. Some customers are early in their cloud adoption. Some have mature enterprise standards. Some require CAB approval before any change. Some allow approved remediation under defined conditions. Some have exceptions that need to be honored because of business, compliance, or operational requirements.
That changes how automation has to be designed.
At this scale, automation is not trusted simply because it runs. It earns confidence because it repeatedly produces the right outcome, reports what happened, handles failure clearly, and leaves enough evidence for operators, customers, and internal teams to understand the result.
In the environment I helped build, the platform operated across roughly 36,000 tenants, 5,000 Azure subscriptions, and 29 million resources. At that scale, the engineering challenge was not just creating automation. The harder challenge was creating automation that could prove its reliability through measured, observable, repeatable outcomes.
Automation across many operational domains
The platform automated a broad set of operational capabilities across customer environments. That included customer onboarding, monitoring and alert onboarding, policy assignment, policy drift remediation, resource inventory, billing processing, Sentinel and Defender onboarding, Azure Arc onboarding, and lifecycle actions across managed resources.
These were not isolated scripts. They were platform capabilities tied to customer state, service eligibility, purchased services, current standards, and operational expectations.
That distinction matters.
A script can perform an action. A platform has to understand whether that action should happen, where it should happen, whether the customer is eligible for it, whether the customer has opted out, whether approval is required, what version should apply, and whether the outcome was successful.
At MSP scale, automation needs context before it acts.
It has to know the customer. It has to know the subscription. It has to know the service standard. It has to know the current version of the alert or policy. It has to know whether remediation is allowed. It has to know whether the action is read-only, change-producing, customer-approved, or waiting on a governance process.
Without that context, automation becomes activity. With that context, automation becomes an operating system for reliability.
Reliability is measured by outcomes, not activity
One of the most important lessons in building automation at this scale is that activity is not the same as progress.
A job can run and still fail to improve the environment. A deployment can complete and still leave the customer in the wrong state. A remediation can execute and still be incomplete if it does not produce the expected outcome.
For this platform, reliability meant several things working together:
It meant automation completed successfully. It meant it ran idempotently. It meant it produced accurate results. It meant it avoided duplicate actions. It meant retries were handled correctly. It meant failure was reported clearly. It meant every important action left an audit trail. It meant the system could recover from partial failure without creating confusion about current state.
But more than anything, reliability meant the environment moved toward the intended outcome.
The most important question was not, “Did the automation run?”
The better question was, “Did the customer environment converge toward the current approved standard?”
That was the measure that mattered.
Versioned standards made convergence measurable
To measure convergence, the platform needed a clear definition of expected state.
Every alert and policy carried metadata describing its current approved version. That version was controlled by the platform’s source of truth and validated before being released to customers. This created a standard the platform could evaluate against.
That meant drift was not based on a vague idea that something “looked wrong.” Drift could be measured against a known version of an approved alert, policy, or monitoring standard.
The platform could determine whether a customer environment had the correct assignment, the correct configuration, the correct version, and the correct expected behavior. If the environment did not match, the platform could identify the gap and determine the next appropriate step.
That next step was not always remediation.
Sometimes the right action was to report. Sometimes it was to skip intentionally because the customer had an approved exception. Sometimes it was to wait for CAB approval. Sometimes it was to remediate because the customer’s service model allowed it.
Versioning created clarity.
It allowed the platform to answer a simple but powerful question: “Is this environment aligned to the current approved version of the service standard that applies to this customer?”
Read-only by default, change-driven by drift
A key design principle was that the platform was read-only by default.
Its first responsibility was to observe, evaluate, compare, and report. The platform needed to know the truth of the environment before it could responsibly change anything.
That mattered because the platform had broad reach. It could see across thousands of subscriptions and millions of resources. But broad visibility does not mean every action should become a broad change.
Change-producing actions only entered the process when drift was identified and the customer’s governance model allowed remediation. That created a deliberate separation between detection and change.
Detection answered: “What is the current state?”
Drift analysis answered: “How does current state compare to expected state?”
Governance answered: “Is the platform allowed to act?”
Remediation answered: “What controlled change should happen to bring the environment back into alignment?”
That sequence matters. It prevents automation from becoming overly aggressive and keeps the platform tied to measured outcomes rather than assumed intent.
Customer governance had to be built into the platform
In an MSP model, customers do not all operate the same way.
Some customers allow specific remediation paths. Some opt out of certain automations. Some allow detection but not automated change. Some require CAB approval before anything modifies their environment. Some need exceptions because of compliance, architecture, contractual obligations, or internal operating models.
The platform had to account for that.
Customer-specific governance was modeled through internal tags, Azure tags, eligibility metadata, and exception handling. These signals helped the platform understand which customers were eligible for specific automations, which remediation paths were allowed, and which actions required additional approval.
For customers with CAB requirements, the workflow looked different:
The platform detected drift. It created a notification or change request. The customer’s approval process reviewed the change. Only after approval did remediation proceed. After the remediation completed, the platform recorded the before-and-after state.
That workflow is important because it shows the difference between automation and governed automation.
The platform did not bypass the customer’s operating model. It respected it.
That is what MSP-scale engineering requires. Customer governance cannot be handled as an afterthought. It has to be part of the system design.
Confidence came from convergence
The strongest measure of the platform was convergence.
Did the customer environment move toward the current approved version? Did the alert standard reach the expected state? Did the policy assignment match the validated version? Did onboarding result in the correct monitoring posture? Did drift remediation restore the environment to the expected baseline?
Those outcomes mattered more than raw execution counts.
A platform can generate a large number of successful job runs and still fail to produce meaningful operational improvement. The real measure is whether customer environments are becoming more aligned, more observable, and more consistent with the services they purchased and the standards that apply to them.
That is why convergence became the key measure.
The platform needed to show not only that it performed actions, but that those actions moved environments toward the intended operating model.
The metrics that built confidence
Confidence came from measurement.
The platform measured success rates, failure rates, retry counts, dead-letter counts, drift corrected, drift intentionally skipped, customer coverage, subscription coverage, time to detect, time to remediate, duplicate suppression, and audit completeness.
It also measured capability, capacity, utilization, and performance. P90 and P95 performance scores helped show how the platform behaved under real operational load, not just in ideal test conditions.
These metrics gave the team a way to evaluate whether the platform was ready to expand scope.
If retry counts increased unexpectedly, that mattered. If dead-letter counts changed, that mattered. If time to remediate degraded, that mattered. If P95 performance moved in the wrong direction, that mattered. If drift was being skipped intentionally because of customer exceptions, that needed to be visible. If drift was not being corrected because of permission issues or customer environment changes, that needed to be clear.
The goal was not simply to collect telemetry.
The goal was to use telemetry to make better rollout decisions.
Fleet-wide capability does not justify fleet-wide change
One of the most important operating principles was that the platform intentionally did not allow unrestricted fleet-wide changes.
Even when a platform has the technical reach to operate across thousands of subscriptions, that does not mean every change should be executed everywhere at once.
Every change needed a reason. Every rollout needed a scope. Every new capability needed a validation path.
That principle created control.
It allowed new features and remediation capabilities to be introduced in stages. It allowed the team to validate behavior in smaller groups before expanding. It reduced the chance that an unexpected issue would affect a large number of customers. It gave the platform room to learn from real execution before increasing reach.
Fleet-wide capability is powerful. But power without controlled rollout creates unnecessary operational risk.
At MSP scale, the better approach is small batches, measured outcomes, and deliberate expansion.
Layered controls created confidence
No single control made the platform reliable. Confidence came from layers.
Idempotency ensured repeated execution did not create inconsistent results. Scope validation limited where an action could apply. Customer eligibility checks ensured the platform respected service boundaries and opt-out decisions. RBAC boundaries constrained what the platform could do. Execution logging created traceability. Retry handling improved resilience. Dead-letter queues made failure visible. Before-and-after state comparison proved what changed. Auditable decision records preserved the why behind the action. Human review remained part of the workflow for high-impact changes.
Each control served a purpose.
Together, they created a model where automation could be measured, explained, repeated, and improved.
That is the difference between automation that merely executes and automation that becomes trusted.
Failure handling required rich telemetry
Failure is unavoidable at this scale.
Customer environments change. Permissions change. Resources are renamed, deleted, moved, or reconfigured. Policies are modified. Dependencies become unavailable. API behavior changes. A platform bug can be introduced. A customer-side change can break an assumption that was previously valid.
The important question is not whether failure can be eliminated. It cannot.
The important question is whether failure can be diagnosed quickly and accurately.
The telemetry had to be rich enough to distinguish between a platform defect and a change in a customer environment. That distinction matters because the resolution path is different.
If the issue is a platform bug, the platform team owns the fix. If the issue is a permission change, the response may involve access validation. If the issue is customer-side drift, the response may involve remediation or customer notification. If the issue is a dependency failure, the response may require retry handling or operational escalation.
Good telemetry does not simply say, “Something failed.”
Good telemetry helps explain why it failed, where it failed, what customer or subscription was affected, what step was running, what the expected state was, what the actual state was, and what needs to happen next.
That level of observability is what allows teams to resolve faults in a timely manner.
Idempotency and database-backed state tracking
Duplicate processing creates confusion.
At small scale, duplicate execution may look like a minor inefficiency. At MSP scale, duplicate execution can create conflicting signals, repeated notifications, unnecessary change attempts, and uncertainty about what actually happened.
Idempotency was critical.
The platform used database-backed records to track each change and its status. That allowed the system to know whether an action was pending, approved, running, completed, failed, skipped, waiting for CAB approval, or already processed.
That state tracking mattered because the platform had to support retries, partial failures, reruns, and delayed approvals without losing control of the workflow.
A remediation should not create a different result simply because it was executed twice. A retry should not create duplicate work. A rerun should not erase the audit trail. A delayed CAB approval should not cause the platform to forget why the remediation was requested in the first place.
The database became part of the reliability model.
It gave the platform memory.
Blast radius and circuit breakers
Blast radius is the scope of impact if something does not behave as expected.
In an MSP platform, blast radius can be measured in customers, tenants, subscriptions, resources, policies, alerts, or service workflows. The goal is always to minimize that impact to the smallest practical scope.
That is another reason broad fleet-wide change was avoided.
The platform needed the ability to roll out in controlled batches, validate results, and stop when something looked wrong. Circuit breakers were part of that mindset. If telemetry showed unexpected behavior, the platform needed a way to halt or contain execution before impact expanded.
This is not just an operational concern. It is an architectural requirement.
A reliable MSP platform has to assume that unexpected conditions will happen. The design must make it possible to limit impact, stop execution, preserve state, and diagnose the problem before continuing.
That is how confidence is preserved during change.
The platform became a source of truth
Over time, the platform became more than an automation engine.
It became a source of truth for each customer’s current state.
It tracked what services a customer purchased. It tracked what standards applied to those services. It tracked which policies and alerts were expected. It tracked current compliance state. It tracked drift. It tracked exceptions. It tracked remediation status. It tracked before-and-after outcomes. It tracked whether environments were converging toward the intended operating model.
That mattered because MSP operations require more than action. They require visibility.
Internal teams need to know what is deployed. Operators need to know what failed. Customer-facing teams need to explain current state. Leadership needs to understand coverage and reliability. Customers need confidence that their environment is being managed according to their needs and expectations.
A platform that only performs automation is useful.
A platform that becomes a trusted source of truth is strategic.
The lesson: batch, validate, certify, expand
The most important advice I would give to engineers building automation across hundreds of customers is simple:
Never do everything in one step.
Start with a small batch. Validate the result. Certify the outcome. Expand only after the data shows the process is working.
That means every rollout should have a reason, a scope, a measurement model, and a validation path. Every change should produce evidence. Every failure should be diagnosable. Every remediation should be traceable. Every expansion should be earned by the results of the previous stage.
At MSP scale, confidence is not declared.
It is measured.
Automation becomes trusted when it consistently produces the right outcome, respects customer governance, reports clearly, handles failure, limits blast radius, and proves convergence toward the approved standard.
That is the difference between automation that runs and automation that can be relied on.