7. Production

Five pillars. The ML-specific ones are cost (where the money actually goes) and observability (where data drift lives alongside the usual latency and errors).

[Image Prompt: 2D minimalistic diagram of five production pillars — security, cost, scaling, observability, and reliability — arranged around a central Azure Machine Learning workspace resource, flat design, clean vector art style, white background]

Security

Identity first

• One user-assigned managed identity shared by compute and online deployments. Every data-plane grant is then made once, to one principal, and revoked once. See Deployment.
• No keys on storage. shared_access_key_enabled = false, identity-based datastores, and Storage Blob Data Reader on the compute identity. A storage key in a datastore is a credential that outlives everyone who knew about it.
• aad_token on online endpoints, not key. Key auth is fine while you're proving something works and is a standing secret thereafter.
• Least privilege by plane. Data scientists get AzureML Data Scientist (run jobs, manage assets) and not Contributor (create compute, change networking). Add AzureML Compute Operator if they must start and stop compute. This split is the main reason the roles exist. ⚠️ Verify current role names and contained actions against Azure docs.

Network

The full-isolation checklist, in the order things get forgotten:

1. Workspace: public_network_access_enabled = false + private endpoint (amlworkspace).
2. Storage: private endpoints on both blob and file, public access disabled.
3. Key Vault: private endpoint, RBAC authorization, firewall default-deny.
4. ACR: Premium SKU, private endpoint, admin user disabled.
5. Managed VNet on the workspace, isolation mode AllowOnlyApprovedOutbound, with explicit FQDN rules for whatever your jobs need to reach (PyPI, conda, your artifact feed, Hugging Face).
6. Private DNS zones for every one of the above, or nothing resolves inside the VNet.
7. Online endpoints: public network access disabled, private endpoint for callers.

Two honest costs. AllowOnlyApprovedOutbound provisions a managed firewall you pay for hourly — budget it. And the day you turn it on, every job that quietly installed a package at runtime fails. Bake dependencies into environments before you flip that switch, not after.

Data

• Customer-managed keys on the workspace encrypt the metadata store as well as your data, if your compliance bar requires it. Adds operational weight — a key you can lose is now a workspace you can lose.
• Training data is production data. The masking, retention, and residency rules that apply to the source table apply to the copy in workspaceblobstore. Job outputs land there by default and are easily forgotten in a data-inventory exercise.
• Model data collection captures request payloads. If those payloads contain personal data, you have just created a new store of it. Enable it deliberately, with retention and access controls.

Supply chain

Models are executable artifacts. A pickle loaded in a scoring script runs arbitrary code as the deployment's identity. Register models only from jobs your pipeline ran, prefer mlflow_model over raw pickles, scan environment base images, and pin them by digest rather than :latest in anything that serves traffic.

Cost — where the money actually goes

The workspace is free. Every line on the bill is something else. In rough order of how often it surprises people:

Cost	Why it surprises	The fix
Compute instances left running	Billed per hour regardless of activity, and they feel like laptops, not VMs	Idle shutdown on every instance, no exceptions. A scheduled stop as a backstop. Deny instance creation in prod by policy
Clusters with min_instances > 0	Someone set a floor "so it's warm" and converted rented compute into owned compute	Set it to 0. If cold start genuinely hurts, that's a serverless-compute conversation
Idle online deployments	Billed per instance-hour with no scale-to-zero, whether or not a request arrives	Right-size instance_count. Move latency-tolerant work to a batch endpoint, which pays only while it runs
Managed VNet firewall	An hourly charge that appears the moment you choose AllowOnlyApprovedOutbound	Budget it, or use AllowInternetOutbound if your threat model permits
Storage accumulation	Every job snapshot, every output, every logged artifact, forever	Lifecycle policies on workspaceblobstore. Nothing prunes it for you
ACR	Environment images accumulate per version	Retention policy on untagged manifests; Basic SKU where private endpoints aren't needed
Application Insights / Log Analytics ingestion	Chatty scoring scripts are surprisingly expensive at volume	Sampling; log at the right level; a retention policy
Two deployments during blue/green	Deliberate, and worth it — but it is double	Delete the retired deployment once the rollback window closes

Three levers with real leverage:

• Spot / low-priority nodes for sweeps and anything checkpointed. Big discount, evictable. ⚠️ Verify current discount, which varies by region and family.
• Reservations or savings plans for genuinely steady inference compute.
• Pipeline step caching. A daily pipeline whose featurise step is unchanged shouldn't recompute it.

And the governance that makes any of it enforceable: mandatory tags (workload, environment, cost-center) via Azure Policy, per-resource-group budgets with alerts, and a VM SKU allowlist so nobody discovers an eight-GPU node by autocomplete.

Scaling — and the quota that gates it

Three things scale, in different ways, and none of them scale past your quota.

Training scales by cluster max_instances and by node size. The binding constraint is dedicated vCPU quota per VM family, per subscription, per region, with low-priority counted separately. A cluster configured for 40 nodes against quota for 12 is a 12-node cluster with a misleading config file. Check with az ml compute list-usage.

Online inference scales two ways:

• Within a deployment: Azure Monitor autoscale rules on instance_count, driven by CPU or a request metric. This is the ordinary VM-scale-set autoscale engine, with its usual cooldowns and its usual inability to react to a step change in under a couple of minutes.
• Across deployments: the traffic split. Not autoscaling — a deliberate operational dial.

The constraint here is managed online endpoint quota, a separate pool from training vCPU, also per subscription per region. Hitting it fails deployment creation outright, which at least is a loud failure rather than a silent queue.

Batch inference scales with the cluster behind the batch endpoint, plus mini_batch_size and max_concurrency_per_instance on the deployment. This is the cheapest way to score a lot of data, because the compute exists only while the job runs.

⚠️ Every number in this section — quotas, caps on endpoints per workspace and deployments per endpoint — varies by subscription type and region. Read them from the portal's Usage + quotas blade rather than from any document, including this one.

Design consequences worth internalising: prod in its own subscription, because quota is per-subscription and a runaway sweep in dev should not starve a prod retrain. And request GPU quota early, because GPU families frequently start at zero and the approval is not instant.

Observability

Four layers. Most teams build the first two and are surprised by the fourth.

1. Platform metrics

A diagnostic setting on the workspace routing to Log Analytics. Without it, nothing is retained long-term. Categories worth capturing: job events, compute cluster events, and quota utilisation. Alert on: a nightly training job that did not complete, cluster node allocation failures, and quota above ~80%.

2. Endpoint telemetry

Application Insights receives request telemetry from online deployments. Watch:

• Latency percentiles, not means. P50 hides everything that matters.
• Error rate by deployment, not just by endpoint — during a canary, a 5% error rate on green at 10% traffic reads as 0.5% overall and is invisible in the aggregate.
• Request concurrency vs. max_concurrent_requests_per_instance. Queuing shows up as latency long before it shows up as errors.
• Cold-start after a scale-out event. New instances run init(); if that loads a 4 GB model, scale-out is slow and autoscale thresholds need to account for it.

3. Job and experiment tracking

This is MLflow, and it is your regression detector. Log metrics for every run, compare candidates against the incumbent, and — the part usually missing — make the evaluation job's exit code a build gate, so a model that regresses cannot be promoted. See Deployment.

4. Model monitoring and data drift

The ML-specific layer, and the one that catches the failure nobody else will.

A model does not break loudly. It degrades because the world moved: a new customer segment, an upstream schema change, a currency switch, a marketing campaign that changes the input distribution. Latency is fine, error rate is zero, and the predictions are quietly worse for weeks.

The mechanism: enable data collection on the deployment so request and response payloads are written to storage, then run model monitoring to compare the production distribution against the training baseline on a schedule. Signals worth configuring: data drift (input distributions moved), data quality (nulls, type violations, out-of-range values), prediction drift (output distribution moved), and — where you eventually get ground truth — feature attribution drift and actual performance metrics.

Two honest caveats. Ground truth usually arrives late, sometimes months late for fraud or churn, so drift on the inputs is often the only signal you have in time to act. And an alert with no runbook is noise — decide in advance whether a drift alert triggers a retrain, an investigation, or a shrug, and write it down.

⚠️ Model monitoring features and their signal names have changed across releases and some are preview with no SLA — verify current capability before designing an alerting strategy on them.

Reliability

The failure modes ranked by likelihood

1. A model regression reaches 100% traffic. By far the most common production incident, and entirely preventable by never deploying straight to 100%. Canary at 10%, watch, then move.
2. Quota exhaustion. A job queues forever, or a deployment can't be created. Alert on quota utilisation; it is a slow-moving, entirely foreseeable failure.
3. Spot eviction mid-training. Expected behaviour, not a fault. Checkpoint, or use dedicated for the long single run your release depends on.
4. A dependent resource fails or is misconfigured. The workspace is only as available as its storage account. ZRS on storage in prod is cheap insurance.
5. A regional outage. The rare one, and the only one requiring architecture.

Within a region

• Multiple instances per deployment, spread by the platform. One instance is a single point of failure with a per-hour cost.
• Health probes tuned honestly. A liveness_probe with a 5-second initial delay against a container that takes 60 seconds to load a model produces a restart loop.
• Zone redundancy where the instance type supports it, and ZRS on the storage account.
• Keep the previous deployment alive through the rollback window. That is the recovery plan.

Across regions

Azure ML workspaces are regional and do not fail over. Multi-region means running two of everything, and the seam that makes it tractable is an Azure ML registry: the model artifact lives in the registry, and each regional workspace deploys the same version from it.

                 Azure ML registry  (models, environments, components)
                       │                        │
        ┌──────────────┘                        └──────────────┐
        ▼                                                      ▼
  mlw-fraud-eastus                                      mlw-fraud-westeurope
  ep-fraud (endpoint)                                   ep-fraud (endpoint)
        │                                                      │
        └────────────► Front Door / Traffic Manager ◄───────────┘
                          health-probed, priority or weighted

What this does and does not buy you: it gives you inference continuity. It does not replicate job history, metrics, or workspace-scoped assets — those are per-workspace and lost with the region. If your recovery requirement includes "we can show which run produced the model that scored this transaction", the registry plus your git history is the durable record, not the workspace.

Recovery objectives, stated plainly:

• Inference RTO — minutes, if the second region is warm. Hours, if you plan to build it during the incident.
• Training RTO — as long as it takes to get quota in the second region, which is why you request it in advance, not during the outage.
• Model RPO — zero, if the artifact is in a registry with geo-replication. Otherwise, whatever your last copy was.

The runbook you should actually have

Written down, tested once a quarter, three pages at most:

1. Model regression → shift traffic to the previous deployment; confirm; then investigate.
2. Endpoint down → check deployment health, then Application Insights, then the dependent resources the scoring script touches (storage, Key Vault) before assuming Azure ML is at fault.
3. Quota exhaustion → whose sweep is it, kill it, and who requests the increase.
4. Regional outage → the Front Door failover command, and who is allowed to run it.
5. Drift alert → the specific decision tree: retrain, investigate, or accept.

The one people skip is #5, and it is the one that turns a monitoring feature into an operational practice.

---

Next: Interview Questions →

← Back to the Azure Machine Learning overview · ← Previous: Integrations