3. Architecture

Core Concepts named the parts. This page traces what actually happens when you press go — on a training job and on a scoring request — because almost every hard bug on this platform lives in the gap between those two paths.

What happens when you submit a job

You type az ml job create -f job.yml. Roughly this follows:

[Image Prompt: 2D minimalistic numbered sequence diagram tracing an Azure Machine Learning training job from CLI submission through Entra ID token acquisition, workspace discovery, code snapshot upload to the default storage account, environment image build in the container registry, cluster scale-out, container start on a node, data read from a datastore using the compute managed identity, MLflow metric logging back to the workspace, model artifact write to storage, and cluster scale-in to zero, flat design, clean vector art style, white background]

1. Authenticate. The CLI acquires an Entra ID token for your identity (or the pipeline's federated credential) and calls ARM to resolve the workspace.
2. Discover the data plane. The workspace returns its regional data-plane address (<region>.api.azureml.ms). Job submission is a data-plane call, not an ARM deployment. This is why a job never appears in the resource group's deployment history.
3. Snapshot the code. The contents of the job's code directory are uploaded to the workspace's default storage account and given a snapshot ID. .amlignore (or .gitignore) controls what goes. Everything not excluded is uploaded — this is why the first job from a repo with a 3 GB data/ folder takes twenty minutes.
4. Resolve the environment. If the image already exists in ACR, it's reused. If not, Azure ML builds it — a build that runs on the platform's own build compute (or inside your managed VNet if you've configured one) and can take five to fifteen minutes. Cached thereafter.
5. Queue against compute. The cluster's autoscaler sees pending work and requests nodes. If quota is available, nodes allocate in a couple of minutes; if not, the job sits in Queued — not failed — possibly indefinitely. On Spot nodes, availability, not just quota, gates this.
6. Run the container. The image starts on the node, the snapshot is mounted, inputs are mounted or downloaded from their datastores using the compute's managed identity, and your command executes.
7. Stream telemetry. stdout/stderr and MLflow metrics flow back to the workspace continuously. This is how the studio shows you a live loss curve.
8. Write outputs. Anything under the job's output paths lands in storage; a registered model becomes a versioned asset with the job ID recorded as its lineage.
9. Scale in. After the idle window, the cluster releases nodes back to zero.

The three latency traps in that list, because "why did my two-minute job take twenty minutes" is the most common first-week question: the code snapshot upload, the environment image build, and the node allocation. All three are one-off per change; none of them are your training loop.

What happens when you score a request

Different path, different address, different identity model.

client ──HTTPS──▶ <endpoint>.<region>.inference.ml.azure.com
                        │  auth: key | aml_token | aad_token
                        ▼
                  traffic router  ── splits by deployment percentage
                        │
              ┌─────────┴─────────┐
              ▼                   ▼
        deployment "blue"    deployment "green"
        (N instances)        (M instances)
              │
              ▼
        scoring container
          init()   ← runs once at instance start; load the model here
          run(raw) ← runs per request
              │
              ▼
        response  +  logs/metrics → Application Insights

Points that matter operationally:

• init() runs once per instance, run() runs per request. Loading a model inside run() is the classic latency bug: every request pays the load cost.
• The traffic split is enforced at the router, per request, not per client. A 90/10 split does not mean 10% of users — it means roughly 10% of calls, distributed however they arrive.
• Health probes gate a deployment going live. A deployment whose container fails its readiness probe never receives traffic, which is a feature: a broken green cannot take down blue.
• Instances are billed whether or not requests arrive. There is no scale-to-zero.
• Autoscale is Azure Monitor autoscale on the deployment, driven by CPU or by a request metric — the same autoscale engine as VM scale sets, not an ML-specific one. It scales within a deployment; it does not shift traffic between deployments.
• Logs go to Application Insights, and — separately — you can enable data collection to capture request and response payloads to storage for drift analysis. Those are two different switches.

Control plane vs. data plane, drawn properly

[Image Prompt: 2D minimalistic diagram of three separate doors into one Azure Machine Learning workspace — a control plane door at management dot azure dot com governed by Owner, Contributor and AzureML Compute Operator roles, a workspace data plane door at region dot api dot azureml dot ms governed by the AzureML Data Scientist role, and an inference data plane door at endpoint dot region dot inference dot ml dot azure dot com governed by a scoring permission, key, or token, flat design, clean vector art style, white background]

	Control plane	Workspace data plane	Inference data plane
Address	management.azure.com	<region>.api.azureml.ms	<endpoint>.<region>.inference.ml.azure.com
Creates	Workspace, compute, endpoints, deployments (as ARM resources), role assignments, private endpoints	Jobs, data assets, environments, model registrations, metrics	Nothing — it serves
Auth	Entra ID, Azure RBAC	Entra ID, Azure RBAC	Entra ID token, Azure ML token, or a key
Key roles	Owner, Contributor, AzureML Compute Operator	AzureML Data Scientist, Reader	The endpoint /score/action permission
Shows in	Activity Log, deployment history, what-if	Job history, MLflow	Application Insights

Three failure patterns fall directly out of this table:

• "I'm Owner and I get 403 scoring." Control-plane ownership is not a data-plane grant. Assign the scoring permission to the caller's identity, or use a key while you debug.
• "It works in the studio and fails in the job." You are authorised on the storage account; the compute's managed identity is not. Grant Storage Blob Data Reader to that identity.
• "The pipeline can submit jobs but not create the cluster." Correct, and usually intentional: AzureML Data Scientist is a data-plane role. Add AzureML Compute Operator if the pipeline must manage compute, and think hard before you do.

⚠️ Role names and their exact permission sets have changed over releases — verify against current Azure docs before encoding them in policy.

Quota, and how throttling actually manifests

There is no autoscale that rescues you from quota. Three separate pools, all counted per subscription, per region:

Pool	Counted per	Symptom when exhausted
Dedicated vCPU, per VM family	subscription × region × family	Cluster stays at its current node count; jobs sit in Queued
Low-priority (Spot) vCPU	subscription × region (a separate pool)	Same, plus mid-run eviction when capacity is reclaimed
Managed online endpoint quota	subscription × region	Deployment creation fails outright with a quota error

Plus structural caps: endpoints per workspace, deployments per endpoint, and workspaces per resource group. ⚠️ All of these numbers vary by subscription type and region — read them from az ml compute list-usage and the portal's Usage + quotas blade rather than trusting any figure written down, including here.

The mental model to keep: Azure ML never buys you capacity you weren't granted. A cluster with max_instances = 40 and quota for 12 is a cluster with 12 nodes and a misleading config file.

Networking paths

Three connections to reason about, and people routinely secure one and forget the others.

1. You → workspace. Locked down with a private endpoint on the workspace (sub-resource amlworkspace) plus public_network_access = Disabled. Now the studio only works from inside the network or over VPN/ExpressRoute.
2. Compute → everything it needs. This is what the managed virtual network handles. In AllowInternetOutbound mode, compute reaches the internet for package installs and inbound is closed. In AllowOnlyApprovedOutbound, egress is limited to FQDN rules and private endpoints you declare — which means pip install from PyPI fails until you allow it, and the managed firewall doing the filtering is a billed resource.
3. Callers → online endpoint. Governed separately by the endpoint's own public network access setting and, for private access, a private endpoint.

The nearly universal mistake: workspace private, storage account still public. The workspace's front door is locked and the filing cabinet is on the pavement. Every dependent resource needs its own private endpoint. See Production.

Identity, in one picture

Four identities are in play and confusing them is the root of most access bugs:

Identity	Used for
Your user identity	Studio, CLI, submitting jobs
The workspace's managed identity	Workspace-level access to the dependent resources
The compute's identity (system- or user-assigned, per compute)	Reading datastores and Key Vault from inside a job
The online deployment's identity	Reading the model, Key Vault, and any resource the scoring script touches

A user-assigned managed identity used consistently across compute and deployments is the configuration that makes this tractable: one principal to grant, one to audit, one to rotate nothing on. Set it up in Deployment.

Common failure modes, and what they usually mean

Symptom	Usual cause
Job stuck in Queued for a long time	No quota in that VM family/region, or Spot capacity unavailable. Not a failure — it will wait
Job fails with a storage 403	The compute's identity lacks a data role on the storage account; yours has one
First job on a new environment takes 15 minutes	Image build. Subsequent jobs reuse the cached image
First job from a repo takes forever to start	Code snapshot upload — add an .amlignore
Spot job dies at 80%	Eviction. Checkpoint, or use dedicated for long single runs
Deployment creation fails on quota	Online endpoint quota is a separate pool from training vCPU
Endpoint returns 424	The scoring container returned an error — read the deployment logs, not the endpoint logs
Endpoint healthy, latency terrible	Model loading inside run() instead of init(), or too few instances
403 scoring while being subscription Owner	Control-plane role, data-plane call
pip install fails only in the managed VNet	AllowOnlyApprovedOutbound with no PyPI FQDN rule
Deleted the workspace, still being billed	Storage, ACR, App Insights and Key Vault survive workspace deletion
Can't recreate a workspace with the same name	Soft delete. It's in the recycle bin; recover it or purge it

The theme: on this service the interesting failures are about identity, quota, and caching, not about the model. Knowing which of the three you are looking at is most of the debugging.

---

Next: Getting Started →

← Back to the Azure Machine Learning overview · ← Previous: Core Concepts