8. Interview Questions

A regional Azure resource — plus a studio, CLI, and SDK over it — that provides tracked training execution on managed compute, versioned data/environment/model assets, and managed endpoints for serving.

The problem is not "how do I train a model"; that's scikit-learn. It is everything around it: which data version produced this model, which package versions, which metrics are real, how do I get a GPU for six hours without owning one for a year, how do I serve it behind authenticated HTTPS, and how do I put 10% of traffic on the new one and take it back. Azure ML answers that list as one resource with one RBAC surface and MLflow as its tracking API.

Bonus marks for noting the tracking layer is open-source MLflow, so the instrumentation in the training code is portable and only the tracking URI is Azure-specific.

Azure ML if the model is yours to train; Foundry if you are consuming someone else's. Training loops, feature engineering, compute clusters, model registries, MLOps → Azure ML. Prompts, retrieval, agents, evaluations over foundation models → Foundry.

The reason they blur: Foundry's classic hub architecture is literally an Azure ML workspace — Microsoft.MachineLearningServices/workspaces with kind = Hub. Same resource provider, different product. Foundry's newer architecture moved to Microsoft.CognitiveServices/accounts and does not.

A strong answer mentions the seam: a model fine-tuned in Azure ML can be registered and then consumed inside a Foundry application, with the model registry as the handoff.

Three levels of indirection:

• Model — a versioned artifact in the registry (fraud-rf:12), plus metadata and lineage.
• Deployment — a named running configuration behind an endpoint: one model version, one environment, one scoring script, an instance type and an instance count.
• Endpoint — a stable HTTPS address with an auth mode. Callers only ever know this.

Traffic is split across deployments by percentage, so blue=90 green=10 is a canary and blue=100 green=0 is a rollback — both one command, both seconds.

The failure to name is collapsing this into a single deployment called default that gets redeployed in place: you've kept the cost of the abstraction and thrown away the benefit.

• Compute instance — a single-user managed dev VM for authoring and debugging. Billed per hour while running. Always set idle shutdown; this is the #1 line on surprise bills.
• Compute cluster (AmlCompute) — autoscaling node pool for jobs. min_instances = 0 so an idle cluster costs nothing. The workhorse.
• Serverless compute — you specify size and count on the job; Azure ML manages the nodes. Right when you don't want to administer clusters. Less control over networking.
• Kubernetes compute — your AKS or Arc cluster with the Azure ML extension. For existing K8s investment, on-prem, or edge. You now own a Kubernetes cluster's operational life.

Cross-cutting: dedicated vs. Spot/low-priority. Spot is much cheaper and evictable — right for sweeps and checkpointed work, wrong for the single long run your release depends on. Quota is tracked separately for the two.

The workspace SKU is a red herring — Basic is effectively the only value since the Enterprise tier was retired, and it changes nothing operationally.

The axes that actually matter:

1. Compute — type (instance/cluster/serverless/Kubernetes), VM family and size, dedicated vs. Spot. This is where essentially all the money and all the capability live.
2. Serving shape — managed online (per instance-hour, no scale-to-zero) vs. batch (per job) vs. Kubernetes online (your cluster).
3. Network isolation mode — Disabled / AllowInternetOutbound / AllowOnlyApprovedOutbound, the last of which provisions a managed firewall you pay for hourly.
4. Dependent-resource SKUs — notably ACR Premium, which is required for private endpoints.

The trap: choosing a managed online endpoint for spiky, latency-tolerant traffic and paying for idle instances 24/7 when a batch endpoint would cost a fraction.

1. Entra ID token acquired; ARM called to resolve the workspace.
2. Workspace returns its regional data-plane address (<region>.api.azureml.ms). Job submission is a data-plane call — which is why jobs never appear in the resource group's deployment history.
3. The code directory is snapshotted and uploaded to the default storage account. Everything not excluded by .amlignore goes with it.
4. The environment image is resolved — reused from ACR if cached, otherwise built, which takes minutes.
5. The job queues against compute; the autoscaler requests nodes. No quota means Queued, not failed, potentially forever.
6. The container starts, inputs are mounted or downloaded from their datastores using the compute's managed identity, and the command runs.
7. stdout and MLflow metrics stream back live; outputs and registered models land in storage with the job ID as lineage.
8. The cluster scales back toward zero after the idle window.

The three latency traps — snapshot upload, image build, node allocation — are the answer to "why did my two-minute job take twenty minutes", and none of them are the training loop.

Three addresses, not two:

Concrete bugs:

• "I'm subscription Owner and I get 403 scoring." Control-plane ownership grants nothing on the inference data plane. The caller's identity needs the scoring permission explicitly.
• "Works in the studio, 403 in the job." You are authorised on the storage account; the compute's managed identity is not. Four identities are in play — yours, the workspace's, the compute's, and the deployment's — and the job uses the compute's.

The design fix: one user-assigned managed identity across compute and deployments, so every data-plane grant is made once to one principal.

azurerm covers the ARM-backed objects: workspace, the dependent resources, compute clusters and instances, identity, role assignments, private endpoints, diagnostics. azapi fills gaps — notably online endpoints and deployments, and registries — by talking to the ARM API directly for resource types the provider hasn't modelled.

What no IaC tool creates, because they are data-plane objects: jobs, data assets, environments, components, model registrations, and the endpoint traffic split.

So you run two pipelines:

• Infrastructure (Terraform, slow cadence, platform team) — everything up to and including the endpoint shell.
• Artifacts (az ml YAML, fast cadence, ML team) — environments, models, deployments, traffic, with an evaluation job as the gate.

Two details that show you've actually done it: the provider "azurerm" { features { machine_learning { purge_soft_deleted_workspace_on_destroy } } } block, because workspaces soft-delete and destroy/apply otherwise fails on a reserved name; and ignore_changes = [traffic] on the endpoint, because the release pipeline owns the split and Terraform would otherwise drag it back every apply.

The workspace soft-deletes: assets and job history go with it, and the name is reserved for a retention period, so recreating with the same name fails until you recover or purge it (az ml workspace list-deleted / az ml workspace purge). ⚠️ Retention period and default-on status vary — verify against current docs.

The four dependent resources survive and keep billing: the storage account, Key Vault, Application Insights, and the container registry. Deleting the resource group is what stops the meter, which is why every hands-on exercise should start by creating a throwaway one.

Worse: Key Vault with purge protection enabled cannot be purged early. It is a one-way door — you have permanently reserved that vault name for the retention period, with no override and no support ticket that helps. Set it deliberately in prod; leave it off in dev.

Also surviving: models in a shared registry (a separate resource — which is the point), diagnostic data already ingested into Log Analytics, and anything under a resource lock, which will make terraform destroy fail outright.

Three separate pools, all per subscription, per region:

• Dedicated vCPU, per VM family. Exhaustion → jobs sit in Queued indefinitely. Not a failure, no error, just silence.
• Low-priority (Spot) vCPU — a separate pool. Plus mid-run eviction when Azure reclaims capacity.
• Managed online endpoint quota — a third pool. Exhaustion → deployment creation fails outright, which is at least loud.

Plus structural caps on endpoints per workspace and deployments per endpoint. ⚠️ All of it varies by subscription type and region; read it from az ml compute list-usage and the Usage + quotas blade.

The key design consequence: prod belongs in its own subscription, because a runaway hyperparameter sweep in dev sharing the subscription can starve a prod retrain of GPU quota. And GPU families commonly start at zero — request quota before you commit to a date, not after.

A cluster with max_instances = 40 against quota for 12 is a 12-node cluster with a misleading config file.

A data asset is a versioned pointer, not a copy. Registering my-data:3 records a URI, a type (uri_file / uri_folder / mltable), and metadata. It does not snapshot the underlying bytes.

So if the blob behind it is mutable, my-data:3 means something different tomorrow, and your "reproducible" pipeline reproduces different results from the same version string. This is the single most misunderstood thing in the asset model.

Real immutability comes from the storage layer: write to date-partitioned or content-hashed paths that are never overwritten, and/or enable blob versioning and immutability policies. Then a version string is a genuine promise.

Contrast with the code snapshot, which is a real copy — the job's code directory is uploaded and frozen at submission.

Environments. dev and staging as resource groups; prod in its own subscription — because vCPU and endpoint quota are per-subscription-per-region and dev must not be able to starve prod. Each has its own workspace, its own Terraform state file, and its own OIDC federated credential scoped to a protected branch/environment.

The promotion seam is a shared Azure ML registry in its own resource group. Training happens in dev or staging; the model is pushed to the registry once; staging and prod deploy the byte-identical artifact. No rebuild, one lineage.

Two pipelines. Infrastructure in Terraform (workspace, dependent resources, one user-assigned managed identity, clusters with min_instances = 0, endpoint shell, private endpoints, diagnostics). Artifacts in az ml YAML, version-pinned, triggered by a model landing in the registry.

Retraining. Event Grid on ModelRegistered, or a scheduled pipeline. Either way the pipeline is: train → evaluate against the incumbent → fail the build on regression → register → deploy green at 0% traffic → smoke-test green directly → canary at 10% → promote.

Serving. Managed online endpoint, aad_token auth, two deployments (blue/green), multiple instances each, autoscale on the deployment, private endpoint, APIM in front if there is more than one consumer.

Audit. This is the part the question is really testing. Every model version carries the job that produced it; the job carries the code snapshot, the environment version, and the data asset versions. Diagnostic settings ship workspace events to Log Analytics with 90-day-plus retention. Azure Policy enforces tags and denies public network access outside dev. And — because a data asset version is only a pointer — training data is written to immutable date-partitioned paths, so "which data produced this model" survives contact with a lawyer.

Cost guardrails. Idle shutdown mandatory, compute instances denied by policy in prod, a VM SKU allowlist, budgets per resource group, and a review of endpoint instance counts against actual QPS.

Stabilise first, diagnose second.

1. Check the traffic split. az ml online-endpoint show --query traffic. If a canary went out today, that is the prime suspect.
2. Roll back. az ml online-endpoint update --traffic "blue=100 green=0". Seconds. Then confirm it took effect before telling anyone it's fixed. Do not delete green yet — that's the evidence.
3. If there was no recent change, look at per-deployment metrics in Application Insights, not endpoint aggregates. Then in order: request concurrency vs. max_concurrent_requests_per_instance (queuing shows as latency before it shows as errors); a recent scale-out whose new instances are cold because init() loads a large model; and dependency latency in the scoring script — a Key Vault or storage or feature-lookup call that got slow.
4. 424s specifically mean the scoring container errored. Read the deployment logs (az ml online-deployment get-logs), not the endpoint metrics.
5. If the dependent resources are healthy and nothing changed on your side, check Azure Service Health for the region before concluding it's you.

Afterwards, the two questions that matter: why did a change reach enough traffic to hurt (was there a canary stage, and did anyone watch it?), and why did a human notice before an alert did?

What happened: most AmlCompute properties — vm_size, vm_priority, and subnet among them — are immutable, so any change forces replacement. Either someone edited the module, or someone changed the cluster in the portal and the plan is now reconciling it back. Only the scale settings are in-place-updatable.

Why it matters: replacing a cluster kills every queued and running job on it. On a Monday morning that is a lost weekend of training.

What to do:

1. Do not apply. Find out which it is: terraform plan -detailed-exitcode on a clean checkout tells you whether the repo changed or the cloud did. The Activity Log tells you who touched it.
2. If it's real drift (portal change), decide whether the manual change was right. If it was, encode it in the module and re-plan so the diff disappears. If it wasn't, schedule the replacement for a window when no jobs are running.
3. If it's an intended change, still schedule it: drain the cluster (az ml job list for running jobs), then apply.
4. Consider a new cluster alongside — cl-cpu-v2 — point new jobs at it, and remove the old one once drained. Additive change, zero blast radius. The same pattern as blue/green deployments.

Preventing the recurrence: scheduled drift detection with -detailed-exitcode in CI so drift is caught within a day rather than at the next release; and Azure Policy or a resource lock so people cannot resize prod compute in the portal at all. Also note the corollary for serving: never change a deployment's model or environment in place on the deployment holding 100% of traffic — that's a replacement too. Create a new deployment instead.

This is drift, and it is the failure mode none of the infrastructure telemetry will show you. The system is perfectly healthy and the model is wrong.

Where to look, in order:

1. Input distribution vs. the training baseline — data drift. A new customer segment, a new channel, a currency or unit change, a marketing campaign. This is the most likely cause and the earliest available signal.
2. Data quality — nulls, type violations, out-of-range values. An upstream schema change that your pipeline silently coerced is a classic: no error anywhere, garbage features.
3. Prediction distribution — if the output histogram has shifted, something upstream did.
4. Actual performance against ground truth, if you have any yet. For fraud, labels arrive weeks or months late, which is exactly why input drift is the signal you have to act on.
5. A silent upstream feature-pipeline change — someone "fixed" a transformation. Check the data asset versions the current deployment's model was trained against versus what's being computed now.

The mechanism: enable data collection on the deployment (writes request/response payloads to storage — note the privacy implications), then run model monitoring comparing production against the training baseline on a schedule, with signals for data drift, data quality, and prediction drift.

The organisational half, which is the part that actually fails: an alert with no runbook is noise. Decide in advance whether a drift alert triggers an automatic retrain, an investigation, or a documented shrug — and who owns it. Two months of silent degradation is not a tooling gap; it's a missing owner.