8. Interview Questions

A managed, regional, private OCI registry — an HTTPS endpoint at <name>.azurecr.io that stores container images and other OCI artifacts, authenticates with Microsoft Entra ID, and is billed per tier-day plus storage plus egress.

The problem is threefold. Rate limits: a cluster pulling from Docker Hub looks like one client and gets throttled mid-deployment, with the root cause outside your contract. Boundary: your application images shouldn't sit on a public service outside your network and access controls. Provenance: once the registry is yours you can enforce immutable tags, scan on push, sign images, and answer "what exactly is running in prod" with a digest.

Before managed registries, teams ran registry:2 on a VM and inherited TLS rotation, an auth server, and a garbage collector nobody scheduled.

The registry is the only ARM resource: Microsoft.ContainerRegistry/registries, sitting in a resource group, in a subscription, in a tenant. Its name is globally unique and alphanumeric only, because it becomes the DNS login server <name>.azurecr.io.

Everything below the registry is data, not infrastructure. A repository is a path string inside the registry, created implicitly by a push. A tag is a mutable pointer to a manifest. A manifest is a JSON document naming a config blob and an ordered list of layer blobs, each by SHA-256 digest. Layers are deduplicated registry-wide.

The consequence that matters: repositories are not ARM resources, so there is no repository-level Azure RBAC scope. AcrPull on the registry is pull on all of it. Repository granularity requires Premium scope maps and tokens.

Content is stored in Microsoft-managed storage in the registry's region, encrypted at rest, integrity-checked by digest — a corrupted layer upload is rejected rather than stored. Zone redundancy (Premium) spreads it across availability zones within the region.

Geo-replication (Premium) adds regional replicas behind a single login server, with Traffic Manager routing each client to the nearest healthy one. The critical property is that replication is asynchronous and eventually consistent: a push in one region is not instantly visible in another. That produces the classic symptom — the same tag running two different builds in two regions.

The fix is to deploy by digest, so a not-yet-replicated image fails loudly with a 404 and the deployment retries, rather than silently resolving a stale tag to the wrong manifest.

And the thing geo-replication does not give you: a backup. Deletes replicate.

ACR when the workload runs in Azure, which covers most cases: the pull stays on Azure's network, the credential is a managed identity with no secret, you can put a private endpoint in front of it, and there's no third-party rate limit on your deployment path.

GHCR is a genuine alternative if the whole supply chain is GitHub-native — you lose Azure private endpoints, Entra RBAC on the registry resource, and geo-replication.

Docker Hub for genuinely public distribution, or as an upstream that ACR caches from. Not as a production pull path.

The counter-case: if you need per-repository IAM with real identities, ACR's grain is the registry, and scope-map tokens are passwords rather than identities. There, the honest answer is one registry per boundary — or a different registry product.

• A fixed daily rate per tier, which buys an included storage allowance.
• Storage above that allowance, per GiB per day.
• Each geo-replication region, at roughly a full additional Premium daily rate.
• Egress leaving the region.
• ACR Tasks compute, per CPU-second above a free monthly allowance.

⚠️ Verify all figures against current Azure pricing.

Idle cost is most of it. The tier rate, all stored bytes, and every replica bill continuously whether or not a single pull happens. Only egress and Tasks are usage-driven — ACR is close to a fixed cost.

The trap: untagged manifests. Every push to a moving tag orphans the previous manifest, which keeps its unshared layers alive forever. A busy CI pipeline grows storage indefinitely until you enable the untagged-manifest retention policy.

1. The client HEADs each layer digest. Layers the registry already has return 200 and are skipped — deduplication is registry-wide, so a layer another team pushed counts.
2. Missing layers upload: POST opens a session, bytes go up, PUT with the digest finalises. The registry verifies the SHA-256 and rejects a mismatch.
3. The config blob uploads the same way.
4. The manifest is PUT at /v2/<repo>/manifests/<tag>. Only now does the image exist as a unit. The response carries Docker-Content-Digest — capture it in CI, it's the only unambiguous name for the build.
5. Tag semantics apply: if the tag existed and immutability is off, the old manifest becomes untagged but still stored and billed. With immutability on, the PUT is rejected.
6. Side effects: webhooks fire, Event Grid emits ImagePushed, Defender scanning picks it up, geo-replication begins, quarantine (if enabled) holds it unpullable.

A pull mirrors this, with one wrinkle: the layer GET usually returns a 307 redirect to Microsoft-managed blob storage, which is why an egress firewall allowing only *.azurecr.io lets manifests through and hangs on layers.

There's no instance count — the ceilings are tier-derived: read ops/minute, write ops/minute, and bandwidth all step up from Basic to Standard to Premium ⚠️ verify current figures. Exceeding them returns HTTP 429 with Retry-After.

Almost every ACR limit is scoped per registry, which is unusually simple for Azure. The per-subscription-per-region ones are ARM-side: number of registries, Tasks concurrency ⚠️ verify.

The realistic throttling scenario isn't steady traffic — it's a burst: an AKS node pool upgrade or a large scale-out where hundreds of kubelets pull the same large image within seconds. Mitigations in order of effectiveness: shrink the image (pays in storage, egress, pull time, and throttling simultaneously), geo-replicate to spread load across replicas, stagger the rollout, and only then raise the tier.

containerd and current docker honour Retry-After; hand-rolled pull loops don't, and are usually the actual cause.

Basic, Standard, and Premium differ in included storage, throughput, and webhook count — but the decision is almost never about size. The features are the difference: private endpoints, geo-replication, customer-managed keys, zone redundancy, scope maps and tokens, connected registry, and dedicated data endpoints are Premium only. Anonymous pull is Standard and Premium.

Basic is the trap tier — not because it's bad, but because it's what you pick for a proof of concept and inherit in production, and it cannot satisfy a "no public network access" baseline.

Moving between tiers is online, non-destructive, and takes seconds (az acr update --sku), so the upgrade itself is cheap. What isn't cheap is the network architecture you built around a public endpoint. And downgrading is dangerous: Premium → Standard silently invalidates private endpoints, replications, and scope maps, and Terraform may plan it cleanly.

• Disable the admin user and enforce it with Azure Policy. It's a shared password with full access, unattributable in logs, and readable by anyone with Contributor.
• Managed identity for every runtime, holding AcrPull at the registry scope. For AKS, az aks update --attach-acr creates exactly that assignment against the kubelet identity — no imagePullSecret, nothing to rotate.
• Workload identity federation (OIDC) for pipelines, holding AcrPush and nothing else. Not Contributor, because Contributor is a data-plane escalation path via the admin credential.
• Premium + private endpoint + privatelink.azurecr.io, with public network access disabled and dedicated data endpoints allow-listed in the egress firewall.
• Tag immutability, scan on push with Defender, and signature verification at admission with Notation (not the retiring content-trust feature).
• Platform engineers get Contributor through PIM, time-bound, not standing.

Where the built-ins fall short: there's no push-without-pull role and no repository-scoped role. Repository granularity means Premium scope maps and tokens — which are passwords — so the better answer is usually more registries.

Control plane is ARM at management.azure.com: create/delete the registry, change SKU, set network rules, add replications. Governed by Owner, Contributor, Reader.

Data plane is <name>.azurecr.io, the OCI Distribution API: pull, push, list repositories, delete manifests. Governed by AcrPull, AcrPush, AcrDelete.

The classic mistake: assigning Reader to a service account and expecting it to pull. It gets a 401. Both role sets are Azure RBAC assigned at the same scope, which is exactly why people assume one implies the other.

The mirror mistake is assuming Owner implies data access. It doesn't directly — but Owner and Contributor can grant themselves AcrPull, or read the admin user's password with az acr credential show. That escalation path is the strongest argument for keeping the admin user disabled.

Two more line-crossings worth naming: network rules are control-plane settings that break the data plane (perfect RBAC plus publicNetworkAccess: Disabled gives you a timeout, not a 403), and the portal's Repositories blade browses over the data plane from your browser, so it errors for an Owner on a fully private registry. That looks like a bug and isn't.

Replacement-forcing: name, resource_group_name, and location. All three destroy the registry and every image in it — there is no in-place move. Enabling customer-managed keys generally has to happen at creation too ⚠️ verify current behaviour, so retrofitting CMK may mean a new registry.

Almost everything else is an in-place update taking seconds: SKU changes, policy toggles, network rules, adding or removing replicas, identity changes.

Two non-replacing changes that are still dangerous: Premium → Standard silently invalidates private endpoints, replications, and scope maps; and setting public_network_access_enabled = false without a working private endpoint is an immediate, total outage of the pull path from a one-line diff.

Mitigations: lifecycle { prevent_destroy = true } on any prod registry, a CanNotDelete resource lock, and reading the plan rather than skimming it.

Read the error, because the four causes have distinct signatures:

1. 401 unauthorized → RBAC. Usually AcrPull was assigned to the cluster identity instead of the kubelet identity. Check azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id. Also possible: the assignment was made under two minutes ago and hasn't propagated.
2. DNS or timeout resolving <registry>.azurecr.io → the privatelink.azurecr.io private DNS zone isn't linked to the cluster's VNet, so nodes resolve the public IP and hang. Verify from a node with nslookup; you want a private address.
3. Manifest resolves, layers hang → the egress firewall allows *.azurecr.io but not the 307 redirect to Microsoft-managed storage. Enable dedicated data endpoints and allow-list <registry>.<region>.data.azurecr.io for every replica region.
4. 404 manifest unknown → you deployed a digest or tag that hasn't replicated to this region yet, or the tag was moved.

Note that this is one of the few Azure problems where the error text reliably discriminates between causes — so the first move is always to read it, not to reassign roles hopefully.

Diagnose. Confirm the throttling: ContainerRegistryRepositoryEvents and the registry's metrics will show the 429 burst, and it should correlate exactly with the upgrade window. The mechanism is that a node pool upgrade cordons, drains, and replaces nodes, and every replacement node pulls the full image set from cold — hundreds of concurrent pulls of the same large layers, against a per-registry ops-per-minute ceiling.

Fix, in the order I'd actually do it:

1. Stagger the upgrade. Lower maxSurge so fewer nodes come up at once. Costs nothing, buys time tonight.
2. Shrink the images. Multi-stage builds, a distroless or Mariner base, and dropping build tooling from the runtime layer. This is the highest-leverage fix because it pays in storage, egress, pull latency, and throttling headroom simultaneously.
3. Geo-replicate if the cluster spans regions — pulls then spread across replicas and stay local.
4. Raise the tier to Premium for headroom. Effective, and the one that costs money rather than engineering, so I'd want it to be a decision rather than a reflex.

What I wouldn't do: add retries to something. The clients already back off correctly; the problem is aggregate demand, not client behaviour.

Registries: one Premium registry per environment (dev, staging, prod), each in a home region, with the prod registry geo-replicated to all three deployment regions and zone redundant in each. Prod in its own subscription, so it's a distinct blast-radius, quota, and policy boundary.

Network: publicNetworkAccess: Disabled on prod. A private endpoint (sub-resource registry) in each region's VNet, one privatelink.azurecr.io private DNS zone linked to every consuming VNet. Dedicated data endpoints enabled and <registry>.<region>.data.azurecr.io allow-listed in the egress firewall — otherwise manifests resolve and layers hang. Accept that the portal's Repositories blade won't work from a laptop; browse via a jump host.

Identity: every runtime uses a managed identity with AcrPull. The build pipeline uses workload identity federation with AcrPush on staging only. Admin user disabled, denied by Azure Policy at the management group, alongside policies denying public network access and non-Premium SKUs in prod.

Supply chain: tag immutability on. CI tags by commit SHA, signs with Notation using a key in Key Vault, and generates an SBOM pushed as an OCI artifact alongside the image. Defender for Cloud scans. Ratify at AKS admission rejects unsigned images, and Azure Policy for AKS restricts image sources to the prod registry — that admission gate, not the registry, is the enforcement point.

Promotion: staging → prod by az acr import by digest, server-side, so no bytes cross the network and the digest is preserved end to end. Note the tension: an export-policy block on the source registry would also block this path, so the export policy goes on prod, not staging.

Traceability: diagnostic settings to Log Analytics with ContainerRegistryRepositoryEvents retained per the regulator's window. "Which build is running, who pushed it, when, and was it signed" is then a single KQL query joined against the digest.

First, scope the blast radius. An ACR problem does not stop running pods — images are already on the nodes. It stops new ones: deployments, scale-outs, node repairs, restarts, anything with imagePullPolicy: Always. So the impact is invisible for a while and then total. That determines urgency: I have minutes, not seconds, but the clock is running and autoscaling could start it early.

Then check what actually landed. ARM deployments are not transactional — a partial deployment leaves partial state. az deployment group show for the operation list, and the activity log for what changed. The specific thing to check is whether publicNetworkAccess got disabled before the private endpoint and DNS zone link succeeded, because that combination is a total pull outage.

Rollback is a redeploy of the previous template, in incremental mode, from the last known good commit. Most ACR properties are in-place updates that take seconds, so this is fast. If the outage is live and the private endpoint isn't working, the immediate mitigation is to re-enable public network access — accept the temporary posture regression, restore the pull path, then fix the endpoint properly.

Complete mode is the trap in the question. --mode Complete deletes every resource in the resource group that the template doesn't declare. A registry-only template run against a shared resource group would delete the private endpoint, the DNS zone links, and the diagnostic settings — the registry itself survives because it's in the template, but every prod pull fails within seconds and now you've lost the very networking you were trying to fix. The rule: never use complete mode against a resource group you don't fully own in one template.

Find out. Three detectors, and they answer different questions:

• Scheduled terraform plan -detailed-exitcode in CI, alerting on exit code 2, tells you that the resource differs from the module. This is the baseline, and nightly is enough for a registry.
• Azure Policy compliance state is the better detector here, because it reports continuously, names the exact non-compliant property, and catches registries Terraform doesn't manage at all. A "deny admin user" policy would have prevented it outright.
• The activity log tells you who and when — Microsoft.ContainerRegistry/registries/write events. In prod the registry's shape changes rarely enough that every one of those is worth an alert.

Get back to clean. Re-apply from the module — both properties are in-place updates, so the apply is seconds and non-disruptive to running workloads. But before applying, check what the public endpoint was unblocking: something has been pulling over it for three weeks, and turning it off without fixing that will break the same pipeline that motivated the change. Look at ContainerRegistryLoginEvents for callers whose source IP isn't in your VNet ranges, and for any authentication as the admin user.

Then rotate. The admin password was live for three weeks and may be in a pipeline variable, a config.json, or a chat message. Rotate both admin passwords, then disable the account.

Then fix the cause. The pipeline needed a path that didn't exist. Give it a private-network runner or an AcrPush federated identity, so the next person doesn't need to click. Drift that recurs is a design problem, not a discipline problem.