8. Interview Questions

AKS is managed Kubernetes. Microsoft runs the control plane — API server, etcd, scheduler, controller manager — in its own subscription, invisible to you, and handles its certificates, backups, and availability. You run the data plane: worker nodes, which are Virtual Machine Scale Sets in a resource group AKS creates in your subscription.

The consequences worth naming: you can't SSH to the control plane or restore etcd, the control plane is free in the Free tier, and everything you're billed for meaningfully is on your side of the line — node VMs, disks, load balancers, egress, and log ingestion.

Microsoft.ContainerService/managedClusters is a regional ARM resource in a resource group you choose, in a subscription, in a tenant — the standard Azure scope hierarchy.

The AKS-specific twist is the second resource group: AKS creates MC_<rg>_<cluster>_<region> (renameable only at creation) and owns the node scale sets, disks, load balancer, public IPs, and NSG inside it. It's in your subscription, on your bill, and under your policies — but it's AKS-managed. Don't lock it, don't hand-edit it.

Inside the cluster there's a second, entirely separate hierarchy: cluster → namespace → workload objects → pods → containers. Kubernetes objects have no ARM resource IDs and are governed by Kubernetes RBAC, not Azure RBAC — unless Azure RBAC for Kubernetes Authorization is enabled.

Free costs nothing for the control plane and gives you a best-effort SLO with no financially backed SLA, plus lower supported cluster sizes. Standard adds the uptime SLA on the API server and better control-plane scaling, for roughly $0.10/cluster/hour ⚠️ verify current pricing. Premium adds Long-Term Support — roughly two years on a designated Kubernetes version rather than the usual ~one — for roughly $0.60/hour.

The point that matters in an interview: the tier cost is a rounding error next to the node bill, so "we used Free to save money" is a bad answer. The reason to leave Free is the SLA. And the SLA covers the API server, not your workloads — pod availability comes from replicas, zone spread, disruption budgets, and probes.

Choose AKS when you need the Kubernetes API itself: CRDs and operators, admission webhooks, DaemonSets, StatefulSets, a service mesh, GPU device plugins, hostNetwork, custom schedulers, or existing Helm charts you want to run unchanged.

Choose Container Apps when you want what Kubernetes does — rolling deployments, service discovery, event-driven autoscaling, scale-to-zero — without owning a cluster. It's built on AKS internals plus KEDA, Dapr, and Envoy, so you're not giving up the machinery, just the operational ownership.

The test question: name the Kubernetes feature you need that Container Apps lacks. If you can't, you're about to buy a cluster to run three containers and then upgrade it forever.

Billed: node VMs (per second of allocation), every managed disk including node OS disks and PVs, the cluster tier per hour, load balancer rules and public IPs, egress including cross-zone traffic, Log Analytics ingestion and retention, and Defender for Containers per node.

Keeps billing when idle: all of it except egress. Nodes bill whether or not a pod is scheduled; the system node pool can never scale to zero; disks bill on provisioned size regardless of use; and PVs with a Retain reclaim policy outlive their PVCs and bill indefinitely. Kubernetes' own scale-to-zero applies to pods, not nodes — you only recover node cost if the cluster autoscaler or node autoprovisioning removes them, which takes minutes and never touches the system pool's floor.

Authenticate (Entra token via kubelogin, or a client certificate with local accounts) → authorize (Kubernetes RBAC, or Azure RBAC if that mode is on) → admission (mutating webhooks, then validating webhooks including the Azure Policy add-on's Gatekeeper) → persist to etcd, at which point kubectl returns success and nothing has run yet.

Then asynchronous reconciliation: the Deployment controller creates a ReplicaSet; the ReplicaSet controller creates unscheduled pod objects; the scheduler filters nodes on requests, taints, affinity, topology constraints, and free pod slots, scores the survivors, and binds. The kubelet on that node asks the CNI for an IP, pulls the image via containerd using the kubelet identity, mounts volumes via CSI, and starts containers. When the readiness probe passes, the endpoints controller adds the pod IP to the Service and kube-proxy (or Cilium eBPF) programs the node.

Naming the etcd boundary is the answer's centre of gravity: everything after it is eventual reconciliation, which is why "apply succeeded" and "it's running" are different claims.

kubenet — overlay pod IPs, user-defined routes. Route table limits cap cluster size, and it's retiring 31 March 2028. Don't start here.

Azure CNI (node subnet) — every pod gets a real VNet IP, so pods are directly addressable from the VNet and on-prem. The cost is address consumption: you must pre-allocate max_pods × nodes addresses, which is how teams exhaust a /16 and discover it can't be changed.

Azure CNI Overlay — nodes on the VNet, pods on a private overlay CIDR, NAT'd at the node. Near-unlimited scale on a small subnet. The default recommendation now. The trade is that pods aren't directly routable inbound from the VNet.

Azure CNI Powered by Cilium — either IPAM mode with an eBPF dataplane and Cilium network policy. Better performance and observability; check feature compatibility.

I'd choose Overlay, with Cilium for a new cluster, unless there's a hard requirement for pods to be addressable from outside the cluster — a legacy system that connects to pods, or an on-prem-initiated flow.

The critical framing: the network plugin is chosen at creation and effectively immutable. Changing it means a new cluster and a traffic migration. It's the highest-stakes decision in an AKS design review.

Three layers, not two:

• Azure control plane (management.azure.com) — Azure RBAC on the managedClusters resource. Governs creating node pools, changing the tier, upgrading, and fetching credentials.
• Kubernetes API (<cluster>.hcp.<region>.azmk8s.io) — Kubernetes RBAC, or Azure RBAC for Kubernetes Authorization if enabled. Governs get pods, create deployments, read secrets.
• The workload's own data plane — your application.

The bridge is credential-fetch, and two built-in roles matter: AKS Cluster User Role calls listClusterUserCredential and gets a kubeconfig that still requires you to pass Kubernetes RBAC; AKS Cluster Admin Role calls listClusterAdminCredential and gets the certificate-based cluster-admin kubeconfig, which bypasses Entra ID and Kubernetes RBAC entirely.

The classic mistake is twofold. First, being subscription Owner and being surprised you can't kubectl get pods — that's correct behaviour on a hardened cluster. Second, and far worse: building careful in-cluster RBAC while leaving local accounts enabled and Cluster Admin Role assigned broadly, so all of it is bypassable with one az aks get-credentials --admin. The fix is --disable-local-accounts.

Microsoft Entra Workload ID, in four steps: enable the cluster's OIDC issuer; annotate a Kubernetes ServiceAccount with a user-assigned managed identity's client ID; create a federated identity credential on that identity trusting the cluster's issuer with subject system:serviceaccount:<namespace>:<name>; label the pod template azure.workload.identity/use: "true". The pod receives a projected service account token, the Azure SDK exchanges it with Entra ID for an access token, and that token is used against Key Vault. No secret exists anywhere.

For mounting the secret as a file, add the Key Vault provider for the Secrets Store CSI driver and a SecretProviderClass, authenticated by that same workload identity, with rotation enabled.

Two things to say without prompting: don't use the kubelet identity for this — granting it Key Vault access gives every pod on every node access, which is a lateral movement path. And AAD Pod Identity is retired; if a candidate (or a blog post) reaches for AzureIdentity CRDs, that's out-of-date knowledge.

If you're coming from AWS: this is IRSA, nearly feature-for-feature.

Replaces the whole cluster: anything in the network profile — plugin, plugin mode, network policy, service CIDR, pod CIDR, DNS service IP — plus dns_prefix and node_resource_group.

Replaces or rotates a node pool: vm_size, OS disk type or size, os_sku, max_pods, and availability zones. In Terraform, changing these on the default_node_pool historically forced replacement of the entire cluster; temporary_name_for_rotation exists so the provider rotates the pool instead. Read the plan output for # forces replacement — don't skim it.

Not reversible at all: a Kubernetes version downgrade. Upgrades are one-way, so the rollback for a bad upgrade is a new cluster — which is why you soak in dev first.

The safe pattern for node-shape changes is blue/green at the pool level: add a pool with the new shape, cordon and drain the old one, delete it. That's a normal operation with a normal rollback. For the cluster itself, blue/green means a second cluster and a traffic switch at Front Door — expensive, and the only real answer for a network-model change.

Two pipelines, deliberately separated, because they have different cadences and different blast radii.

Infrastructure: Terraform with the azurerm backend (state in a blob, locked with a native blob lease — no separate lock table). The pipeline authenticates with workload identity federation / OIDC against an Entra app registration whose federated credential trusts one repository and one environment. plan on pull request, apply on merge, manual approval gate on the prod environment. Never a client secret. azapi covers AKS features azurerm hasn't caught up with yet.

Applications: GitOps — Flux (available as an AKS cluster extension, so it's provisioned as infrastructure) or Argo CD reconciling from a manifests repository. The pipeline builds an image, pushes to ACR, and commits a new tag; the in-cluster operator pulls it. This works identically on private clusters, gives drift correction for free, and makes the desired state auditable in Git. The alternative — a pipeline running helm upgrade — needs network access to the API server, which on a private cluster means a VNet-resident runner or az aks command invoke.

The anti-pattern to name: managing application manifests with the Terraform kubernetes/helm providers. It couples a five-second app rollout to a state-locked infrastructure apply, and creates a chicken-and-egg planning problem when the cluster doesn't exist yet.

kubectl describe pod first — the scheduler writes its reason into the events, and it's usually explicit. Work through the candidates:

• Nothing could ever fit. The pod requests more CPU or memory than any node in any pool provides. The cluster autoscaler will not invent a bigger VM size; it only adds nodes to existing pools. Node autoprovisioning would. This is the most common cause.
• The autoscaler hit max_count. Check the pool's bounds.
• Subscription vCPU quota exhausted for that VM family in that region. The scale set fails to provision; az vm list-usage -l <region> shows it. This is why environments should be in separate subscriptions — a dev load test shouldn't be able to starve prod.
• Taints without tolerations, node selectors, affinity rules, or topology spread constraints that no node satisfies. only_critical_addons_enabled on the system pool taints it deliberately.
• max_pods reached on every node, or with node-subnet CNI, VNet IP exhaustion.
• PVC pinned to a zone with no schedulable node in that zone.

The authoritative source is the cluster-autoscaler diagnostic log category, which narrates its own decisions — no.scale.up events with reasons. If that category isn't enabled, enabling it is the first fix, because you're otherwise guessing.

Start by saying the constraint: a cluster is regional. Multi-region means multiple clusters, so this is a design about traffic, images, and data — not about a cluster feature.

Within each region: Standard or Premium tier for the API server SLA. Node pools spread across three availability zones, with topologySpreadConstraints so pods actually distribute rather than clustering on one node. A tainted system pool separate from application pools. Satisfiable PodDisruptionBudgets with replica counts above one. Real readiness probes and a preStop drain delay. Azure CNI Overlay with Cilium network policy, default-deny per namespace. Private API server, egress through Azure Firewall.

Across regions: two active clusters in paired regions, both serving. Azure Front Door in front with health probes per region and a WAF. ACR with geo-replication (Premium SKU) so a regional registry outage doesn't stop pulls. Identical GitOps configuration reconciling both from the same repository with per-cluster overlays — which is also the disaster recovery story, since rebuilding a cluster is a terraform apply plus a Flux sync.

Data is the hard part, and I'd lead with it in the interview. Payments implies strong consistency and durability requirements. Cosmos DB with multi-region writes if the model tolerates its consistency options; Azure SQL failover groups with an explicit, agreed RPO/RTO if not; and an idempotency key on every write path so a retry across regions doesn't double-charge anyone. For regulated data-residency requirements, region choice is a compliance decision before it's an availability one.

Say active/active, not active/passive. A passive cluster nobody has deployed to in six months does not come up cleanly on the day it's needed.

What I'd defer: service mesh unless mutual TLS or fine-grained traffic shifting is a stated requirement — it's a large operational commitment for benefits many teams don't use.

An AKS upgrade is control plane first (fast), then a per-pool cordon, drain, replace cycle. A stalled upgrade is almost always a stalled drain, and there are three usual causes:

1. An unsatisfiable PodDisruptionBudget — minAvailable: 1 on a single-replica Deployment means the eviction API can never allow the eviction, so the drain retries until it times out. Find it: look for PDBs with disruptionsAllowed: 0.
2. No spare capacity — evicted pods can't be scheduled anywhere, so the drain waits. Check for Pending pods and whether the autoscaler is able to add nodes (quota, max_count).
3. Long terminationGracePeriodSeconds multiplied across many pods, turning twenty minutes into three hours. Arithmetic, not a fault.

What I'd do: don't cancel mid-flight if avoidable — a half-upgraded pool is a worse state. Identify the blocking PDB and temporarily relax it, or scale the affected Deployment up so the budget can be met. Then let it complete. Afterwards, fix the underlying issue: PDBs only make sense with more than one replica, and if an upgrade hurts, the workload wouldn't have survived a node failure at 3 a.m. either.

Prevention: pre-flight checks in the upgrade runbook (the Ansible playbook in Deployment does exactly this), max_surge set so replacement nodes come up before old ones drain, a planned maintenance window, and upgrading dev first with a soak period.

And the thing to say without being asked: there is no downgrade. If the new version breaks something, the fix is forward or a new cluster — which is why the dev soak is not optional.

Detect: a scheduled terraform plan in CI, nightly, failing on a non-empty diff — the single highest-value control. Supplement with Azure Policy compliance state (catches shape violations like a Free-tier cluster or a pool without zones) and the activity log, which records every ARM write with the caller's identity, so "who and when" is answerable in one query.

Decide before you act: is the change something you want? If yes, codify it — write it into the module, or terraform import the resource if someone created a whole node pool by hand. If no, terraform apply reverts it — but read the plan carefully first, because a revert of a node pool property may be a replacement, and reverting drift shouldn't cause an outage.

Add the ignore_changes nuance: some drift is legitimate and expected. The cluster autoscaler owns node_count; without lifecycle { ignore_changes = [node_count] } every plan shows a false positive and the team learns to ignore plan output — which is how real drift gets missed.

Prevent structurally: remove portal write access in prod (Reader plus a break-glass PIM role), enforce with Azure Policy at management-group scope, and use GitOps for anything inside the cluster — Flux and Argo CD revert hand-edited Kubernetes objects automatically, which turns in-cluster drift from a detection problem into a non-event.

And the AKS-specific note: changes inside the node resource group are a different category. That group is AKS-managed; hand-editing it isn't drift you should reconcile, it's damage you should undo, and the cluster's own reconciler may already be fighting it.