3. Architecture

The machinery most tutorials skip: what actually happens between kubectl apply and a running container, where Azure's control plane ends and Kubernetes' begins, how three authorization systems stack, and the specific ways all of it fails.

The shape of the system

An AKS cluster is two halves in two subscriptions.

The managed control plane, in a Microsoft-owned subscription you cannot see: the API server (the only component anything talks to), etcd (the cluster's entire state), the scheduler (decides which node a pod lands on), the controller manager (runs the reconciliation loops for Deployments, ReplicaSets, endpoints, and more), and the cloud controller manager — the piece that translates Kubernetes objects into Azure resources.

Your data plane, in the MC_* node resource group: Virtual Machine Scale Sets running the node image, each node running a kubelet (the agent that starts and watches containers), a kube-proxy or eBPF equivalent (programs the node's networking for Services), a container runtime (containerd — Docker as a runtime has been gone for several Kubernetes versions), and the CNI plugin.

The two halves talk over a tunnel — historically tunnelfront/aks-link, now Konnectivity — initiated outbound from the nodes. This is why nodes need egress to the API server's FQDN and why a locked-down egress firewall breaks a cluster in a way that looks like a node problem.

[Image Prompt: 2D minimalistic architecture diagram of an AKS cluster showing the Microsoft-managed control plane with API server, etcd, scheduler, and cloud controller manager on one side, connected over an outbound tunnel to customer nodes running kubelet, containerd, and kube-proxy in a node resource group, flat design, clean vector art style, white background]

Tracing one operation: kubectl apply -f deployment.yaml

1. Authenticate. kubectl presents a token. With Entra integration, that's an Entra ID token obtained by the kubelogin credential plugin; with local accounts, it's a client certificate from the cluster's admin kubeconfig. The API server validates it.
2. Authorize. The API server asks its authorizers whether this identity may create a Deployment in this namespace — Kubernetes RBAC, or Azure RBAC if Azure RBAC for Kubernetes Authorization is enabled (see below).
3. Admit. Admission controllers run: built-ins, then any mutating webhooks (which can change the object — this is how sidecar and identity webhooks work), then validating webhooks (the Azure Policy add-on's Gatekeeper constraints live here). A webhook that's down can block every write to the cluster; its failurePolicy decides whether that's fail-open or fail-closed.
4. Persist. The object is written to etcd. At this instant kubectl returns success — and nothing has run yet. Everything after this is asynchronous reconciliation. Internalising this is the difference between reading Kubernetes error states fluently and guessing.
5. Reconcile. The Deployment controller notices a Deployment with no matching ReplicaSet and creates one. The ReplicaSet controller notices N desired and 0 existing pods and creates pod objects — with no node assigned.
6. Schedule. The scheduler filters nodes (does it have enough requested CPU and memory? do taints match tolerations? do node selectors, affinity rules, and topology spread constraints allow it? is there a free pod slot within max_pods?), scores the survivors, and binds the pod to the winner. If no node passes, the pod sits Pending — and that's the signal the cluster autoscaler watches for.
7. Start. The kubelet on that node sees the binding, asks the CNI plugin for an IP, pulls the image via containerd (using the kubelet identity to authenticate to ACR), mounts volumes (CSI driver attaches an Azure disk, if any), and starts containers.
8. Become ready. The readiness probe passes; the endpoints controller adds the pod IP to the Service's endpoint list; kube-proxy (or Cilium's eBPF dataplane) programs the node so traffic to the Service IP reaches it.

Two side-effects worth naming, because they're where Azure re-enters: if the Service is type LoadBalancer, the cloud controller manager creates an Azure load-balancer rule and public IP in the node resource group. If a PVC is bound, the CSI driver creates and attaches an Azure managed disk to the node's VM.

[Image Prompt: 2D minimalistic numbered sequence diagram tracing a kubectl apply from authentication through admission, etcd persistence, controller reconciliation, scheduling, image pull, and pod readiness, flat design, clean vector art style, white background]

Control plane vs. data plane vs. Kubernetes RBAC

This is the AKS question people get wrong, and it's three layers, not two.

Layer	Endpoint	Governed by	Example permission
Azure control plane	management.azure.com	Azure RBAC on the managedClusters resource	Create a node pool, change the tier, upgrade the cluster, fetch credentials
Kubernetes API (the cluster's own control plane)	<cluster>.hcp.<region>.azmk8s.io	Kubernetes RBAC (Roles/ClusterRoles + bindings) — or Azure RBAC for Kubernetes Authorization if enabled	get pods, create deployments, read secrets
Workload data plane	Your application's endpoints	Your application	Whatever your app does

The bridge between the first two is the credential-fetch operation, and the two Azure built-in roles that matter are:

• Azure Kubernetes Service Cluster User Role — may call listClusterUserCredential. Gets you a kubeconfig that still requires you to pass Kubernetes RBAC.
• Azure Kubernetes Service Cluster Admin Role — may call listClusterAdminCredential. Gets you the cluster-admin certificate, which bypasses Entra ID and Kubernetes RBAC entirely.

That second role is the AKS privilege-escalation path. A user with it doesn't need any Kubernetes permission at all; the admin kubeconfig is a break-glass credential. Disabling local accounts (--disable-local-accounts) removes it, which is why that flag appears on every AKS hardening checklist.

When Azure RBAC for Kubernetes Authorization is enabled, in-cluster actions are authorized by Azure role assignments instead of Kubernetes RoleBindings, using four built-in roles — RBAC Reader, RBAC Writer, RBAC Admin, and RBAC Cluster Admin — scoped to the cluster or, with a namespace in the scope path, to a single namespace. The advantage is one identity system and one audit trail; the cost is that portable Kubernetes RBAC manifests no longer describe who can do what.

The classic mistake: being subscription Owner and being unable to kubectl get pods. Owner gives you the control plane, including the ability to grant yourself the cluster-admin credential — but not, by itself, any in-cluster permission on a properly configured cluster. That's correct behaviour, not a bug.

[Image Prompt: 2D minimalistic layered diagram showing Azure RBAC governing the AKS cluster resource, a credential-fetch bridge, and Kubernetes RBAC governing in-cluster actions, with the cluster-admin credential shown as a bypass path, flat design, clean vector art style, white background]

The network data path

Pod to pod, same node: through the node's local bridge or eBPF datapath. Never leaves the host.

Pod to pod, different nodes: depends entirely on the plugin. With Azure CNI (node subnet), pod IPs are real VNet IPs and the VNet routes them directly — no encapsulation, and pods are addressable from anywhere in the VNet. With Azure CNI Overlay, the pod CIDR is private to the cluster; traffic is encapsulated between nodes and the VNet never sees pod IPs. The trade is stark: node-subnet CNI burns VNet address space at max_pods × nodes and gives you direct addressability; overlay is nearly unbounded on IPs and gives you an extra hop plus indirection when debugging.

Pod to a Service: kube-proxy programs iptables or IPVS rules (or Cilium programs eBPF maps) so that packets to the Service's ClusterIP are DNAT'd to one of the ready endpoint pod IPs. There is no proxy process in the path — the ClusterIP is a rule, not a listener. This is why pinging a ClusterIP fails and confuses people.

Inbound from the internet: an Azure Load Balancer rule (created by the cloud controller manager for a LoadBalancer Service) forwards to a node port on every node; kube-proxy forwards on to a pod. With externalTrafficPolicy: Local the LB only targets nodes actually running a ready pod, preserving the client source IP at the cost of uneven load distribution.

Outbound to the internet: by default, SNAT through the cluster's outbound load balancer. Each public IP provides a fixed pool of SNAT ports divided among nodes — so a workload making many short-lived outbound connections (a scraper, a chatty HTTP client without connection pooling) can exhaust SNAT ports, producing intermittent connection timeouts that look like a DNS or application problem. The fixes, in order of preference: a NAT Gateway as the outbound type (vastly more ports and a longer idle timeout), more outbound IPs, connection pooling in the application, or a private endpoint so the traffic never goes out at all.

DNS: CoreDNS runs as a Deployment in kube-system. Every in-cluster name resolution goes through it, and it forwards unknown names to Azure DNS. Under-provisioned CoreDNS is a common source of tail-latency mysteries; ndots:5 in the default pod resolver config means external lookups can generate several failed queries first.

Scaling loops and their interactions

• HPA watches metrics and changes replica counts. Its default sync interval means it reacts in tens of seconds, not instantly.
• Cluster Autoscaler watches for Pending pods that would fit if a node existed, and adds nodes; separately it removes nodes that stay underutilised and whose pods can be moved. It cannot help a pod that would never fit on any node in any pool — a pod requesting 64 GiB in a pool of 16 GiB VMs stays Pending forever, which is the single most common "autoscaler is broken" report.
• Node Autoprovisioning removes the "any pool" constraint by creating right-sized pools on demand.
• KEDA drives HPA from external event sources, and can scale a Deployment to zero replicas — something plain HPA cannot do.

The loops compose in a specific order and each adds latency: metric appears → HPA scales pods → pods go Pending → autoscaler requests a node → VMSS provisions and the node registers (minutes, not seconds) → scheduler binds → image pulls → readiness. Plan for node provisioning latency explicitly: over-provision with low-priority "pause" pods if you need fast burst, and keep images small or pre-pull them.

Upgrades — the operation that defines AKS operations

An AKS upgrade is two distinct operations and people conflate them:

1. Control plane upgrade. Azure upgrades the API server to the target minor version. Fast, low-risk, and must happen first — Kubernetes tolerates nodes at most a couple of minor versions behind the control plane, never ahead.
2. Node pool upgrade. For each pool: cordon a node (mark unschedulable), drain it (evict its pods, respecting PodDisruptionBudgets and the eviction grace period), then replace it with a node running the new version. max_surge controls how many extra nodes are added at once, trading upgrade speed against transient capacity cost.

Separately, node image upgrades patch the OS without changing the Kubernetes version — more frequent and equally necessary. Auto-upgrade channels (patch, stable, rapid, node-image) automate this; planned maintenance windows confine it to hours you choose.

Where upgrades go wrong, in order of frequency:

• A PodDisruptionBudget that can never be satisfied — minAvailable: 1 on a single-replica Deployment means the drain can never evict that pod, and the upgrade hangs until it times out.
• No spare capacity — evicted pods can't be scheduled elsewhere, so the drain stalls.
• Deprecated APIs — a minor-version upgrade removes API versions your manifests or Helm charts still use. Check before upgrading; the cluster will tell you if you ask.
• Long terminationGracePeriods multiplied across many pods, turning a 20-minute upgrade into a three-hour one.

Failure modes worth recognising on sight

Symptom	Usual cause
Pod Pending forever	No node satisfies its requests/affinity/taints, or the autoscaler has hit max, or a subscription vCPU quota blocks new nodes
ImagePullBackOff	The kubelet identity lacks AcrPull, the image tag doesn't exist, or egress to the registry is blocked
CrashLoopBackOff	The application exits. Kubernetes is reporting your bug faithfully — read the previous container's logs
OOMKilled	The container exceeded its memory limit. The kernel killed it; Kubernetes just recorded it
Node NotReady	kubelet lost contact — node full on disk, kernel issue, or lost egress to the control plane
Intermittent outbound timeouts	SNAT port exhaustion. Move to a NAT Gateway
429 from ARM during scaling	Azure Resource Manager throttling the cluster's own control-plane calls, often from many simultaneous LB or disk operations
Cluster stuck in Failed provisioning state	A reconcile failed — commonly a policy denial, a lock on the node resource group, or a quota limit. az aks update with no changes forces a reconcile and often clears it
Everything works but nothing can be created	An admission webhook is down with failurePolicy: Fail

Consistency, durability, and what the SLA actually covers

etcd is a strongly consistent, quorum-replicated store, and the API server is a linearizable read-through to it — so kubectl get after kubectl apply reflects your write. Microsoft owns etcd's backup and replication; you never touch it, and you cannot restore it yourself. Your disaster recovery plan for cluster state is therefore your manifests in Git, not a database backup. This is the strongest practical argument for GitOps on AKS: the cluster is reproducible only to the extent its desired state lives outside it.

The Standard-tier SLA is on API server availability. Even at 100%, an API server outage does not stop running pods — the data plane keeps serving traffic because kubelets and kube-proxy work from local state. What you lose during a control-plane outage is changes: no scheduling, no scaling, no rollouts, no self-healing. Your application's availability comes from replica count, zone spread, disruption budgets, and probes — not from the cluster SLA.

---

Next: Getting Started →

← Back to the Azure Kubernetes Service overview · ← Previous: Core Concepts