7. Production

The difference between "I made a cluster in the portal" and "I run this". Five pillars — security, cost, scaling, observability, reliability — plus the one that dominates AKS specifically and doesn't appear on the generic list: upgrade discipline.

[Image Prompt: 2D minimalistic diagram of five production pillars — security, cost, scaling, observability, reliability — arranged around a central AKS cluster resource, flat design, clean vector art style, white background]

Security

Identity and access

The hardening list, in the order it matters:

• Disable local accounts (--disable-local-accounts). This removes the certificate-based cluster-admin kubeconfig, which bypasses Entra ID and Kubernetes RBAC entirely. As long as it exists, anyone with the Azure Kubernetes Service Cluster Admin Role is cluster-admin regardless of every in-cluster control you've configured. Establish Entra access before you set this, or you'll lock yourself out.
• Entra ID integration with Azure RBAC for Kubernetes Authorization, assigned to groups, never to individual users. Grant Azure Kubernetes Service RBAC Reader broadly, RBAC Writer scoped to a namespace, and RBAC Cluster Admin to a small group covered by Privileged Identity Management so it's time-bound and approved rather than standing.
• Namespace-scoped role assignments. Azure RBAC scopes can include the namespace path, so a team gets writer on /…/managedClusters/aks-prod/namespaces/payments and nothing else.
• Workload identity for pods — never a connection string in a Secret, never the kubelet identity as a shared credential. See Integrations for the full mechanism.

Network exposure

• Private cluster or authorized IP ranges. A public API server with no IP restrictions is reachable by the whole internet; it's authenticated, but it's also an enumeration and credential-stuffing surface. Private clusters (API server behind Private Link) or API Server VNet Integration are the strong options; authorized IP ranges are the cheap one. Private clusters create real operational friction — your build agents must be in the VNet, or use az aks command invoke — so decide deliberately rather than reflexively.
• Network policy, on. Without it, every pod can reach every other pod in every namespace. Default-deny per namespace, then allow explicitly. This is the control that turns a compromised container into a contained incident rather than a cluster-wide one.
• Egress control. outbound_type = userDefinedRouting forces traffic through Azure Firewall, where you allow the FQDNs AKS itself requires (the API server, MCR, Ubuntu/Azure Linux package repositories, and the add-ons') plus your own. Under-allow and nodes fail to register; over-allow and you've bought a firewall for nothing. Microsoft publishes the required FQDN list ⚠️ check the current list before designing the rules.
• No public IPs on nodes. The default. Keep it.

Workload hardening

The Azure Policy add-on runs Gatekeeper and enforces at admission. The initiative worth starting with is the Kubernetes cluster pod security baseline. The rules that matter most:

• No privileged containers, no hostNetwork, no hostPID, no host path mounts.
• Read-only root filesystem, non-root user, dropped capabilities.
• Images only from your own ACR — this alone stops a large class of supply-chain problems.
• Every container has CPU and memory limits (without which one pod can starve a node).

Start every constraint in Audit, read the compliance report for two weeks, then move to Deny. Going straight to Deny on a live cluster breaks deployments the same afternoon and burns the team's willingness to run policy at all.

Alongside it: Microsoft Defender for Containers for runtime threat detection, image vulnerability scanning in ACR, and control-plane audit analysis. It costs per node ⚠️ verify current pricing, and it's the difference between finding out about a cryptominer from Defender and finding out from the bill.

Finally, keep node images current. A node running a months-old image has months of unpatched kernel and runtime CVEs, and it's the most common real-world AKS vulnerability — not anything exotic.

Cost

What you actually pay for

Line item	Notes
Node VMs	Essentially the entire bill. Billed per second of allocation, whether or not a pod is scheduled
Managed disks	OS disks for every node, plus every PV. Billed on provisioned size, not used
Cluster tier	Free $0 / Standard ~$0.10 / Premium ~$0.60 per cluster-hour ⚠️ verify current pricing. Rounding error next to nodes
Load balancer + public IPs	Per rule and per IP, plus data processing
Egress	Cross-zone and cross-region traffic is not free, and a chatty service mesh spread across three zones generates a surprising amount of it
Log Analytics ingestion	Per GB ingested and per GB retained. Container Insights on a busy cluster is routinely the second-largest line on the bill
Defender for Containers	Per node

The traps, in order of how much money they've cost people

1. Cluster sprawl. Every cluster needs a system node pool (which never scales to zero), its own monitoring, and its own attention. Ten clusters at one node each cost more than one cluster at ten nodes and cost far more in human time. Consolidate with namespaces.
2. Requests set far above actual usage. The scheduler bin-packs on requests, so a fleet requesting 2 CPU and using 0.2 runs at 10% utilisation and you pay for the other 90%. This is the single biggest lever on most clusters. Use the VPA in recommendation mode, or Container Insights' own recommendations, to right-size.
3. Autoscaler bounds set as an afterthought. A min_count sized for a load test that finished in March.
4. Container Insights collecting everything. Default configuration captures all container stdout across all namespaces, including kube-system. Tune the data collection rule.
5. Orphaned disks from Retain PVs and unassociated public IPs. They bill indefinitely and appear on no dashboard.
6. Idle dev clusters running overnight and at weekends. A dev cluster running 168 hours a week costs roughly four times one running 45.

Concrete optimisations

• Right-size requests first. Nothing else comes close in impact.
• Spot node pools for anything interruptible — CI runners, batch, stateless workers with a tolerant queue. Discounts are large and eviction is real; use a taint and a toleration so only workloads that opted in land there, and always keep a regular pool for the rest.
• Reservations or savings plans for the steady-state floor of your node capacity. You know your minimum; buy it at a discount and pay on-demand for the peak.
• Scale dev and staging to zero out of hours. A scheduled job that scales user pools to zero overnight is a few lines and a large percentage.
• Standard_D*ds with ephemeral OS disks — faster node provisioning and no OS disk charge.
• Set the Log Analytics daily cap, and choose the Basic logs tier for high-volume, low-query tables where it applies.

Scaling and limits

The scopes that matter

Azure quotas mean nothing without their scope, and AKS interacts with two different quota systems:

Limit	Scope it's counted at	Notes
vCPU quota per VM family	Per subscription, per region	The limit that actually stops you scaling. A load test in one cluster can exhaust the family quota for every cluster in that subscription and region — the strongest argument for a subscription per environment
Nodes per cluster	Per cluster; higher on Standard/Premium than Free ⚠️ verify current numbers
Node pools per cluster	Per cluster ⚠️ verify
Pods per node (max_pods)	Per node pool, set at creation and immutable	With node-subnet CNI this also determines VNet IP consumption
Pods per cluster	Per cluster, a function of nodes × max_pods ⚠️ verify
Public IPs, load balancer rules	Per subscription per region
SNAT ports	Per public IP, divided among nodes	Not a documented "quota" so much as a hard physical ceiling. NAT Gateway raises it dramatically

Raise vCPU quota through Azure Quota in the portal or az quota; it's usually automatic for modest increases and a support ticket for large ones. Request the increase before the migration, not during it — quota approval is not an incident-time operation.

Making autoscaling actually work

• Set requests accurately. Every scaling decision — HPA, cluster autoscaler, scheduler — is driven by requests. Wrong requests break all three simultaneously.
• Give the autoscaler headroom. Node provisioning takes minutes: VMSS allocation, boot, node registration, image pull. If your traffic ramps faster than that, over-provision with low-priority "pause" pods that real workloads evict instantly.
• A pod that fits no node stays Pending forever. The autoscaler won't invent a larger VM size — unless you're using node autoprovisioning, which will.
• Keep images small and consider pre-pulling. A 3 GB image adds minutes to every scale-out.
• Use topologySpreadConstraints, not just replica counts, to spread across zones and nodes. Three replicas on one node survive nothing.

Observability

Nothing is collected by default. The minimum viable setup:

1. Diagnostic settings on the cluster resource, routing control-plane logs to Log Analytics: kube-apiserver, kube-audit-admin (the write-only subset — full kube-audit is enormous), kube-controller-manager, cluster-autoscaler, and guard.
2. Managed Prometheus + Managed Grafana for metrics and dashboards.
3. Container Insights for container logs, with a tuned data collection rule.
4. Application Insights in the application itself, for traces and dependencies — cluster metrics tell you a pod restarted, not why the checkout call took nine seconds.

What to alert on

Alert on symptoms users feel and on conditions that will become symptoms:

• Pods in CrashLoopBackOff or restart count climbing.
• Pods Pending for more than a few minutes — the autoscaler has failed or hit a quota.
• Node NotReady, and node disk or memory pressure conditions.
• PersistentVolume nearly full — a silent killer for stateful workloads.
• API server latency and error rate from the control-plane metrics.
• Cluster approaching Kubernetes end-of-support — a scheduled check, not a metric, but the alert that saves the most pain.
• Certificate expiry on ingress TLS.

Deliberately not on the list: node CPU utilisation on its own. High utilisation on a well-packed cluster is the goal, not an incident.

The queries worth having saved

// What is restarting, and how often
KubePodInventory
| where TimeGenerated > ago(6h)
| where ClusterName == "aks-payments-prod"
| summarize Restarts = max(PodRestartCount) by Namespace, Name, ContainerName
| where Restarts > 0
| order by Restarts desc

// Why didn't the cluster scale up? The autoscaler narrates its own decisions.
AzureDiagnostics
| where Category == "cluster-autoscaler"
| where TimeGenerated > ago(1h)
| where log_s contains "no.scale.up" or log_s contains "max node group size reached"
| project TimeGenerated, log_s
| order by TimeGenerated desc

// Who changed what on the cluster resource itself
AzureActivity
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.CONTAINERSERVICE"
| where OperationNameValue !endswith "/read"
| project TimeGenerated, Caller, OperationNameValue, ActivityStatusValue, _ResourceId
| order by TimeGenerated desc

Reliability

Within a region

• Spread node pools across availability zones, and give the workload topologySpreadConstraints so pods actually land in different zones rather than clustering. A three-zone node pool with all three replicas on one node buys nothing.
• Set PodDisruptionBudgets — and set them satisfiably. minAvailable: 1 on a one-replica Deployment blocks every drain forever. The rule: PDBs only make sense alongside a replica count greater than one.
• Multiple replicas, real readiness probes, and correct terminationGracePeriodSeconds. A readiness probe that returns healthy before the app can serve is why rollouts drop traffic; a preStop sleep of a few seconds is the standard fix for connections draining after endpoint removal.
• Standard or Premium tier in production, for the API server SLA.
• Watch the zone-affinity trap for disks. A ReadWriteOnce managed disk pins its pod to one zone. Stateful workloads need either zone-redundant storage (Azure Files ZRS, ZRS disks where supported) or application-level replication across zones.

Across regions

There is no "multi-region AKS cluster". A cluster is regional, full stop. Multi-region means multiple clusters plus:

• Azure Front Door or Traffic Manager in front, health-probing each region's ingress.
• Replicated container images — an ACR with geo-replication (Premium SKU), so a regional registry outage doesn't stop pulls everywhere.
• A data story, which is the actual hard part: Cosmos DB multi-region writes, Azure SQL failover groups, or accepting an RPO.
• Fleet Manager if you're coordinating configuration and upgrades across many clusters.

Active/active is the honest goal; active/passive with a cold cluster is a plan that fails on the day, because a cluster nobody has deployed to in six months does not come up cleanly.

Backup and restore

Two separate concerns, and people conflate them:

• Cluster state — your manifests. The answer is Git plus GitOps, not a backup product. You cannot restore Microsoft's etcd yourself.
• Persistent volume data — real data needing real backup. Azure Backup for AKS covers cluster resources and disk snapshots; alternatively Velero. Test the restore; an untested backup is a belief, not a control.

Run the drill. Delete a node from the portal and watch pods reschedule. Cordon and drain a node during business hours. Fail a zone in a game day. The failure modes in Architecture are much cheaper to learn on purpose.

The sixth pillar: upgrade discipline

This is what actually consumes an AKS team's time, and no generic production checklist mentions it.

• Know your end-of-support date. Each Kubernetes minor version is supported for roughly a year ⚠️ verify the current support policy; AKS supports about N-2. Falling off the end means no patches, no support, and a forced upgrade under pressure.
• Automate patch upgrades with the patch auto-upgrade channel and node image upgrades with the NodeImage channel, both confined to a planned maintenance window.
• Do minor upgrades deliberately, dev first, with a soak period. Check for removed APIs before each one — a minor version can drop an API your Helm charts still use.
• Premium tier's LTS buys about two years on a designated version. Use it to deal with a genuine constraint, not to avoid building the upgrade habit; the eventual jump is larger.
• Every upgrade is a full node roll. Nodes are replaced, not patched in place. That means every upgrade exercises your PodDisruptionBudgets, readiness probes, and graceful shutdown — which is a feature: if an upgrade hurts, your workloads aren't as resilient as you thought, and a node failure at 3 a.m. would have hurt more.

---

Next: Interview Questions →

← Back to the Azure Kubernetes Service overview · ← Previous: Integrations