5. Deployment

From "it worked in the portal" to "it ships through a pipeline, in three environments, and someone can roll it back at 2 a.m."

The AKS-specific complication: there are two deployments

Every other topic in this article has one deployment story. AKS has two, and mixing them is the most common structural mistake:

Do not manage application manifests in Terraform. The kubernetes and helm Terraform providers exist and work, but coupling a five-second app rollout to a state-locked infrastructure apply means every deploy waits on infrastructure and every infrastructure failure blocks deploys. Worse, a provider that must talk to the cluster's API server can't plan when the cluster doesn't exist yet, producing the classic chicken-and-egg apply failure.

The clean split: Terraform builds the cluster and stops at the API server boundary. Whatever lives inside the cluster is delivered by GitOps or by a separate application pipeline. The one justified exception is bootstrapping the GitOps operator itself — and even that is better done as the Flux cluster extension, which is an ARM resource and therefore genuinely infrastructure.

Tool order

All three apply to AKS and all three appear below.

Terraform module shape

A small, parameterised module: variables.tf, main.tf, outputs.tf. The goal is one module and three .tfvars files, not three copies of the code.

variables.tf

variable "name_prefix" {
  description = "Short workload identifier, used in every resource name."
  type        = string
}

variable "environment" {
  description = "dev | staging | prod"
  type        = string
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging, or prod."
  }
}

variable "location" {
  type    = string
  default = "uksouth"
}

variable "kubernetes_version" {
  description = "Minor version only, e.g. 1.31. AKS selects the latest supported patch."
  type        = string
}

variable "system_node_vm_size" {
  type    = string
  default = "Standard_D4ds_v5"
}

variable "user_node_vm_size" {
  type    = string
  default = "Standard_D8ds_v5"
}

variable "user_node_min_count" {
  type    = number
  default = 1
}

variable "user_node_max_count" {
  type    = number
  default = 10
}

variable "availability_zones" {
  type    = list(string)
  default = ["1", "2", "3"]
}

variable "admin_group_object_ids" {
  description = "Entra ID group object IDs granted cluster-admin. Groups, never users."
  type        = list(string)
}

variable "acr_id" {
  description = "Resource ID of the ACR the kubelet identity may pull from."
  type        = string
}

main.tf

terraform {
  required_version = ">= 1.9"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
    azapi = {
      source  = "Azure/azapi"
      version = "~> 2.0"
    }
  }
}

provider "azurerm" {
  features {}
}

locals {
  is_prod = var.environment == "prod"

  tags = {
    Environment = var.environment
    Workload    = var.name_prefix
    ManagedBy   = "terraform"
  }
}

resource "azurerm_resource_group" "this" {
  name     = "rg-${var.name_prefix}-aks-${var.environment}"
  location = var.location
  tags     = local.tags
}

# --- Networking: bring your own VNet. AKS-managed VNets are fine for demos and
# --- a dead end for anything that needs peering, firewalls, or private endpoints.
resource "azurerm_virtual_network" "this" {
  name                = "vnet-${var.name_prefix}-${var.environment}"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  address_space       = ["10.10.0.0/16"]
  tags                = local.tags
}

resource "azurerm_subnet" "nodes" {
  name                 = "snet-aks-nodes"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.this.name
  address_prefixes     = ["10.10.0.0/20"]
}

# --- A user-assigned identity for the control plane, created explicitly so its
# --- role assignments can exist before the cluster does. A system-assigned
# --- identity would have to be granted permissions after creation, which is a
# --- race you lose on the first apply.
resource "azurerm_user_assigned_identity" "cluster" {
  name                = "id-${var.name_prefix}-aks-${var.environment}"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  tags                = local.tags
}

resource "azurerm_role_assignment" "cluster_network" {
  scope                = azurerm_virtual_network.this.id
  role_definition_name = "Network Contributor"
  principal_id         = azurerm_user_assigned_identity.cluster.principal_id
}

resource "azurerm_kubernetes_cluster" "this" {
  name                = "aks-${var.name_prefix}-${var.environment}"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  dns_prefix          = "${var.name_prefix}${var.environment}"

  kubernetes_version  = var.kubernetes_version
  sku_tier            = local.is_prod ? "Standard" : "Free"

  # Name the node resource group explicitly so it is predictable in policy,
  # cost reports, and scripts. It cannot be renamed later.
  node_resource_group = "rg-${var.name_prefix}-aks-${var.environment}-nodes"

  # --- Hardening: no certificate-based break-glass admin, Entra groups only.
  local_account_disabled    = true
  azure_policy_enabled      = true
  workload_identity_enabled = true
  oidc_issuer_enabled       = true   # required for workload identity
  role_based_access_control_enabled = true

  azure_active_directory_role_based_access_control {
    azure_rbac_enabled     = true
    admin_group_object_ids = var.admin_group_object_ids
  }

  api_server_access_profile {
    # In prod, prefer a private cluster or API server VNet integration.
    authorized_ip_ranges = local.is_prod ? ["203.0.113.0/24"] : null
  }

  default_node_pool {
    name                        = "system"
    vm_size                     = var.system_node_vm_size
    zones                       = var.availability_zones
    vnet_subnet_id              = azurerm_subnet.nodes.id
    only_critical_addons_enabled = true   # taints the pool: system workloads only
    auto_scaling_enabled        = true
    min_count                   = local.is_prod ? 3 : 1
    max_count                   = local.is_prod ? 5 : 3
    os_sku                      = "AzureLinux"
    max_pods                    = 50
    upgrade_settings {
      max_surge = "33%"
    }
    # Lets the provider rotate this pool in place instead of replacing the
    # entire cluster when vm_size or disk settings change. Read the plan.
    temporary_name_for_rotation = "systemtmp"
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.cluster.id]
  }

  network_profile {
    network_plugin      = "azure"
    network_plugin_mode = "overlay"          # not kubenet: retiring 31 March 2028
    network_policy      = "cilium"
    network_data_plane  = "cilium"
    load_balancer_sku   = "standard"
    outbound_type       = "managedNATGateway" # far more SNAT ports than the LB default
    service_cidr        = "172.16.0.0/16"     # must not overlap the VNet or anything peered
    dns_service_ip      = "172.16.0.10"
    pod_cidr            = "192.168.0.0/16"
  }

  auto_scaler_profile {
    balance_similar_node_groups = true
    scale_down_unneeded         = "10m"
  }

  automatic_upgrade_channel = "patch"       # patch-level Kubernetes upgrades, automatically
  node_os_upgrade_channel   = "NodeImage"

  maintenance_window_auto_upgrade {
    frequency   = "Weekly"
    interval    = 1
    duration    = 4
    day_of_week = "Tuesday"
    start_time  = "02:00"
    utc_offset  = "+00:00"
  }

  oms_agent {
    log_analytics_workspace_id = azurerm_log_analytics_workspace.this.id
  }

  key_vault_secrets_provider {
    secret_rotation_enabled = true
  }

  tags = local.tags

  lifecycle {
    ignore_changes = [
      # The autoscaler owns the live count; do not let Terraform fight it.
      default_node_pool[0].node_count,
    ]
  }
}

# --- Application workloads go in their own pool, so a noisy app cannot evict CoreDNS.
resource "azurerm_kubernetes_cluster_node_pool" "apps" {
  name                  = "apps"
  kubernetes_cluster_id = azurerm_kubernetes_cluster.this.id
  vm_size               = var.user_node_vm_size
  zones                 = var.availability_zones
  vnet_subnet_id        = azurerm_subnet.nodes.id
  os_sku                = "AzureLinux"
  auto_scaling_enabled  = true
  min_count             = var.user_node_min_count
  max_count             = var.user_node_max_count
  max_pods              = 50
  upgrade_settings {
    max_surge = "33%"
  }
  tags = local.tags

  lifecycle {
    ignore_changes = [node_count]
  }
}

# --- The kubelet identity is what pulls images. Grant AcrPull to THAT, not to
# --- the cluster identity, and not to a workload identity.
resource "azurerm_role_assignment" "acr_pull" {
  scope                = var.acr_id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
}

resource "azurerm_log_analytics_workspace" "this" {
  name                = "log-${var.name_prefix}-aks-${var.environment}"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  sku                 = "PerGB2018"
  retention_in_days   = local.is_prod ? 90 : 30
  tags                = local.tags
}

Where azapi earns its place

AKS ships features faster than the azurerm provider absorbs them — node autoprovisioning, newer add-on profiles, and preview cluster settings routinely land in the REST API months before they get an azurerm argument. Rather than clicking them on in the portal (creating permanent drift), patch the same resource with azapi:

# Enable a setting azurerm does not expose yet, without leaving Terraform.
resource "azapi_update_resource" "preview_feature" {
  type        = "Microsoft.ContainerService/managedClusters@2024-09-01"  # ⚠️ verify current API version
  resource_id = azurerm_kubernetes_cluster.this.id

  body = {
    properties = {
      # e.g. nodeProvisioningProfile = { mode = "Auto" }
    }
  }
}

Two cautions: preview features have no SLA and can change or disappear, and an azapi_update_resource that overlaps a property azurerm also manages will produce a permanent diff war between the two resources. Patch only what azurerm doesn't own.

outputs.tf

output "cluster_id" {
  value = azurerm_kubernetes_cluster.this.id
}

output "cluster_name" {
  value = azurerm_kubernetes_cluster.this.name
}

output "node_resource_group" {
  value       = azurerm_kubernetes_cluster.this.node_resource_group
  description = "AKS-owned. Do not lock it, do not hand-edit it."
}

output "oidc_issuer_url" {
  value       = azurerm_kubernetes_cluster.this.oidc_issuer_url
  description = "Needed to create federated credentials for workload identity."
}

output "kubelet_identity_object_id" {
  value = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
}

output "get_credentials" {
  value = "az aks get-credentials -g ${azurerm_resource_group.this.name} -n ${azurerm_kubernetes_cluster.this.name}"
}

Deliberately not output: kube_config_raw. It is a credential, it lands in plaintext in state, and anything that needs it can call az aks get-credentials with its own identity and its own audit trail.

The features {} block

provider "azurerm" { features {} } is mandatory even when empty. For AKS specifically it's less consequential than for Key Vault — but if your module also manages a Key Vault for cluster secrets, features { key_vault { purge_soft_delete_on_destroy = false } } decides whether a destroy leaves a soft-deleted vault holding its name hostage. See the teardown section.

The loop

terraform init -backend-config=envs/prod.backend.hcl
terraform plan  -var-file=envs/prod.tfvars -out=tfplan
terraform show tfplan | less        # read it — see the replacement warning below
terraform apply tfplan

Remote state and locking

Local state on a team is a footgun: it can't be shared, it holds secrets in plaintext, and two concurrent applies will corrupt it. Use the azurerm backend, which stores state in a blob and locks it with a native blob lease — no separate lock table, which is a genuine simplification over the AWS equivalent.

az group create -n rg-tfstate -l uksouth
az storage account create -n sttfstate$RANDOM -g rg-tfstate -l uksouth \
  --sku Standard_ZRS --min-tls-version TLS1_2 \
  --allow-blob-public-access false
az storage container create -n tfstate --account-name <account> --auth-mode login

# envs/prod.backend.hcl
resource_group_name  = "rg-tfstate"
storage_account_name = "sttfstateXXXX"
container_name       = "tfstate"
key                  = "aks/prod.terraform.tfstate"
use_azuread_auth     = true   # Entra ID rather than a storage account key

Enable blob versioning and soft delete on that container. A corrupted or truncated state file is recoverable in thirty seconds with versioning and a very bad week without it.

Workspaces or directories? Directories (a key per environment, as above) win for anything with different shapes per environment — and AKS environments differ in shape constantly, because prod has a different tier, different zone spread, private API access, and more node pools. Workspaces are better when environments are genuinely identical, which AKS environments almost never are.

Ansible — the day-2 half

Ansible's honest role here is operations on a cluster Terraform owns, not ownership of the cluster. The genuinely good fit is an upgrade runbook: a sequenced, resumable, idempotent procedure across the control plane and every node pool, with checks in between. Terraform can declare a version; it cannot easily orchestrate the twenty minutes of draining that follows.

# upgrade-cluster.yml — orchestrated AKS upgrade with pre-flight checks
- name: Upgrade an AKS cluster safely
  hosts: localhost
  connection: local
  gather_facts: false

  vars:
    resource_group: "rg-payments-aks-prod"
    cluster_name: "aks-payments-prod"
    target_version: "1.31.3"

  tasks:
    - name: Fetch current cluster state
      azure.azcollection.azure_rm_aks_info:
        resource_group: "{{ resource_group }}"
        name: "{{ cluster_name }}"
      register: aks

    - name: Fail fast if the cluster is not in a Succeeded provisioning state
      ansible.builtin.assert:
        that: aks.aks[0].provisioning_state == "Succeeded"
        fail_msg: >-
          Cluster is {{ aks.aks[0].provisioning_state }}. Resolve the failed
          reconcile before upgrading.

    # Pre-flight: an unsatisfiable PodDisruptionBudget is the single most common
    # cause of an upgrade that hangs for hours and then times out.
    - name: List PodDisruptionBudgets with zero allowed disruptions
      kubernetes.core.k8s_info:
        kind: PodDisruptionBudget
        api_version: policy/v1
      register: pdbs

    - name: Warn on PDBs that block eviction entirely
      ansible.builtin.debug:
        msg: "BLOCKING PDB: {{ item.metadata.namespace }}/{{ item.metadata.name }}"
      loop: "{{ pdbs.resources }}"
      when: item.status.disruptionsAllowed | default(0) == 0
      loop_control:
        label: "{{ item.metadata.name }}"

    - name: Upgrade the control plane, then the node pools
      azure.azcollection.azure_rm_aks:
        resource_group: "{{ resource_group }}"
        name: "{{ cluster_name }}"
        location: "{{ aks.aks[0].location }}"
        kubernetes_version: "{{ target_version }}"
        dns_prefix: "{{ aks.aks[0].dns_prefix }}"
        agent_pool_profiles: "{{ aks.aks[0].agent_pool_profiles }}"
        state: present

    - name: Confirm every node reports the target version
      kubernetes.core.k8s_info:
        kind: Node
      register: nodes

    - name: Assert node versions converged
      ansible.builtin.assert:
        that: >-
          nodes.resources
          | map(attribute='status.nodeInfo.kubeletVersion')
          | select('search', target_version)
          | list | length == nodes.resources | length

Idempotency, demonstrated: run it twice.

# First run
PLAY RECAP: ok=6  changed=1  unreachable=0  failed=0

# Second run
PLAY RECAP: ok=6  changed=0  unreachable=0  failed=0   ← changed=0 is the assertion

Authenticate the playbook with a managed identity on the runner where possible, or a service principal via AZURE_CLIENT_ID / AZURE_TENANT_ID and a federated credential — never a client secret in a vars file.

Bicep / ARM equivalent

// aks.bicep
@description('Short workload identifier')
param namePrefix string

@allowed(['dev', 'staging', 'prod'])
param environment string

param location string = resourceGroup().location
param kubernetesVersion string
param adminGroupObjectIds array
param logAnalyticsWorkspaceId string

var isProd = environment == 'prod'

resource clusterIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'id-${namePrefix}-aks-${environment}'
  location: location
}

resource aks 'Microsoft.ContainerService/managedClusters@2024-09-01' = {
  name: 'aks-${namePrefix}-${environment}'
  location: location
  sku: {
    name: 'Base'
    tier: isProd ? 'Standard' : 'Free'
  }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${clusterIdentity.id}': {}
    }
  }
  properties: {
    dnsPrefix: '${namePrefix}${environment}'
    kubernetesVersion: kubernetesVersion
    nodeResourceGroup: 'rg-${namePrefix}-aks-${environment}-nodes'
    enableRBAC: true
    disableLocalAccounts: true
    oidcIssuerProfile: { enabled: true }
    securityProfile: {
      workloadIdentity: { enabled: true }
    }
    aadProfile: {
      managed: true
      enableAzureRBAC: true
      adminGroupObjectIDs: adminGroupObjectIds
    }
    networkProfile: {
      networkPlugin: 'azure'
      networkPluginMode: 'overlay'
      networkPolicy: 'cilium'
      networkDataplane: 'cilium'
      loadBalancerSku: 'standard'
      outboundType: 'managedNATGateway'
      serviceCidr: '172.16.0.0/16'
      dnsServiceIP: '172.16.0.10'
    }
    agentPoolProfiles: [
      {
        name: 'system'
        mode: 'System'
        vmSize: 'Standard_D4ds_v5'
        osSKU: 'AzureLinux'
        availabilityZones: ['1', '2', '3']
        enableAutoScaling: true
        minCount: isProd ? 3 : 1
        maxCount: isProd ? 5 : 3
        upgradeSettings: { maxSurge: '33%' }
      }
      {
        name: 'apps'
        mode: 'User'
        vmSize: 'Standard_D8ds_v5'
        osSKU: 'AzureLinux'
        availabilityZones: ['1', '2', '3']
        enableAutoScaling: true
        minCount: 1
        maxCount: 10
        upgradeSettings: { maxSurge: '33%' }
      }
    ]
    addonProfiles: {
      omsagent: {
        enabled: true
        config: { logAnalyticsWorkspaceResourceID: logAnalyticsWorkspaceId }
      }
      azureKeyvaultSecretsProvider: {
        enabled: true
        config: { enableSecretRotation: 'true' }
      }
      azurepolicy: { enabled: true }
    }
  }
}

output clusterName string = aks.name
output oidcIssuerUrl string = aks.properties.oidcIssuerProfile.issuerURL
output nodeResourceGroup string = aks.properties.nodeResourceGroup

# Preview before committing — ARM's closest analogue to terraform plan
az deployment group what-if \
  -g rg-payments-aks-prod \
  -f aks.bicep \
  -p namePrefix=payments environment=prod kubernetesVersion=1.31 \
     adminGroupObjectIds='["<group-guid>"]' \
     logAnalyticsWorkspaceId=<workspace-id>

az deployment group create -g rg-payments-aks-prod -f aks.bicep -p @prod.bicepparam

CI/CD — OIDC, never a secret

The pipeline authenticates to Azure with workload identity federation: GitHub (or Azure DevOps) mints a short-lived OIDC token, Entra ID trusts that token for one specific repository and one specific environment, and exchanges it for an Azure access token. No client secret, no publish profile, nothing to rotate or leak.

# One-time setup: an app registration that trusts one repo and one environment.
APP_ID=$(az ad app create --display-name "gha-aks-payments" --query appId -o tsv)
az ad sp create --id "$APP_ID"

az ad app federated-credential create --id "$APP_ID" --parameters '{
  "name": "gha-prod",
  "issuer": "https://token.actions.githubusercontent.com",
  "subject": "repo:my-org/payments-infra:environment:prod",
  "audiences": ["api://AzureADTokenExchange"]
}'

# Scope the role at the resource group, not the subscription.
az role assignment create --assignee "$APP_ID" \
  --role "Contributor" \
  --scope "/subscriptions/$SUB/resourceGroups/rg-payments-aks-prod"

# .github/workflows/infra.yml
name: aks-infrastructure

on:
  pull_request:
    paths: ['infra/**']
  push:
    branches: [main]
    paths: ['infra/**']

permissions:
  id-token: write        # required to request the OIDC token
  contents: read
  pull-requests: write

jobs:
  plan:
    runs-on: ubuntu-latest
    environment: prod-plan
    defaults:
      run:
        working-directory: infra
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -backend-config=envs/prod.backend.hcl
      - run: terraform plan -var-file=envs/prod.tfvars -out=tfplan
      - uses: actions/upload-artifact@v4
        with: { name: tfplan, path: infra/tfplan }

  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: prod            # ← the manual approval gate lives on this environment
    defaults:
      run:
        working-directory: infra
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - uses: hashicorp/setup-terraform@v3
      - uses: actions/download-artifact@v4
        with: { name: tfplan, path: infra }
      - run: terraform init -backend-config=envs/prod.backend.hcl
      - run: terraform apply tfplan

env: ARM_USE_OIDC: true (or the provider's use_oidc = true) tells the azurerm provider to use the federated token rather than looking for a secret.

Deploying into the cluster

The application pipeline is a separate workflow with a different trust boundary. Two shapes:

Push (pipeline runs helm upgrade) — simple, familiar, and requires the pipeline to reach the API server. On a private cluster that means a self-hosted runner in the VNet or az aks command invoke, which runs a command from inside the cluster and neatly sidesteps network access:

az aks command invoke -g rg-payments-aks-prod -n aks-payments-prod \
  --command "helm upgrade --install payments ./chart -f values-prod.yaml" \
  --file .

Pull (GitOps) — Flux or Argo CD runs inside the cluster and reconciles from Git. The pipeline never touches the cluster; it commits a new image tag to a manifests repository. This is the better default for AKS because it works identically on private clusters, gives you drift correction for free, and makes the cluster's desired state auditable in Git. Flux is available as a cluster extension, so it's provisioned as infrastructure:

resource "azurerm_kubernetes_flux_configuration" "apps" {
  name       = "apps"
  cluster_id = azurerm_kubernetes_cluster.this.id
  namespace  = "flux-system"

  git_repository {
    url                      = "https://github.com/my-org/payments-manifests"
    reference_type           = "branch"
    reference_value          = "main"
    sync_interval_in_seconds = 60
  }

  kustomizations {
    name = "apps"
    path = "./clusters/prod"
  }

  depends_on = [azurerm_kubernetes_cluster_extension.flux]
}

Environments

The honest recommendation for AKS: one subscription per environment, one cluster per environment, namespaces for teams within a cluster.

• Subscription per environment. In Azure the subscription is the natural blast-radius, quota, and policy boundary — and AKS makes the quota point concrete, because a load test in dev that consumes the region's vCPU quota will stop prod from scaling. That is not a theoretical risk.
• Cluster per environment, not per team. Cluster sprawl is the dominant AKS cost failure: every cluster carries a system node pool, an upgrade cadence, an add-on surface, and someone's attention. Isolate teams with namespaces + network policy + resource quotas + Azure RBAC, and reach for separate clusters only when you have a real isolation requirement (compliance boundary, incompatible Kubernetes versions, genuinely hostile multi-tenancy).
• How they differ: tier (Free in dev, Standard/Premium in prod), node pool sizes and autoscaler bounds, zone spread (single zone in dev, three in prod), API server exposure (public with authorized IPs in dev, private in prod), log retention, and upgrade channel (rapid in dev so you find breakage first, patch in prod).
• Where policy enforces it, not convention. Assign at the management group scope so nobody can opt out:

# Prod: no Free-tier clusters, ever
az policy assignment create \
  --name "aks-require-paid-tier" \
  --scope "/providers/Microsoft.Management/managementGroups/mg-prod" \
  --policy "<built-in or custom policy definition id>"

The Azure Policy add-on for AKS extends this inside the cluster — Gatekeeper constraints enforced at admission, so "no privileged containers", "images only from our ACR", and "every pod has resource limits" are enforced by the cluster rather than by code review. Start in Audit effect, read the compliance report for a fortnight, then move to Deny. Going straight to Deny on an existing cluster breaks deployments the same afternoon.

[Image Prompt: 2D minimalistic pipeline diagram showing a git commit flowing through terraform plan, manual approval, and apply into dev, staging, and prod AKS clusters in separate subscriptions, with a parallel GitOps path syncing application manifests into the clusters, flat design, clean vector art style, white background]

Rollback and blast radius

"Undo" means different things at the two layers, and knowing which one you're in is the first question at 2 a.m.

Application layer — fast, safe, well-supported.

kubectl rollout status  deployment/payments
kubectl rollout undo    deployment/payments             # previous ReplicaSet
kubectl rollout history deployment/payments
helm rollback payments 3                                # previous Helm release

With GitOps, the rollback is git revert and waiting for the next reconcile — which has the enormous advantage of leaving an audit trail and keeping Git as the truth.

Infrastructure layer — slower, and sometimes not reversible. Re-applying the previous commit usually works. What doesn't reverse cleanly:

Operations that replace rather than update

Read these in a plan before approving it. # forces replacement in Terraform output is the warning.

The pattern that makes AKS infrastructure changes safe is blue/green at the node pool level: add a new pool with the new shape, cordon and drain the old one, delete it. That is a normal, reversible operation, and it's why "change the VM size" should never be an in-place edit on a production pool.

For the cluster itself, blue/green means a second cluster and a traffic shift at Front Door or DNS. Expensive, and the only real answer for a network-model change.

The Azure-specific traps

• Resource locks. A CanNotDelete lock on the node resource group breaks scaling, upgrades, and cluster deletion, with errors that never mention the lock. Lock the cluster's resource group if you must; never the MC_* group. Also remember a lock makes terraform apply fail in a way that looks exactly like a permissions problem.
• Soft delete and purge protection. If your module also creates a Key Vault with purge protection, terraform destroy leaves a soft-deleted vault holding its name — and the next apply fails with a name conflict. Same for a deleted cluster's diagnostic settings and any Recovery Services vault items.
• Orphaned disks and IPs. Deleting a cluster deletes its node resource group, but PVs created with a Retain reclaim policy live in your resource groups and bill indefinitely.
• Role assignments made outside Terraform — including the ones the portal creates when you "attach" an ACR through the UI — are invisible to state and survive a destroy.

Drift detection

AKS drifts more than most Azure resources, because there are three ways to change it: ARM, the Kubernetes API, and the cluster's own reconciler.

• Scheduled terraform plan in CI, nightly, failing the build on a non-empty diff. The single highest-value control. Add ignore_changes for genuinely autoscaler-owned fields (node counts) so the signal stays clean.
• az deployment group what-if for Bicep-managed clusters, accepting the noise caveat above.
• Azure Policy compliance state — the fastest way to see that someone created a Free-tier cluster or a pool without zones.
• GitOps drift correction for anything inside the cluster: Flux and Argo CD will revert a hand-edited Deployment automatically, which converts "drift" from a detection problem into a non-event. This is a real reason to prefer GitOps over pipeline-push on AKS.
• Activity log and Change Analysis to answer "who changed this and when" — every ARM write is recorded with the caller's identity.
• kubectl diff -f manifests/ to compare live cluster objects against your manifests before a reconcile.

When you find drift you want to keep, import it rather than reverting it:

terraform import azurerm_kubernetes_cluster_node_pool.gpu "<node-pool-resource-id>"

Teardown

terraform destroy -var-file=envs/dev.tfvars

What destroy will not remove:

• Soft-deleted Key Vaults created alongside the cluster, if purge protection is on — the name stays reserved and blocks the next apply.
• Anything behind a resource lock, including a lock someone applied to the node resource group.
• PersistentVolumes with a Retain reclaim policy and their managed disks, which continue to bill.
• Role assignments and diagnostic settings created outside Terraform, including anything the portal's "attach ACR" or "enable monitoring" buttons created.
• The Log Analytics workspace's ingested data, which bills for its retention period.
• Entra ID objects — app registrations, federated credentials, and groups are tenant-scoped and outside the subscription's lifecycle entirely.

Worth running monthly, regardless:

# Clusters still on the Free tier — no SLA
az aks list --query "[?sku.tier=='Free'].{name:name, rg:resourceGroup}" -o table

# Clusters approaching end of support ⚠️ compare against the current support policy
az aks list --query "[].{name:name, version:kubernetesVersion, rg:resourceGroup}" -o table

# Orphaned disks left behind by deleted PVCs
az disk list --query "[?diskState=='Unattached'].{name:name, rg:resourceGroup, gb:diskSizeGb}" -o table

# Unassociated public IPs — these bill forever and appear on no dashboard
az network public-ip list --query "[?ipConfiguration==null].{name:name, rg:resourceGroup}" -o table

Next: Integrations →

← Back to the Azure Kubernetes Service overview · ← Previous: Getting Started