5. Deployment

Getting Started proved the service exists. This page makes it repeatable, reviewable, and reversible — a parameterised Terraform module, remote state, an Ansible playbook for day-two work, the Bicep equivalent, an OIDC pipeline, three environments, blue/green rollback, and drift.

Azure ML has a structural wrinkle you have to design around before writing a line of HCL: you are deploying two different things on two different cadences.

Trying to force models and jobs into Terraform state produces a state file that fights your data-science team every sprint. Trying to create workspaces with az ml from a notebook produces infrastructure nobody can audit. Draw the line at the ARM boundary, and put an evaluation gate, not a terraform apply, between a new model and production traffic.

[Image Prompt: 2D minimalistic pipeline diagram of an Azure Machine Learning deployment with two parallel tracks — an infrastructure track flowing from a git commit through terraform plan, approval, and apply into dev, staging and prod workspaces, and an artifact track flowing from a training job through model registration, an evaluation gate, registry promotion, and a blue-green traffic shift on an online endpoint, flat design, clean vector art style, white background]

Tool order for this topic

All three apply, with one honest caveat stated plainly: Ansible is the weakest of the three on this service. There is no azure_rm_machinelearningworkspace module. If your shop has no existing Ansible investment, use Terraform plus az ml and skip it — the section below exists because plenty of shops do, and doing it with azure_rm_resource is legitimate.

Terraform — the module shape

Three files, parameterised on environment. The features {} block is load-bearing here: both Azure ML workspaces and Key Vault soft-delete, and getting those flags wrong is the single most common reason a destroy/apply cycle fails in CI.

variables.tf

variable "environment" {
  type        = string
  description = "dev | staging | prod"
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging, or prod."
  }
}

variable "location" {
  type    = string
  default = "eastus"
}

variable "workload" {
  type        = string
  description = "Short workload name used in resource names."
  default     = "fraud"
}

variable "compute_clusters" {
  description = "Named training clusters. Keep min_nodes at 0 unless measured otherwise."
  type = map(object({
    vm_size     = string
    vm_priority = string # Dedicated | LowPriority
    min_nodes   = number
    max_nodes   = number
  }))
  default = {
    cpu = { vm_size = "Standard_DS3_v2", vm_priority = "Dedicated",   min_nodes = 0, max_nodes = 4 }
    gpu = { vm_size = "Standard_NC6s_v3", vm_priority = "LowPriority", min_nodes = 0, max_nodes = 2 }
  }
}

variable "public_network_access_enabled" {
  type    = bool
  default = true # flipped to false for staging/prod in the tfvars
}

variable "managed_network_isolation" {
  type        = string
  description = "Disabled | AllowInternetOutbound | AllowOnlyApprovedOutbound"
  default     = "Disabled"
}

variable "tags" {
  type    = map(string)
  default = {}
}

main.tf

terraform {
  required_version = ">= 1.6"
  required_providers {
    azurerm = { source = "hashicorp/azurerm", version = "~> 4.0" }
    azapi   = { source = "Azure/azapi",       version = "~> 2.0" }
  }
}

provider "azurerm" {
  features {
    machine_learning {
      # Purge the soft-deleted workspace so a destroy/apply cycle can reuse the name.
      # Set this to FALSE in prod: you want the recycle bin there.
      purge_soft_deleted_workspace_on_destroy = var.environment == "dev"
    }
    key_vault {
      purge_soft_delete_on_destroy    = var.environment == "dev"
      recover_soft_deleted_key_vaults = true
    }
    resource_group {
      # Refuse to delete an RG that still contains resources Terraform doesn't know about.
      prevent_deletion_if_contains_resources = true
    }
  }
}

provider "azapi" {}

locals {
  suffix = "${var.workload}-${var.environment}"
  tags = merge(var.tags, {
    workload    = var.workload
    environment = var.environment
    managed_by  = "terraform"
  })
  is_prod = var.environment == "prod"
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "this" {
  name     = "rg-aml-${local.suffix}"
  location = var.location
  tags     = local.tags
}

# ---------------------------------------------------------------------------
# One identity to rule them all. Compute and deployments both use this, so
# every data-plane grant is made once, to one principal.
# ---------------------------------------------------------------------------
resource "azurerm_user_assigned_identity" "aml" {
  name                = "id-aml-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  tags                = local.tags
}

# ---------------------------------------------------------------------------
# Dependent resources. Name them yourself — the portal's generated names are
# permanent and unreadable.
# ---------------------------------------------------------------------------
resource "azurerm_storage_account" "aml" {
  name                            = replace("stlaml${local.suffix}", "-", "")
  resource_group_name             = azurerm_resource_group.this.name
  location                        = azurerm_resource_group.this.location
  account_tier                    = "Standard"
  account_replication_type        = local.is_prod ? "ZRS" : "LRS"
  is_hns_enabled                  = false
  allow_nested_items_to_be_public = false
  shared_access_key_enabled       = false # force Entra ID auth to the datastore
  min_tls_version                 = "TLS1_2"
  tags                            = local.tags

  blob_properties {
    versioning_enabled = true
    delete_retention_policy { days = local.is_prod ? 30 : 7 }
  }
}

resource "azurerm_key_vault" "aml" {
  name                       = "kv-aml-${local.suffix}"
  resource_group_name        = azurerm_resource_group.this.name
  location                   = azurerm_resource_group.this.location
  tenant_id                  = data.azurerm_client_config.current.tenant_id
  sku_name                   = "standard"
  enable_rbac_authorization  = true
  soft_delete_retention_days = 7
  purge_protection_enabled   = local.is_prod # irreversible once true — read the note below
  tags                       = local.tags
}

resource "azurerm_container_registry" "aml" {
  name                = replace("cracraml${local.suffix}", "-", "")
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  sku                 = local.is_prod ? "Premium" : "Basic" # Premium is required for private endpoints
  admin_enabled       = false
  tags                = local.tags
}

resource "azurerm_log_analytics_workspace" "aml" {
  name                = "log-aml-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  sku                 = "PerGB2018"
  retention_in_days   = local.is_prod ? 90 : 30
  tags                = local.tags
}

resource "azurerm_application_insights" "aml" {
  name                = "appi-aml-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  workspace_id        = azurerm_log_analytics_workspace.aml.id
  application_type    = "web"
  tags                = local.tags
}

# ---------------------------------------------------------------------------
# The workspace
# ---------------------------------------------------------------------------
resource "azurerm_machine_learning_workspace" "this" {
  name                = "mlw-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  friendly_name       = "${var.workload} (${var.environment})"

  storage_account_id      = azurerm_storage_account.aml.id
  key_vault_id            = azurerm_key_vault.aml.id
  container_registry_id   = azurerm_container_registry.aml.id
  application_insights_id = azurerm_application_insights.aml.id

  public_network_access_enabled = var.public_network_access_enabled
  # Block the legacy path that lets a notebook mint storage keys.
  # ⚠️ verify the current argument name against the azurerm provider docs.
  # v1_legacy_mode_enabled = false

  identity {
    type         = "SystemAssigned, UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.aml.id]
  }

  managed_network {
    isolation_mode = var.managed_network_isolation
  }

  tags = local.tags
}

# ---------------------------------------------------------------------------
# Data-plane access for the ONE identity, on the dependent resources.
# This is the step people forget, and the cause of most "403 from the job".
# ---------------------------------------------------------------------------
resource "azurerm_role_assignment" "identity_blob" {
  scope                = azurerm_storage_account.aml.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.aml.principal_id
}

resource "azurerm_role_assignment" "identity_kv" {
  scope                = azurerm_key_vault.aml.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.aml.principal_id
}

resource "azurerm_role_assignment" "identity_acr_pull" {
  scope                = azurerm_container_registry.aml.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_user_assigned_identity.aml.principal_id
}

# The workspace's own system identity needs the same on storage.
resource "azurerm_role_assignment" "workspace_blob" {
  scope                = azurerm_storage_account.aml.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_machine_learning_workspace.this.identity[0].principal_id
}

# ---------------------------------------------------------------------------
# Compute clusters, from the map. min_nodes = 0 is the whole economic argument.
# ---------------------------------------------------------------------------
resource "azurerm_machine_learning_compute_cluster" "this" {
  for_each = var.compute_clusters

  name                          = "cl-${each.key}"
  location                      = azurerm_resource_group.this.location
  machine_learning_workspace_id = azurerm_machine_learning_workspace.this.id
  vm_size                       = each.value.vm_size
  vm_priority                   = each.value.vm_priority
  tags                          = local.tags

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.aml.id]
  }

  scale_settings {
    min_node_count                       = each.value.min_nodes
    max_node_count                       = each.value.max_nodes
    scale_down_nodes_after_idle_duration = "PT5M"
  }

  lifecycle {
    # Most cluster properties force replacement; recreating a cluster mid-sprint
    # kills every queued job on it. Make that an explicit decision, not a surprise.
    create_before_destroy = false
  }
}

# ---------------------------------------------------------------------------
# The online endpoint. azurerm covers the endpoint; azapi covers the deployment
# and the traffic split, which the provider does not model.
# ---------------------------------------------------------------------------
resource "azurerm_machine_learning_workspace_network_outbound_rule_fqdn" "pypi" {
  count = var.managed_network_isolation == "AllowOnlyApprovedOutbound" ? 1 : 0

  name                          = "allow-pypi"
  machine_learning_workspace_id = azurerm_machine_learning_workspace.this.id
  destination                   = "pypi.org"
}

resource "azapi_resource" "online_endpoint" {
  type      = "Microsoft.MachineLearningServices/workspaces/onlineEndpoints@2024-04-01"
  name      = "ep-${local.suffix}"
  parent_id = azurerm_machine_learning_workspace.this.id
  location  = azurerm_resource_group.this.location
  tags      = local.tags

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.aml.id]
  }

  body = {
    properties = {
      authMode            = "AADToken" # not "Key". A key is a standing secret.
      publicNetworkAccess = var.public_network_access_enabled ? "Enabled" : "Disabled"
    }
  }

  lifecycle {
    # The traffic split is owned by the release pipeline, not by Terraform.
    # Without this, every apply drags traffic back to whatever the code says.
    ignore_changes = [body.properties.traffic]
  }
}

output "endpoint_scoring_uri" {
  value = try(azapi_resource.online_endpoint.output.properties.scoringUri, null)
}

⚠️ Provider argument names — particularly managed_network, the outbound-rule resources, and the azapi API version — move between releases. Pin your provider version and check the registry docs before copying.

outputs.tf

output "workspace_id"   { value = azurerm_machine_learning_workspace.this.id }
output "workspace_name" { value = azurerm_machine_learning_workspace.this.name }
output "resource_group" { value = azurerm_resource_group.this.name }

output "identity_client_id" {
  description = "Use this in job YAML and deployment specs so everything runs as one principal."
  value       = azurerm_user_assigned_identity.aml.client_id
}

output "acr_login_server" { value = azurerm_container_registry.aml.login_server }
output "storage_account"  { value = azurerm_storage_account.aml.name }

The loop

terraform init -backend-config=backends/dev.hcl
terraform plan  -var-file=env/dev.tfvars -out=tfplan
terraform apply tfplan

env/prod.tfvars differs in exactly the ways you'd expect and no others:

environment                   = "prod"
public_network_access_enabled = false
managed_network_isolation     = "AllowOnlyApprovedOutbound"
compute_clusters = {
  cpu = { vm_size = "Standard_DS4_v2",  vm_priority = "Dedicated", min_nodes = 0, max_nodes = 16 }
  gpu = { vm_size = "Standard_NC24ads_A100_v4", vm_priority = "Dedicated", min_nodes = 0, max_nodes = 4 }
}

Note prod uses Dedicated for GPU. Spot is right for sweeps and wrong for the one long training run your release depends on.

Remote state and locking

Azure Storage, with native blob-lease locking — no DynamoDB-equivalent table, because the blob lease is the lock.

terraform {
  backend "azurerm" {}
}

backends/dev.hcl:

resource_group_name  = "rg-tfstate"
storage_account_name = "sttfstateamlprod"
container_name       = "tfstate"
key                  = "aml/dev.terraform.tfstate"
use_azuread_auth     = true   # no storage keys in CI
use_oidc             = true

Bootstrap the backend once, by hand, outside Terraform:

az group create -n rg-tfstate -l eastus
az storage account create -n sttfstateamlprod -g rg-tfstate -l eastus \
  --sku Standard_ZRS --allow-blob-public-access false \
  --allow-shared-key-access false --min-tls-version TLS1_2
az storage container create -n tfstate --account-name sttfstateamlprod --auth-mode login
az lock create --name protect-tfstate --lock-type CanNotDelete \
  --resource-group rg-tfstate

Turn on blob versioning and soft delete on that account. State corruption is rare and unrecoverable without them.

One state file per environment, keyed by path. A single state holding dev and prod means a bad terraform apply -target in dev can plan a destroy in prod.

Ansible — day-two operations

State it plainly: azure.azcollection ships no first-class Azure ML modules. What it does ship is azure_rm_resource, a generic wrapper over the ARM REST API, and that is genuinely useful for control-plane day-two work. Data-plane work (jobs, models, traffic) goes through az ml, wrapped with changed_when so the playbook is honest about idempotency.

playbooks/aml-day2.yml:

---
- name: Azure ML day-two operations
  hosts: localhost
  connection: local
  gather_facts: false

  vars:
    subscription_id: "{{ lookup('env', 'ARM_SUBSCRIPTION_ID') }}"
    resource_group: "rg-aml-fraud-prod"
    workspace: "mlw-fraud-prod"
    endpoint: "ep-fraud-prod"

  tasks:
    # --- Control plane, via the generic ARM wrapper -------------------------
    - name: Resize the GPU cluster for a scheduled retrain window
      azure.azcollection.azure_rm_resource:
        api_version: "2024-04-01"
        resource_group: "{{ resource_group }}"
        provider: MachineLearningServices
        resource_type: workspaces
        resource_name: "{{ workspace }}"
        subresource:
          - type: computes
            name: cl-gpu
        idempotency: true          # PATCH-compare before writing
        body:
          properties:
            properties:
              scaleSettings:
                minNodeCount: 0
                maxNodeCount: 8
                nodeIdleTimeBeforeScaleDown: "PT5M"
      register: cluster_resize

    - name: Stop all compute instances outside working hours
      azure.azcollection.azure_rm_resource:
        api_version: "2024-04-01"
        method: POST
        resource_group: "{{ resource_group }}"
        provider: MachineLearningServices
        resource_type: workspaces
        resource_name: "{{ workspace }}"
        subresource:
          - type: computes
            name: "{{ item }}"
          - type: stop
        body: {}
      loop: "{{ compute_instances | default([]) }}"

    # --- Data plane, via az ml ---------------------------------------------
    - name: Read current endpoint traffic
      ansible.builtin.command: >-
        az ml online-endpoint show
        --name {{ endpoint }} -g {{ resource_group }} -w {{ workspace }}
        --query traffic -o json
      register: current_traffic
      changed_when: false

    - name: Shift traffic to green only if it is not already there
      ansible.builtin.command: >-
        az ml online-endpoint update
        --name {{ endpoint }} -g {{ resource_group }} -w {{ workspace }}
        --traffic "blue=0 green=100"
      when: (current_traffic.stdout | from_json).get('green', 0) != 100
      changed_when: true

Where Ansible earns its place here: the "stop every compute instance at 19:00" job across twelve workspaces, the scheduled cluster resize before a nightly retrain, and cutting a region over during an incident. Where it hurts: it has no idea what Terraform intends. An Ansible-created cluster is drift the next time someone runs terraform plan. Keep Ansible to mutating properties Terraform ignores — which is exactly why the endpoint's traffic is in ignore_changes above.

The idempotency: true flag on azure_rm_resource makes it compare before writing; without it, every run reports changed. Run with --check in CI to prove a playbook is a no-op before letting it near prod.

Bicep / ARM

targetScope = 'resourceGroup'

@allowed(['dev', 'staging', 'prod'])
param environment string
param location string = resourceGroup().location
param workload string = 'fraud'

var suffix = '${workload}-${environment}'
var isProd = environment == 'prod'
var tags = { workload: workload, environment: environment, managed_by: 'bicep' }

resource uami 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'id-aml-${suffix}'
  location: location
  tags: tags
}

resource sa 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: replace('stlaml${suffix}', '-', '')
  location: location
  sku: { name: isProd ? 'Standard_ZRS' : 'Standard_LRS' }
  kind: 'StorageV2'
  properties: {
    allowSharedKeyAccess: false
    minimumTlsVersion: 'TLS1_2'
    allowBlobPublicAccess: false
  }
  tags: tags
}

resource kv 'Microsoft.KeyVault/vaults@2023-07-01' = {
  name: 'kv-aml-${suffix}'
  location: location
  properties: {
    tenantId: subscription().tenantId
    sku: { family: 'A', name: 'standard' }
    enableRbacAuthorization: true
    enableSoftDelete: true
    enablePurgeProtection: isProd ? true : null   // note: cannot be set to false once true
    softDeleteRetentionInDays: 7
  }
  tags: tags
}

resource acr 'Microsoft.ContainerRegistry/registries@2023-11-01-preview' = {
  name: replace('cracraml${suffix}', '-', '')
  location: location
  sku: { name: isProd ? 'Premium' : 'Basic' }
  properties: { adminUserEnabled: false }
  tags: tags
}

resource appi 'Microsoft.Insights/components@2020-02-02' = {
  name: 'appi-aml-${suffix}'
  location: location
  kind: 'web'
  properties: { Application_Type: 'web' }
  tags: tags
}

resource ws 'Microsoft.MachineLearningServices/workspaces@2024-04-01' = {
  name: 'mlw-${suffix}'
  location: location
  sku: { name: 'Basic', tier: 'Basic' }
  identity: {
    type: 'SystemAssigned, UserAssigned'
    userAssignedIdentities: { '${uami.id}': {} }
  }
  properties: {
    friendlyName: '${workload} (${environment})'
    storageAccount: sa.id
    keyVault: kv.id
    containerRegistry: acr.id
    applicationInsights: appi.id
    publicNetworkAccess: isProd ? 'Disabled' : 'Enabled'
    managedNetwork: {
      isolationMode: isProd ? 'AllowOnlyApprovedOutbound' : 'Disabled'
    }
  }
  tags: tags
}

resource cluster 'Microsoft.MachineLearningServices/workspaces/computes@2024-04-01' = {
  parent: ws
  name: 'cl-cpu'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: { '${uami.id}': {} }
  }
  properties: {
    computeType: 'AmlCompute'
    properties: {
      vmSize: 'Standard_DS3_v2'
      vmPriority: 'Dedicated'
      scaleSettings: {
        minNodeCount: 0
        maxNodeCount: isProd ? 16 : 4
        nodeIdleTimeBeforeScaleDown: 'PT5M'
      }
    }
  }
}

resource endpoint 'Microsoft.MachineLearningServices/workspaces/onlineEndpoints@2024-04-01' = {
  parent: ws
  name: 'ep-${suffix}'
  location: location
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: { '${uami.id}': {} }
  }
  properties: {
    authMode: 'AADToken'
    publicNetworkAccess: isProd ? 'Disabled' : 'Enabled'
  }
}

output workspaceName string = ws.name
output identityClientId string = uami.properties.clientId

az deployment group what-if \
  --resource-group rg-aml-fraud-prod \
  --template-file main.bicep \
  --parameters environment=prod \
  --mode Incremental

Data-plane deployment — the az ml half

The assets no IaC tool models. These belong in a second pipeline, owned by the ML team, triggered by a model landing in the registry.

azureml/environment.yml:

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: fraud-scoring
version: 4
image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu22.04:latest
conda_file: ./conda.yml

azureml/deployment-green.yml:

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: green
endpoint_name: ep-fraud-prod
model: azureml://registries/reg-mlops/models/fraud-rf/versions/12
environment: azureml:fraud-scoring:4
instance_type: Standard_DS3_v2
instance_count: 3
request_settings:
  max_concurrent_requests_per_instance: 2
  request_timeout_ms: 5000
liveness_probe:
  initial_delay: 30
  period: 10
  failure_threshold: 3
app_insights_enabled: true

Note every reference is version-pinned — versions/12, :4. @latest in a production deployment spec means your reproducible platform reproduces whatever happened most recently.

# Create green at zero traffic. Blue keeps serving throughout.
az ml online-deployment create -f azureml/deployment-green.yml --no-wait

# Smoke-test green directly, bypassing the traffic split:
az ml online-endpoint invoke --name ep-fraud-prod \
  --deployment-name green --request-file sample.json

# Then move the dial, in stages.
az ml online-endpoint update --name ep-fraud-prod --traffic "blue=90 green=10"

CI/CD — OIDC, never a secret

Workload identity federation. No client secret, no publish profile, no key in a variable group. The pipeline's Entra ID app trusts a token issued by GitHub for a specific repo, branch, and environment.

Set up the federated credential once:

az ad app create --display-name "gh-aml-prod"
# capture appId, then:
az ad app federated-credential create --id <appId> --parameters '{
  "name": "gh-main",
  "issuer": "https://token.actions.githubusercontent.com",
  "subject": "repo:my-org/my-repo:environment:prod",
  "audiences": ["api://AzureADTokenExchange"]
}'

The subject is the security boundary. repo:my-org/my-repo:environment:prod means only a job running against the protected prod environment can assume it — a PR from a fork cannot.

.github/workflows/infra.yml:

name: aml-infrastructure

on:
  push:
    branches: [main]
    paths: ['infra/**']
  pull_request:
    paths: ['infra/**']

permissions:
  id-token: write      # required for OIDC
  contents: read

jobs:
  plan:
    runs-on: ubuntu-latest
    environment: dev
    defaults: { run: { working-directory: infra } }
    env:
      ARM_USE_OIDC: true
      ARM_CLIENT_ID:       ${{ vars.AZURE_CLIENT_ID }}
      ARM_TENANT_ID:       ${{ vars.AZURE_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -backend-config=backends/dev.hcl
      - run: terraform validate
      - run: terraform plan -var-file=env/dev.tfvars -out=tfplan
      - uses: actions/upload-artifact@v4
        with: { name: tfplan-dev, path: infra/tfplan }

  apply-prod:
    needs: plan
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: prod           # protection rules = the human gate
    defaults: { run: { working-directory: infra } }
    env:
      ARM_USE_OIDC: true
      ARM_CLIENT_ID:       ${{ vars.AZURE_CLIENT_ID_PROD }}
      ARM_TENANT_ID:       ${{ vars.AZURE_TENANT_ID }}
      ARM_SUBSCRIPTION_ID: ${{ vars.AZURE_SUBSCRIPTION_ID_PROD }}
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -backend-config=backends/prod.hcl
      - run: terraform plan -var-file=env/prod.tfvars -out=tfplan
      - run: terraform apply -auto-approve tfplan

Apply the plan file, never re-plan at apply time. Otherwise the thing a human approved is not the thing that ran.

The second pipeline: models, evaluation, and traffic

Infrastructure CI/CD that does not gate on model quality is deploying a coin flip on a schedule. The artifact pipeline:

  promote-and-shift:
    runs-on: ubuntu-latest
    environment: prod
    steps:
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID_PROD }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID_PROD }}

      - name: Run the evaluation job and fail the build if it regresses
        run: |
          az ml job create -f azureml/evaluate.yml --stream \
            --set inputs.candidate=azureml://registries/reg-mlops/models/fraud-rf/versions/${{ inputs.version }}

      - name: Deploy green at zero traffic
        run: az ml online-deployment create -f azureml/deployment-green.yml

      - name: Canary at 10%
        run: az ml online-endpoint update --name ep-fraud-prod --traffic "blue=90 green=10"

Two properties worth copying: green is created at zero traffic (a failed deployment cannot affect users), and the evaluation job is a build step whose exit code matters. A model that fails the metric threshold never reaches step two.

Environments

Three environments, and the boundary between them is the thing to get right.

Prod in its own subscription is the recommendation because Azure ML's most painful limits — vCPU quota per family, online endpoint quota — are per subscription, per region. Sharing a subscription means a runaway hyperparameter sweep in dev can starve a prod retrain of GPU quota. That is not a theory; it is a Tuesday.

A shared Azure ML registry, in its own resource group and typically its own subscription, is what makes promotion real. ws-dev pushes the model; ws-staging and ws-prod deploy the byte-identical artifact. No rebuild, one lineage, one thing to audit.

Azure Policy is where the environment strategy stops being a convention:

• Deny Microsoft.MachineLearningServices/workspaces with publicNetworkAccess = Enabled outside dev.
• Deny compute instance creation in the prod subscription outright.
• Deny VM SKUs outside an allowed list (the GPU-cost guardrail).
• Require tags — workload, environment, cost-center — so the bill is attributable.
• Audit workspaces without a customer-managed key, if that's your bar.
• Deny public blob access and shared-key access on storage accounts in the ML resource groups.

Rollback and blast radius

Know which changes are in-place, which replace, and which are irreversible. This table is the one to internalise.

The rollback runbook, in order:

# 1. Model or scoring regression → shift traffic. Seconds.
az ml online-endpoint update --name ep-fraud-prod --traffic "blue=100 green=0"

# 2. Confirm it took effect before telling anyone it's fixed.
az ml online-endpoint show --name ep-fraud-prod --query traffic

# 3. Only then delete the bad deployment. Keeping it costs money;
#    deleting it before you've confirmed the rollback costs your evidence.
az ml online-deployment delete --endpoint-name ep-fraud-prod --name green --yes

Note there is no "deployment slot swap" here as there is on App Service — the traffic percentage is the swap, and it is strictly better, because you can sit at 10% for an hour.

Soft delete and purge protection are your accident insurance, and they cut both ways:

• Workspace soft delete means a terraform destroy in dev followed by an apply fails on a reserved name unless purge_soft_deleted_workspace_on_destroy = true. That flag is right for dev and wrong for prod — in prod you want the recycle bin.
• Key Vault purge protection is a one-way door. Once true, it cannot be set back to false, and a deleted vault cannot be purged before its retention expires. Set it in prod deliberately, knowing you have permanently reserved that vault name.
• Storage blob versioning and soft delete are what save you when a job overwrites the wrong prefix.

Resource locks for the resources whose loss is unrecoverable:

az lock create --name protect-prod-workspace --lock-type CanNotDelete \
  --resource-group rg-aml-fraud-prod \
  --resource-name mlw-fraud-prod \
  --resource-type Microsoft.MachineLearningServices/workspaces

az lock create --name protect-prod-storage --lock-type CanNotDelete \
  --resource-group rg-aml-fraud-prod \
  --resource-name stlamlfraudprod \
  --resource-type Microsoft.Storage/storageAccounts

Locks are inherited by child resources and will make terraform destroy fail, which is the point. The cost: your pipeline needs a documented, approved procedure to remove one, and CanNotDelete still allows data-plane writes — a lock on a storage account does not stop someone deleting a blob.

Drift

Three kinds, and only one of them is Terraform's problem.

1. Infrastructure drift. Someone resized a cluster in the portal. Detect it on a schedule:

  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init -backend-config=backends/prod.hcl
      - id: plan
        run: terraform plan -var-file=env/prod.tfvars -detailed-exitcode
        continue-on-error: true
      - if: steps.plan.outputs.exitcode == '2'
        run: echo "::error::Infrastructure drift detected in prod"

-detailed-exitcode returns 0 for no changes, 1 for error, 2 for drift. That third value is what makes this a monitor rather than a report nobody reads.

Expect and accept some drift. Compute instances created by data scientists, deployments created by the release pipeline, and the traffic split are all supposed to exist outside Terraform. That is why traffic is in ignore_changes. Drift detection that screams about legitimate activity gets muted within a week, and then it isn't detection.

2. Asset drift — the deployment in prod points at fraud-rf:11 and the repo says :12. Terraform cannot see this. Check it explicitly:

az ml online-deployment show --endpoint-name ep-fraud-prod --name blue --query model -o tsv

3. Data drift — the input distribution has moved away from the training distribution. Not an IaC problem at all, and the one most likely to actually hurt you. Covered in Production.

Teardown

terraform destroy -var-file=env/dev.tfvars

What terraform destroy will not remove:

• Anything created outside Terraform — compute instances, online deployments, data assets, models, environments. The workspace destroy will fail or hang while children exist; delete the deployments first with az ml online-deployment delete.
• The soft-deleted workspace, unless purge_soft_deleted_workspace_on_destroy = true.
• The soft-deleted Key Vault, and if purge protection is on, nothing can remove it early.
• Anything under a resource lock — destroy fails; remove the lock first.
• The remote state backend — deliberately, since it holds the state of everything else.
• Diagnostic data already ingested into Log Analytics, and its retention charges.
• Registry-held models, if you used a shared registry. That is a separate resource in a separate resource group, which is exactly why it survives environment teardown.

Belt-and-braces, for a dev environment you truly want gone:

az ml online-deployment list --endpoint-name ep-fraud-dev -o tsv --query "[].name" \
  | xargs -I{} az ml online-deployment delete --endpoint-name ep-fraud-dev --name {} --yes
terraform destroy -var-file=env/dev.tfvars
az ml workspace list-deleted -o table    # confirm nothing lingers
az group delete -n rg-aml-fraud-dev --yes

Next: Integrations →

← Back to the Azure Machine Learning overview · ← Previous: Getting Started