5. Deployment

Getting Started proved the pipes connect. This page makes monitoring repeatable, reviewable, and reversible — a parameterised Terraform module, an Ansible playbook for the day-2 work Terraform is bad at, a Bicep equivalent, a CI/CD pipeline authenticated without secrets, an environment strategy, and an honest account of what rollback means for a service whose data cannot be un-deleted.

One framing before the code. Most services are deployed and then monitored. Azure Monitor is different: it is deployed as part of every other deployment. A diagnostic setting is not a thing you provision once — it is a thing that must exist on every resource anyone ever creates. That makes Azure Policy the most important tool on this page, and it is the section teams retrofit most painfully. Read the Policy section even if you skip the rest.

[Image Prompt: 2D minimalistic pipeline diagram of an Azure Monitor deployment flowing from a git commit through terraform plan, a manual approval gate, and terraform apply into dev, staging, and prod subscriptions, with an Azure Policy DeployIfNotExists assignment shown attaching diagnostic settings to newly created resources on the right, flat design, clean vector art style, white background]

Tool order

The same everywhere in this article: Terraform primary, Ansible secondary, Bicep/ARM third. All three apply to Azure Monitor, and the division of labour is unusually clean here:

That last row is not a formality. If you enforce diagnostic settings with Azure Policy — and you should — the embedded remediation template inside the policy definition is ARM JSON. You will write ARM whether or not you use Bicep.

The Terraform module

Shape: main.tf, variables.tf, outputs.tf, one .tfvars per environment. The module owns a workspace, its cost controls, a reusable action group, a baseline alert rule set, and a reusable diagnostic-setting pattern.

variables.tf

variable "environment" {
  description = "dev | staging | prod"
  type        = string
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging, or prod."
  }
}

variable "name_prefix" {
  description = "Short workload identifier, lowercase, no hyphens"
  type        = string
}

variable "location" {
  type    = string
  default = "uksouth"
}

variable "retention_in_days" {
  description = "Interactive retention for the workspace default"
  type        = number
  default     = 30
}

variable "daily_quota_gb" {
  description = "Daily ingestion cap in GB. -1 means unlimited. A cap STOPS COLLECTION when hit."
  type        = number
  default     = -1
}

variable "commitment_tier_gb" {
  description = "Daily GB commitment. null = pay-as-you-go (PerGB2018)."
  type        = number
  default     = null
}

variable "alert_email" {
  description = "Where severity 1-2 alerts go until a real ITSM connector exists"
  type        = string
}

variable "monitored_resource_ids" {
  description = "Resources that should get a diagnostic setting from this module"
  type        = map(string)
  default     = {}
}

variable "tags" {
  type    = map(string)
  default = {}
}

main.tf

terraform {
  required_version = ">= 1.6"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
    azapi = {
      source  = "Azure/azapi"
      version = "~> 2.0"
    }
  }
}

provider "azurerm" {
  features {
    # Azure Monitor itself has few feature-block knobs, but the block is mandatory.
    # If this module also manages Key Vault or Log Analytics *solutions*, set the
    # purge behaviour here explicitly rather than inheriting a default you did not read.
    log_analytics_workspace {
      permanently_delete_on_destroy = false # keep soft delete; see the rollback section
    }
  }
}

locals {
  suffix = "${var.name_prefix}-${var.environment}"

  tags = merge(var.tags, {
    Environment = var.environment
    ManagedBy   = "terraform"
    Component   = "observability"
  })

  # Environment drives retention and cost posture, not a human's mood on the day
  retention = {
    dev     = 30
    staging = 30
    prod    = 90
  }
}

resource "azurerm_resource_group" "this" {
  name     = "rg-monitor-${local.suffix}"
  location = var.location
  tags     = local.tags
}

resource "azurerm_log_analytics_workspace" "this" {
  name                = "log-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location

  # PerGB2018 is pay-as-you-go. A commitment tier is expressed by reservation_capacity_in_gb_per_day.
  sku               = var.commitment_tier_gb == null ? "PerGB2018" : "CapacityReservation"
  retention_in_days = local.retention[var.environment]

  reservation_capacity_in_gb_per_day = var.commitment_tier_gb

  daily_quota_gb = var.daily_quota_gb

  # Lock the data plane down; see the Production page
  internet_ingestion_enabled = true  # false requires AMPLS for all senders
  internet_query_enabled     = true  # false requires AMPLS for all readers

  local_authentication_disabled = true # force Entra auth, no workspace shared keys

  tags = local.tags
}

⚠️ SKU names, the exact argument for commitment tiers, and which arguments the provider exposes have all changed across azurerm major versions — verify against the provider docs for the version you pin.

Table plans and long-term retention

The single biggest cost lever, and the part most modules omit. Per-table settings are managed separately from the workspace:

# Expensive, verbose, rarely queried interactively -> Basic plan, long retention
resource "azurerm_log_analytics_workspace_table" "container_logs" {
  workspace_id        = azurerm_log_analytics_workspace.this.id
  name                = "ContainerLogV2"
  plan                = "Basic"
  total_retention_in_days = 365 # long-term retention beyond the interactive window
}

# Security-relevant, must support alerting -> stays on Analytics
resource "azurerm_log_analytics_workspace_table" "signin" {
  workspace_id            = azurerm_log_analytics_workspace.this.id
  name                    = "SigninLogs"
  plan                    = "Analytics"
  retention_in_days       = 90
  total_retention_in_days = 730
}

Footgun. Setting a table to Basic disables log search alert rules on it, immediately and silently from Terraform's point of view. If an alert rule elsewhere in your estate queries that table, your plan will succeed and your alerting will stop. Grep for the table name across your alert rules before re-planning it.

Diagnostic settings, applied over a set

The reusable pattern — one for_each over whatever this module is asked to monitor:

resource "azurerm_monitor_diagnostic_setting" "this" {
  for_each = var.monitored_resource_ids

  name                           = "diag-to-law"
  target_resource_id             = each.value
  log_analytics_workspace_id     = azurerm_log_analytics_workspace.this.id
  log_analytics_destination_type = "Dedicated"

  # Enumerate categories per resource type; `allLogs` is convenient and grows without warning
  dynamic "enabled_log" {
    for_each = data.azurerm_monitor_diagnostic_categories.this[each.key].log_category_types
    content {
      category = enabled_log.value
    }
  }

  metric {
    category = "AllMetrics"
  }

  lifecycle {
    # The API returns categories in a different order and adds new ones over time,
    # which produces permanent diffs if you are not deliberate about it.
    ignore_changes = [metric]
  }
}

data "azurerm_monitor_diagnostic_categories" "this" {
  for_each    = var.monitored_resource_ids
  resource_id = each.value
}

Two honest warnings on this resource, both of which cost people afternoons:

• azurerm_monitor_diagnostic_setting is one of the most diff-noisy resources in the provider. The API normalises category lists and adds new categories over time, so a plan that was clean last week shows changes this week. ignore_changes on the metric block is the usual pragmatic fix; the alternative is enumerating categories explicitly and accepting the maintenance.
• Its scope is the monitored resource. The service principal running Terraform needs write access there, not just on the workspace. In a hub-and-spoke setup where the workspace is central and the resources are not, this is the permission that will fail first.

Data Collection Rules

For anything with an agent — VMs, VM Scale Sets, Arc-enabled servers — the DCR is the deployable artifact, and the ingestion-time transformation is where cost control lives:

resource "azurerm_monitor_data_collection_rule" "vm_baseline" {
  name                = "dcr-vm-baseline-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  tags                = local.tags

  destinations {
    log_analytics {
      workspace_resource_id = azurerm_log_analytics_workspace.this.id
      name                  = "law-destination"
    }
  }

  data_sources {
    performance_counter {
      name                          = "perf-basic"
      streams                       = ["Microsoft-Perf"]
      sampling_frequency_in_seconds = 60
      counter_specifiers = [
        "\\Processor(_Total)\\% Processor Time",
        "\\Memory\\Available Bytes",
        "\\LogicalDisk(_Total)\\% Free Space",
      ]
    }

    syslog {
      name           = "syslog-warn-and-above"
      streams        = ["Microsoft-Syslog"]
      facility_names = ["auth", "authpriv", "cron", "daemon", "kern", "syslog"]
      log_levels     = ["Warning", "Error", "Critical", "Alert", "Emergency"]
    }
  }

  data_flow {
    streams      = ["Microsoft-Perf"]
    destinations = ["law-destination"]
  }

  data_flow {
    streams      = ["Microsoft-Syslog"]
    destinations = ["law-destination"]
    # Ingestion-time transformation: drop the noise before you pay for it
    transform_kql = "source | where SyslogMessage !contains 'CRON' | project-away ProcessID"
  }
}

resource "azurerm_monitor_data_collection_rule_association" "vm" {
  for_each                = var.vm_resource_ids
  name                    = "dcra-baseline"
  target_resource_id      = each.value
  data_collection_rule_id = azurerm_monitor_data_collection_rule.vm_baseline.id
}

Note log_levels on the Syslog source: collecting Info and below from a fleet of Linux machines is one of the most reliable ways to produce a five-figure monthly bill from nothing.

Action groups and alert rules

Build action groups by severity, not by team-member. People leave; severities do not.

resource "azurerm_monitor_action_group" "critical" {
  name                = "ag-critical-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  short_name          = "crit"
  tags                = local.tags

  email_receiver {
    name          = "oncall"
    email_address = var.alert_email
  }

  # In prod, replace the email with a real integration
  dynamic "webhook_receiver" {
    for_each = var.environment == "prod" ? [1] : []
    content {
      name                    = "itsm"
      service_uri             = var.itsm_webhook_uri
      use_common_alert_schema = true
    }
  }
}

use_common_alert_schema = true matters more than it looks: without it, different alert types deliver structurally different payloads to your webhook, and your handler has to branch on all of them. Turn it on everywhere.

A metric alert, a log search alert, and the activity-log alert everyone eventually wishes they had:

# 1. Metric alert - fast, cheap, near-real-time
resource "azurerm_monitor_metric_alert" "vm_cpu" {
  name                = "alert-vm-cpu-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  scopes              = values(var.vm_resource_ids) # multi-resource scope: one rule, many targets
  description         = "Sustained high CPU"
  severity            = 2
  frequency           = "PT1M"
  window_size         = "PT5M"

  criteria {
    metric_namespace = "Microsoft.Compute/virtualMachines"
    metric_name      = "Percentage CPU"
    aggregation      = "Maximum" # not Average - a 5-minute average hides the spike you care about
    operator         = "GreaterThan"
    threshold        = 90
  }

  action {
    action_group_id = azurerm_monitor_action_group.critical.id
  }
}

# 2. Log search alert - flexible, slower, priced per rule
resource "azurerm_monitor_scheduled_query_rules_alert_v2" "kv_failures" {
  name                = "alert-kv-auth-failures-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location
  scopes              = [azurerm_log_analytics_workspace.this.id]
  severity            = 2
  evaluation_frequency = "PT5M"
  window_duration      = "PT15M"

  criteria {
    query                   = <<-KQL
      AZKVAuditLogs
      | where ResultSignature != "OK"
      | summarize failures = count() by CallerIPAddress, _ResourceId
    KQL
    time_aggregation_method = "Count"
    threshold               = 10
    operator                = "GreaterThan"

    failing_periods {
      minimum_failing_periods_to_trigger_alert = 1
      number_of_evaluation_periods             = 1
    }
  }

  auto_mitigation_enabled = true

  action {
    action_groups = [azurerm_monitor_action_group.critical.id]
  }
}

# 3. Activity log alert - "who deleted the thing"
resource "azurerm_monitor_activity_log_alert" "resource_deleted" {
  name                = "alert-prod-delete-${local.suffix}"
  resource_group_name = azurerm_resource_group.this.name
  location            = "global"
  scopes              = [data.azurerm_subscription.current.id]
  description         = "A delete operation succeeded somewhere in this subscription"

  criteria {
    category       = "Administrative"
    operation_name = "Microsoft.Resources/subscriptions/resourceGroups/delete"
    status         = "Succeeded"
  }

  action {
    action_group_id = azurerm_monitor_action_group.critical.id
  }
}

data "azurerm_subscription" "current" {}

The Azure Policy piece — the one that actually scales

Deploying diagnostic settings resource-by-resource in Terraform works until someone creates a resource outside Terraform. Policy closes the gap: a DeployIfNotExists assignment at management-group or subscription scope attaches the diagnostic setting automatically, forever, to resources nobody told you about.

# Use a built-in policy initiative where one exists for your resource types;
# this shows the assignment mechanics, which are the part people get wrong.
resource "azurerm_subscription_policy_assignment" "diag_to_law" {
  name                 = "diag-settings-to-law"
  subscription_id      = data.azurerm_subscription.current.id
  policy_definition_id = var.diagnostic_policy_definition_id
  location             = var.location
  description          = "Ensure every supported resource ships logs to the central workspace"

  # DeployIfNotExists REQUIRES a managed identity with rights to do the deploying
  identity {
    type = "SystemAssigned"
  }

  parameters = jsonencode({
    logAnalytics = { value = azurerm_log_analytics_workspace.this.id }
  })
}

# The identity needs permission on the targets AND on the workspace
resource "azurerm_role_assignment" "policy_identity_contributor" {
  scope                = data.azurerm_subscription.current.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_subscription_policy_assignment.diag_to_law.identity[0].principal_id
}

resource "azurerm_role_assignment" "policy_identity_law" {
  scope                = azurerm_log_analytics_workspace.this.id
  role_definition_name = "Log Analytics Contributor"
  principal_id         = azurerm_subscription_policy_assignment.diag_to_law.identity[0].principal_id
}

Three things about this that are not obvious:

• DeployIfNotExists and Modify policies need a managed identity, and Terraform must grant it roles itself. A policy assignment whose identity lacks permission fails silently at remediation time and shows as non-compliant with no useful error.
• Assignment only affects resources created after it. Existing non-compliant resources need a remediation task (az policy remediation create), which you should run explicitly and deliberately — it can generate an enormous amount of new ingestion in one go.
• Policy is also the enforcement point for your environment strategy — see below.

Preview features and azapi

azurerm lags new Azure Monitor features by weeks to months, and Azure Monitor ships a lot. When the resource you need does not exist in the provider yet, drop to azapi rather than clicking in the portal:

# Example shape: an Azure Monitor Workspace (managed Prometheus store), where provider
# coverage of newer sub-features may lag. Verify whether azurerm now covers what you need.
resource "azapi_resource" "prometheus" {
  type      = "Microsoft.Monitor/accounts@2023-04-03"
  name      = "amw-${local.suffix}"
  parent_id = azurerm_resource_group.this.id
  location  = var.location
  tags      = local.tags

  body = {
    properties = {}
  }
}

⚠️ Preview features have no SLA and their API shapes change. Label them as preview in your own module docs, and pin the API version explicitly as above.

outputs.tf

output "workspace_id" {
  description = "ARM resource ID - what diagnostic settings and policy assignments consume"
  value       = azurerm_log_analytics_workspace.this.id
}

output "workspace_guid" {
  description = "Customer ID (GUID) - what the query API and agents consume"
  value       = azurerm_log_analytics_workspace.this.workspace_id
}

output "critical_action_group_id" {
  value = azurerm_monitor_action_group.critical.id
}

Exporting both IDs is deliberate. The ARM resource ID and the workspace GUID are different things, they are needed in different places, and mixing them up is the single most common wiring error in Azure Monitor automation.

Remote state and locking

Local state is a footgun the moment a second person or a pipeline touches this. Azure's backend is comfortable:

terraform {
  backend "azurerm" {
    resource_group_name  = "rg-tfstate"
    storage_account_name = "sttfstateprod01"
    container_name       = "tfstate"
    key                  = "azure-monitor/prod.tfstate"
    use_azuread_auth     = true # Entra auth, not the storage account key
  }
}

Two Azure-specific points worth stating plainly:

• Locking is native. The azurerm backend takes a blob lease on the state file. There is no separate lock table to provision — a real simplification compared with the DynamoDB table AWS requires. A crashed apply leaves the lease held; terraform force-unlock releases it, and you should confirm nothing is actually running before you do.
• use_azuread_auth = true so the pipeline's federated identity is used rather than a storage account key. Pair it with shared_access_key_enabled = false on the state storage account.

Workspaces vs. directory-per-environment. Terraform workspaces keep one codebase and one backend key prefix with separate state per workspace — light, and a single terraform workspace select typo away from applying dev config to prod. Directory-per-environment (envs/dev, envs/staging, envs/prod, each with its own backend block and .tfvars) is more files and far more obvious in a pull request. For Azure Monitor, take directory-per-environment: the blast radius of a mistake here is your entire alerting posture, and obviousness beats elegance.

Ansible — the day-2 layer

Ansible is genuinely useful here, and not as a Terraform substitute. Its jobs: agent installation on machines Terraform does not own, DCR associations at fleet scale, and imperative operational tasks.

# monitor-agent.yml - install AMA and associate the baseline DCR
- name: Onboard servers to Azure Monitor
  hosts: localhost
  connection: local
  gather_facts: false
  vars:
    resource_group: "rg-monitor-{{ name_prefix }}-{{ environment }}"
    dcr_id: "{{ lookup('env', 'DCR_RESOURCE_ID') }}"
  tasks:

    - name: Ensure the Azure Monitor Agent extension is present on each VM
      azure.azcollection.azure_rm_virtualmachineextension:
        resource_group: "{{ resource_group }}"
        virtual_machine_name: "{{ item }}"
        name: AzureMonitorLinuxAgent
        publisher: Microsoft.Azure.Monitor
        virtual_machine_extension_type: AzureMonitorLinuxAgent
        type_handler_version: "1.0"
        auto_upgrade_minor_version: true
        state: present
      loop: "{{ vm_names }}"
      register: ama

    - name: Associate the baseline data collection rule
      azure.azcollection.azure_rm_resource:
        api_version: "2022-06-01"
        resource_group: "{{ resource_group }}"
        provider: Insights
        resource_type: dataCollectionRuleAssociations
        resource_name: dcra-baseline
        subresource:
          - type: providers/Microsoft.Compute/virtualMachines
            name: "{{ item }}"
        body:
          properties:
            dataCollectionRuleId: "{{ dcr_id }}"
        state: present
      loop: "{{ vm_names }}"

Idempotency, demonstrated. Run it twice:

# First run
changed=12  ok=12

# Second run, no configuration changed
changed=0   ok=12

If the second run reports changes, something in the playbook is not declarative — the usual culprit is a command/shell task without a creates or changed_when guard.

A second, honestly-imperative use: suppressing alerts during a planned maintenance window. Prefer an alert processing rule over disabling rules, because a disabled rule that nobody re-enabled is a classic post-incident finding:

    - name: Suppress alerts on the app resource group during maintenance
      azure.azcollection.azure_rm_resource:
        api_version: "2021-08-08"
        resource_group: "{{ resource_group }}"
        provider: AlertsManagement
        resource_type: actionRules
        resource_name: "maint-{{ change_ticket }}"
        body:
          location: Global
          properties:
            scopes: ["{{ app_resource_group_id }}"]
            enabled: true
            actions:
              - actionType: RemoveAllActionGroups
            schedule:
              effectiveFrom: "{{ window_start }}"
              effectiveUntil: "{{ window_end }}"
        state: present

The schedule block is the important part: the suppression expires on its own. ⚠️ Verify current API versions for the AlertsManagement provider.

Where Ansible is the wrong tool here: creating the workspace, managing table plans, or owning alert rule definitions. It will do all three and will never tell you when someone changed them by hand.

Bicep / ARM equivalent

targetScope = 'resourceGroup'

@description('dev | staging | prod')
@allowed(['dev', 'staging', 'prod'])
param environment string

param namePrefix string
param location string = resourceGroup().location
param alertEmail string
param monitoredResourceId string

var suffix = '${namePrefix}-${environment}'
var retention = environment == 'prod' ? 90 : 30

resource law 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
  name: 'log-${suffix}'
  location: location
  properties: {
    sku: { name: 'PerGB2018' }
    retentionInDays: retention
    features: {
      disableLocalAuth: true
    }
    publicNetworkAccessForIngestion: 'Enabled'
    publicNetworkAccessForQuery: 'Enabled'
  }
}

resource ag 'Microsoft.Insights/actionGroups@2023-01-01' = {
  name: 'ag-critical-${suffix}'
  location: 'Global'
  properties: {
    groupShortName: 'crit'
    enabled: true
    emailReceivers: [
      {
        name: 'oncall'
        emailAddress: alertEmail
        useCommonAlertSchema: true
      }
    ]
  }
}

// Extension resource: note the `scope` property - this is the Bicep spelling of
// "attach to the monitored resource, not to this resource group"
resource diag 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'diag-to-law'
  scope: kv
  properties: {
    workspaceId: law.id
    logAnalyticsDestinationType: 'Dedicated'
    logs: [
      { categoryGroup: 'audit', enabled: true }
    ]
    metrics: [
      { category: 'AllMetrics', enabled: true }
    ]
  }
}

resource kv 'Microsoft.KeyVault/vaults@2023-07-01' existing = {
  name: last(split(monitoredResourceId, '/'))
}

output workspaceId string = law.id
output workspaceGuid string = law.properties.customerId

az deployment group what-if \
  --resource-group rg-monitor-prod \
  --template-file monitor.bicep \
  --parameters environment=prod namePrefix=shop alertEmail=oncall@example.com \
               monitoredResourceId=$KV_ID

⚠️ Deployment modes — the footgun worth naming every time

az deployment group create defaults to incremental mode: resources in the template are created or updated, and resources already in the resource group that are not in the template are left alone. Complete mode (--mode Complete) will delete anything in the resource group that is not in the template.

This is bad everywhere and especially bad in a monitoring resource group, because alert rules, action groups, and saved searches accumulate there from many sources — Policy remediation, portal clicks during incidents, other teams' templates. A complete-mode deploy of your "monitoring baseline" template can silently delete every alert rule you did not think to include. Always what-if first, and treat complete mode as an explicit, reviewed decision.

CI/CD

GitHub Actions, authenticating with workload identity federation (OIDC) against a Microsoft Entra app registration. No client secrets, no publish profiles, nothing long-lived in a repository.

name: azure-monitor

on:
  pull_request:
    paths: ['infra/monitor/**']
  push:
    branches: [main]
    paths: ['infra/monitor/**']

permissions:
  id-token: write      # required for OIDC - the pipeline fails cryptically without it
  contents: read
  pull-requests: write

jobs:
  plan:
    runs-on: ubuntu-latest
    environment: ${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
    defaults:
      run:
        working-directory: infra/monitor/envs/${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
    steps:
      - uses: actions/checkout@v4

      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}          # app registration, federated credential
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}

      - uses: hashicorp/setup-terraform@v3

      - run: terraform init
      - run: terraform validate
      - run: terraform plan -out=tfplan -var-file=terraform.tfvars

      - uses: actions/upload-artifact@v4
        with:
          name: tfplan-${{ github.sha }}
          path: infra/monitor/envs/**/tfplan

  apply:
    needs: plan
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: prod            # <- GitHub environment protection rule = the manual approval gate
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - uses: hashicorp/setup-terraform@v3
      - uses: actions/download-artifact@v4
        with:
          name: tfplan-${{ github.sha }}
      - run: terraform init
      - run: terraform apply -auto-approve tfplan

Setup notes that save an afternoon:

• The federated credential on the Entra app registration must match the exact subject — repo:org/repo:environment:prod for an environment-gated job, repo:org/repo:ref:refs/heads/main for a branch-gated one. A mismatch produces an authentication error that does not name the subject.
• The plan artifact is applied, not re-planned. Re-planning at apply time means you approved something other than what ships.
• The pipeline identity needs unusual breadth for Azure Monitor. Diagnostic settings write at the monitored resource's scope, so a pipeline that only has rights on the monitoring resource group cannot create them. Grant Monitoring Contributor at the scope containing the monitored resources plus Log Analytics Contributor on the workspace — and prefer that pair over subscription Contributor.
• Azure Pipelines equivalent: an Azure Resource Manager service connection using Workload Identity federation, with approvals and checks on the prod environment.

Environments

The recommendation, stated plainly: one subscription per environment, with prod separate. In Azure the subscription is the natural blast-radius, quota, and billing boundary — quotas are counted per-subscription-per-region, and Azure Monitor's ingestion limits are workspace-scoped inside that. This is a stronger argument than the AWS "account per environment" one, because the subscription is also where management groups and Azure Policy attach, which is how you make the difference between environments enforced rather than conventional:

• A management group per environment tier, with the prod MG carrying Deny policies (no public network access on the workspace, mandatory tags, allowed regions for data residency).
• DeployIfNotExists for diagnostic settings assigned at the MG, not per subscription, so a new subscription inherits monitoring on day one.
• An audit policy in dev with the same rules, so drift is visible before it reaches prod.

Data residency deserves a line. The workspace is regional and the data is in that region. Multi-region estates with residency requirements need a workspace per region, and cross-region queries via union workspace("…"). Decide this before you have a year of data in the wrong region; there is no move operation for a workspace's contents.

Rollback and blast radius

Rollback for Azure Monitor is unusual, and the reason is worth internalising: configuration is reversible, data is not. Re-applying the previous commit restores a deleted alert rule perfectly. It does not restore the twelve hours of logs that were not collected while a diagnostic setting was missing. Plan accordingly — when you break monitoring, you also destroy the evidence of the window in which it was broken.

What "undo" means, by change type:

Operations that force replacement rather than update in place. ARM will silently destroy and recreate a resource for some property changes, and for a workspace that means losing all its data:

• Changing a workspace's location — replacement, all data lost.
• Renaming a workspace — replacement.
• Changing a Data Collection Rule's kind or region — replacement, and associations must be re-made.
• Changing an Application Insights component's workspace_id to a different workspace — check the plan output carefully; historical telemetry does not follow.

Always read terraform plan for the -/+ destroy and then create replacement marker. In a monitoring module, treat any replacement of a workspace as a change-controlled event, not a merge.

Soft delete and purge protection. A deleted Log Analytics workspace enters a soft-delete state for a recovery window ⚠️ verify the current window length during which:

• The data is recoverable — az monitor log-analytics workspace recover restores it.
• The name is still taken. A terraform apply that tries to recreate a workspace with the same name fails with a conflict that does not obviously say "it is soft-deleted". This is the exact Key Vault footgun in a different costume, and it is the number-one cause of confusing re-apply failures in monitoring modules.
• Permanent deletion requires an explicit purge, and the azurerm provider's permanently_delete_on_destroy feature flag controls whether terraform destroy does that for you. Leaving it false is the safe default; setting it true in ephemeral test environments is what makes repeated CI runs possible.

Resource locks. A CanNotDelete lock on the monitoring resource group — which is a good idea for prod — will make terraform destroy and some apply operations fail with an error that reads like a permissions problem. Know that this is the cause before you spend an hour on RBAC. Remove the lock deliberately, do the work, put it back; ideally manage the lock itself in a separate state so the pipeline cannot casually remove it.

Blast radius, ranked. Deleting an alert rule affects one signal. Deleting an action group breaks every rule pointing at it — this is the highest-leverage single object on the page. Deleting the workspace breaks everything and destroys the data. Changing a table plan breaks alerts silently. Order your review attention accordingly.

Drift detection

Monitoring configuration drifts more than most, because incidents cause drift — people change alert thresholds at 3 a.m. and nobody reverts them. Four detectors, and you want more than one:

1. Scheduled terraform plan in CI. A nightly job that runs plan -detailed-exitcode against prod and opens an issue on exit code 2. This is the backbone.
2. az deployment group what-if for the Bicep-managed parts.
3. Azure Policy compliance state. The most valuable one here, because it catches resources that were never in your Terraform at all — a new storage account with no diagnostic setting shows as non-compliant, which terraform plan would never notice.
4. The activity log, monitoring itself. An activity log alert on write and delete operations against Microsoft.Insights/* and Microsoft.OperationalInsights/* tells you the moment someone changes an alert rule by hand:

AzureActivity
| where TimeGenerated > ago(7d)
| where ResourceProvider in ("MICROSOFT.INSIGHTS", "MICROSOFT.OPERATIONALINSIGHTS")
| where OperationNameValue endswith "/write" or OperationNameValue endswith "/delete"
| where ActivityStatusValue == "Success"
| where Caller !contains "terraform-sp"   // exclude the pipeline identity
| project TimeGenerated, Caller, OperationNameValue, _ResourceId
| order by TimeGenerated desc

Getting back to a clean plan after someone clicked in the portal: decide whether the manual change was right. If it was, codify it — copy the new threshold into the module and merge. If it was not, terraform apply reverts it. If the resource was created entirely outside Terraform, terraform import (or an import block) brings it under management rather than leaving a permanent divergence. Do not leave a known-drifted resource unmanaged "for now"; that is how the module stops being trusted.

Teardown

terraform destroy -var-file=terraform.tfvars

What destroy will not remove, which for this service is a longer list than usual:

• The soft-deleted workspace. Unless permanently_delete_on_destroy = true, the workspace lingers and holds its name for the recovery window.
• Data already exported elsewhere — anything a diagnostic setting sent to a storage account or an Event Hub is now that resource's problem, and its own bill.
• Diagnostic settings on resources outside this state file. If Policy created them, Policy owns them; destroy will not touch them and they will keep sending to a workspace that no longer exists, producing failed-delivery noise.
• Policy assignments' role assignments if they were created outside this module, and any role assignment made by a human in the portal.
• Resources behind a CanNotDelete lock — destroy fails rather than skipping.
• Long-term retention data for a workspace you recover later; it comes back with the workspace, which is either a relief or a compliance problem depending on why you deleted it.
• Application Insights classic-to-workspace migration artifacts, if any remain in an older estate.

Next: Integrations →

← Back to the Azure Monitor overview · ← Previous: Getting Started