8. Interview Questions

It is Azure's platform observability service — a family covering metrics, logs, traces, alerts, and autoscale under one name. The problem it solves is that without it you would build and run an entire observability stack yourself, and you still could not see what the platform sees: control-plane operations, resource health, and per-service diagnostic streams that only Azure can produce.

The framing that earns credit: it hands you the platform's own telemetry, in exchange for accepting its data model, its query language, and its per-gigabyte bill.

Two separate stores, not two views of one.

Metrics is a numeric time-series database. Platform metrics are collected automatically for every resource, free, with low latency, fixed retention, and limited dimensionality. Queried by choosing a metric, an aggregation, and a time grain.

Logs is a schema-on-write columnar store inside a Log Analytics workspace you create and pay for. Arbitrary shape, arbitrary retention, full KQL including joins, higher ingestion latency, metered per gigabyte.

The practical rule: if the question is "what was the number", it is a metric; if it is "what happened, to whom, from where", it is a log. Alerting on metrics is faster and cheaper; investigating requires logs.

The Log Analytics workspace is a regional resource in a resource group in a subscription — the ordinary hierarchy. Two things sit off to the side and are worth naming:

• Diagnostic settings are extension resources. Their ARM ID is the monitored resource's ID plus the diagnostic-setting path, so permission to create one is held against the monitored resource, not the workspace.
• Metrics and the activity log are not resources at all. They are platform features of the subscription and its resources; you never create a metrics store.

Above the subscription, management groups matter because that is where the Azure Policy assignments that enforce diagnostic settings should live.

Sentinel is not an alternative — it is a SIEM built on a Log Analytics workspace. Same store, same KQL, additional analytics and a second meter. Choose to enable it when the question is security detection and response; the operational monitoring keeps working either way. The real decision is whether security and operations share one workspace: sharing avoids ingesting the same logs twice, separating simplifies RBAC and chargeback.

A third-party tool (Datadog, Splunk, Elastic, Grafana Cloud) is the right call when most of the estate is outside Azure. In that case keep Azure Monitor as the collector and export via a diagnostic setting to an Event Hub — that is the supported door out.

Billed primarily for data ingested, per GB, priced by table plan (Analytics > Basic > Auxiliary); then retention beyond the included period; then alert rules (log search alerts per rule and by evaluation frequency, metric alerts per monitored time series); then queries on Basic/Auxiliary tables per GB scanned; then search jobs, restores, web tests, and Prometheus samples.

Billing when idle: retained data (every GB every month until it ages out), alert rules (evaluating on schedule whether or not they fire), and web tests (pinging on schedule). A decommissioned application whose workspace still holds two years of logs is a bill for nothing.

1. Key Vault emits the audit event internally.
2. Azure checks whether a diagnostic setting exists on the vault with the AuditEvent category enabled. If not, the event is discarded permanently — this is the step candidates most often miss.
3. The event is fanned out to each destination on each diagnostic setting; each copy is billed separately.
4. For a workspace destination the row is mapped either into the legacy wide AzureDiagnostics table or, if the setting specifies resource-specific mode, into AZKVAuditLogs. Where a DCR is involved, its transformation runs here and dropped rows are never billed and never recoverable.
5. Ingestion commits. TimeGenerated records when the event happened; ingestion_time() records when it became visible. The gap is ingestion latency, measured in minutes.
6. Anything reading it — portal, workbook, alert rule, Grafana — is a client of the same query API.
7. A log search alert evaluates on its own schedule, so time-to-notice is ingestion latency plus up to one evaluation interval plus action-group delivery.

Control plane is ARM at management.azure.com: creating the workspace, changing retention or pricing tier, creating diagnostic settings, DCRs, alert rules, action groups. Data plane is the query endpoints (api.loganalytics.azure.com, api.applicationinsights.io) and the ingestion endpoint (a Data Collection Endpoint).

Roles: Monitoring Reader reads both sides read-only. Monitoring Contributor is control-plane — it can create alert rules and diagnostic settings but does not by itself grant broad data read. Log Analytics Reader grants querying. Log Analytics Contributor grants both, plus the ability to reconfigure collection. Monitoring Metrics Publisher is the narrow data-plane write role.

Three classic mistakes:

• Assuming subscription Owner means you can read a workspace's data the way a colleague does — the workspace access mode decides that.
• Granting Log Analytics Contributor when Reader was meant, handing over the ability to change what is collected and inflate the bill.
• Granting Monitoring Contributor to a workload that only needs to publish a custom metric, where Monitoring Metrics Publisher is the correct, far narrower role.

Three plans trading ingestion price against capability: Analytics (full KQL, alerting, workbooks, highest ingestion price), Basic (much cheaper ingestion, restricted single-table queries, queries billed per GB scanned, no log search alert rules), Auxiliary (cheapest ingest, most restricted access, aimed at high-volume compliance logs reached mostly by search jobs).

What breaks on the move to Basic: any log search alert rule targeting that table stops working, and it does so silently from Terraform's point of view — the plan applies cleanly. Queries using joins or multi-table unions against it start failing. Retention behaviour and query cost change.

The correct process: grep every alert rule and workbook for the table name, confirm nothing depends on it interactively, then re-plan and set long-term retention for the compliance case. ⚠️ Which tables support which plan changes over time — verify against current docs.

Set the workspace access mode to use resource or workspace permissions — resource-context access. A user with Reader on their own Azure resources can then query those resources' rows in the shared workspace without any workspace-level permission, because rows carry a _ResourceId. Grant no workspace role to the application teams at all.

Caveats worth raising unprompted: rows without a _ResourceId — some custom tables, some agent data — are invisible in resource context, which surprises people. For tables that need to be tighter than resource context allows (SigninLogs, anything with customer data), add table-level RBAC with a custom role on …/workspaces/query/<Table>/read.

No keys: set local_authentication_disabled = true on the workspace so shared keys cannot be used, and have every workload authenticate with a managed identity — Monitoring Metrics Publisher on the DCR for ingestion, Monitoring Reader for readers like Grafana.

Terraform (azurerm, with azapi for features the provider has not caught up with) owns the workspace, table plans, retention, DCRs and associations, action groups, alert rules, and Application Insights. Remote state in an Azure Storage backend with native blob-lease locking — no separate lock table, unlike AWS — and use_azuread_auth = true. Directory-per-environment rather than Terraform workspaces, because the blast radius of selecting the wrong workspace is your entire alerting posture. CI/CD with workload identity federation / OIDC, plan on PR, apply on merge behind an environment approval gate, applying the saved plan artifact rather than re-planning.

Why Terraform alone is insufficient: a diagnostic setting must exist on every resource anyone ever creates, including resources created outside your Terraform. That gap is closed by an Azure Policy DeployIfNotExists assignment at management-group scope, with a managed identity granted Contributor on the targets and Log Analytics Contributor on the workspace. Note that the policy only affects resources created after assignment; existing ones need an explicit remediation task, which can generate a great deal of ingestion at once.

Ansible fills the day-2 gap: agent installation on Arc-enabled or non-Azure machines, DCR associations at fleet scale, and creating time-bounded alert processing rules for maintenance windows.

The ones that matter:

• Changing a workspace's location — replacement, and all data is lost. There is no move operation for a workspace's contents.
• Renaming a workspace — replacement, same consequence.
• Changing a DCR's region or kind — replacement, and every association must be recreated.
• Repointing an Application Insights component at a different workspace — historical telemetry does not follow; check the plan carefully.

Always read the plan for the -/+ destroy and then create replacement marker. And name the soft-delete interaction: a deleted workspace enters soft delete, its name stays reserved, and a re-apply that tries to recreate it with the same name fails with a conflict that does not obviously say why. That is the number-one cause of confusing re-apply failures in monitoring modules.

Diagnose, in order. Start in the workspace, not in Cost Management — Cost Management tells you which subscription, the workspace tells you which log:

Usage
| where TimeGenerated > ago(60d)
| where IsBillable == true
| summarize GB = sum(Quantity) / 1000 by DataType, bin(TimeGenerated, 1d)
| render timechart

That gives the table and the date the step change happened. Then attribute it to a resource using _BilledSize and _ResourceId on the offending table. Cross-check the activity log for a diagnostic setting created around that date, and check whether anyone enabled Container Insights or turned on allLogs on something busy.

The usual causes, in order of likelihood: AKS Container Insights collecting stdout from every namespace; Application Gateway or Front Door access logs; kube-audit; NSG flow logs; a debug flag left on after a deployment; a DeployIfNotExists policy remediation task that just onboarded a few hundred resources.

Fix, in order of payoff: stop collecting what nobody queries; move verbose tables to the Basic plan (after checking no alert rule targets them); add a DCR transformation to drop rows and columns at ingestion; buy a commitment tier once volume is stable; tune Application Insights sampling; shorten interactive retention while lengthening long-term retention.

What not to do: set a daily cap as the fix. It stops collection when hit, blinding you during exactly the incident you would want data for. Use it as a circuit breaker with an alert, not as cost control.

Residency first, because it is the constraint with no workaround. A workspace is regional and its data cannot be moved. So: one workspace per region, and cross-region investigation via union workspace("…").Table, accepting that the querier needs permission on each and is billed against each.

Seven-year security retention is not a workspace retention setting — that is expensive and not the right tool. Route the security-relevant categories to two destinations on the same diagnostic setting: the workspace for the interactive window (say 90 days, on the Analytics plan so alerting works), and an immutable storage account for the seven years. Where the regime insists the data remain queryable, use long-term retention with search jobs as the access path, and rehearse a search job so you know what it costs and how long it takes.

Tenant-level logs — Entra ID sign-in and audit logs — need a diagnostic setting on the tenant, not on a resource. They are the first thing an investigation asks for and the thing most estates are missing.

Team isolation: one workspace per region shared across teams, with access mode set to resource-context so teams see their own resources' rows without any workspace role. Table-level RBAC on SigninLogs and anything holding customer data so only the security team reads it. If Sentinel is in play, decide deliberately whether it shares the workspace — sharing avoids double ingestion, separating simplifies the RBAC story you just built.

Enforcement: a management group per environment, with DeployIfNotExists assigned at MG scope pointing at the regionally correct workspace (parameterised by region), plus Deny policies for disallowed regions and for workspaces with public network access enabled. Private connectivity via AMPLS, rolled out in Open mode first and closed only after verifying the Heartbeat table.

What I would flag as the risk: cross-region queries are the operational cost of this design. Investigations get slower and workbooks get more complex. That is the price of residency, and it should be stated to the business rather than discovered by the on-call engineer.

Blast radius first, ranked. A missing alert rule affects one signal. A missing or changed action group breaks every rule pointing at it — that is the highest-leverage single object. A missing diagnostic setting is worse than either, because it is silently destroying data. A deleted workspace is catastrophic and also blocks recreation by name.

Rollback. Re-apply the previous commit. Configuration is fully reversible: alert rules, action groups, and diagnostic settings all come back exactly. State the asymmetry explicitly, because it is the insight the question is testing: configuration is reversible; data is not. Logs not collected during the gap are gone permanently, and that includes the evidence of what happened during the window in which monitoring was broken.

The Azure-specific complications to name:

• Complete deployment mode. If the failed deployment was an ARM/Bicep deploy run with --mode Complete, it deleted everything in the resource group that was not in the template — which in a monitoring resource group means alert rules and action groups created by Policy remediation, by other teams, or by hand during a previous incident. Incremental is the default for a reason. Recovery is re-applying every source that contributed to that resource group, and you may not know what they all were.
• Resource locks. A CanNotDelete lock will make an apply fail with an error that reads like a permissions problem. Check for locks before debugging RBAC.
• Soft delete. If the workspace itself was deleted, recover it within the soft-delete window rather than recreating — recreating by the same name will fail while the soft-deleted one holds it.

Afterwards: an activity-log alert on write and delete operations against Microsoft.Insights/* and Microsoft.OperationalInsights/*, excluding the pipeline identity, so the next occurrence is noticed in minutes rather than during the next incident.

Find out — four detectors, and a mature answer uses more than one:

1. A scheduled terraform plan -detailed-exitcode in CI against prod, opening an issue on exit code 2. This is the backbone.
2. az deployment group what-if for anything Bicep-managed.
3. Azure Policy compliance state — the only one that catches resources that were never in your Terraform at all.
4. The activity log, which names the human:

AzureActivity
| where TimeGenerated > ago(7d)
| where ResourceProvider in ("MICROSOFT.INSIGHTS", "MICROSOFT.OPERATIONALINSIGHTS")
| where OperationNameValue endswith "/write" or OperationNameValue endswith "/delete"
| where ActivityStatusValue == "Success"
| where Caller !contains "terraform-sp"
| project TimeGenerated, Caller, OperationNameValue, _ResourceId

Get back to clean — the judgement, not the mechanics, is what is being assessed. Decide whether the change was right. At 3 a.m. under incident pressure, it very often was: the old threshold was generating noise and the new one is correct. If so, codify it — copy the new value into the module and merge, so the human's knowledge is preserved rather than reverted. If it was not right, terraform apply reverts it, and the conversation is with the person, not the tool.

If the resource was created entirely outside Terraform, use terraform import or an import block to bring it under management. Do not leave a known-drifted resource unmanaged "for now" — that is how a module stops being trusted, and an untrusted module is one nobody runs.