7. Production

Five pillars, as everywhere in this article. Azure Monitor inverts one of them: for most services cost is the fourth thing you think about, and here it is the first, because uncontrolled ingestion is the most common Azure Monitor incident by a wide margin.

[Image Prompt: 2D minimalistic diagram of five production pillars — security, cost, scaling and limits, observability, and reliability — arranged around a central Log Analytics workspace resource, flat design, clean vector art style, white background]

Security

Your logs are among the most sensitive data you hold. They contain IP addresses, user principal names, resource names, query strings, and — whenever someone logs carelessly — tokens and personal data. A workspace is a database of everything that has happened, and it should be treated as one.

Least privilege, concretely. The role you want in each situation:

Situation	Role	Not this
On-call engineer who reads everything	Monitoring Reader at subscription scope	Contributor "so they can fix things"
Application team reading only their own logs	Reader on their resources + workspace access mode set to resource-context	Log Analytics Reader on the shared workspace
Workload publishing custom metrics	Monitoring Metrics Publisher on the target	Monitoring Contributor
Workload pushing to the Logs Ingestion API	Monitoring Metrics Publisher on the DCR	Any workspace role
Pipeline deploying alert rules	Monitoring Contributor at the deployment scope	Subscription Contributor
Pipeline deploying diagnostic settings	Monitoring Contributor on the monitored scope and Log Analytics Contributor on the workspace	Owner
Grafana instance	Monitoring Reader via its managed identity	A service principal with a secret

Log Analytics Contributor is not a reader role. It permits reconfiguring collection, which means it can both read everything and materially increase your bill. Grant it to pipelines, not people.

Table-level RBAC for the sensitive tables. SigninLogs, any table containing customer data, and anything a compliance regime names should be restricted with a custom role granting or denying Microsoft.OperationalInsights/workspaces/query/<Table>/read. This is the only way to say "everyone can see the app logs, nobody but security sees sign-ins" within one workspace — and the alternative, a separate workspace, costs you cross-workspace query complexity.

Turn off key-based auth. A workspace historically supported shared keys for agent ingestion. Set local_authentication_disabled = true (portal: the workspace's Agents settings) so only Microsoft Entra tokens work. If something breaks when you do, you have found an agent nobody knew about, which is itself worth knowing.

Encryption. Data is encrypted at rest with platform-managed keys by default. Customer-managed keys (CMK) are supported for workspaces via Key Vault, at the price of a stricter operational posture — losing the key loses the data, and CMK has feature interactions worth checking ⚠️ verify current CMK limitations against Azure docs. Take CMK when a compliance regime requires it, not reflexively.

Network isolation. Two switches on the workspace, publicNetworkAccessForIngestion and publicNetworkAccessForQuery, plus AMPLS for private connectivity. The sequencing that avoids an outage: create the AMPLS, add the workspace, attach the private endpoint, verify agents are still reporting via the Heartbeat table, and then set access to private-only. Doing it in the other order cuts off every agent at once, and the tool you would use to diagnose that is the thing you just cut off.

The public-exposure gotcha specific to this service: workbooks and dashboards can leak. A shared dashboard renders data with the viewer's permissions, but a workbook's saved query text can itself disclose resource names, architecture, and sometimes credentials pasted into a query during an incident. Review saved searches and workbooks the way you review code.

Alerts are an exfiltration path. An action group can send a webhook containing alert payload data to an arbitrary URL. Anyone who can create an alert rule and an action group can therefore stream selected log data out of your tenant. Restrict who holds Monitoring Contributor accordingly, and consider an activity-log alert on action group creation.

Cost

What you pay for, roughly in order of how much it usually is:

1. Data ingestion, per GB. The dominant line item. Priced by table plan — Analytics is the most expensive, then Basic, then Auxiliary.
2. Retention beyond the included period. Each table has an included interactive retention window ⚠️ verify current included period; beyond it you pay per GB per month, and long-term retention is cheaper still.
3. Application Insights — same ingestion meter, and it is easy to forget it is the same bill.
4. Alert rules. Log search alerts are priced per rule and by evaluation frequency; a one-minute-frequency rule costs several times a fifteen-minute one. Metric alerts are priced per monitored time series, which is why a multi-resource rule over 500 VMs is not free.
5. Queries on Basic and Auxiliary tables, priced per GB scanned — the trade you accepted for cheap ingestion.
6. Search jobs and data restores from long-term retention.
7. Managed Prometheus samples, web tests, and ITSM connector volume. ⚠️ All of this is pricing structure, not prices — verify current rates.

The biggest cost trap is a single verbose diagnostic category enabled with allLogs on a high-traffic resource. The usual offenders, in rough order of how often they surprise people:

• AKS Container Insights collecting stdout/stderr from every namespace.
• Application Gateway and Front Door access logs.
• AKS kube-audit — enormous, and rarely read outside a security investigation.
• NSG flow logs / traffic analytics.
• Storage account StorageRead on a busy account.
• An application with debug logging left on after a deployment.

Find it in ninety seconds. This is the single most useful query in Azure Monitor and it belongs in a pinned workbook:

// Which tables cost the most, last 30 days
Usage
| where TimeGenerated > ago(30d)
| where IsBillable == true
| summarize BillableGB = sum(Quantity) / 1000 by DataType
| order by BillableGB desc

// Which specific resources are responsible, for the worst table
AzureDiagnostics
| where TimeGenerated > ago(7d)
| summarize GB = sum(_BilledSize) / 1000000000 by _ResourceId
| order by GB desc
| take 20

_BilledSize and _IsBillable exist on every row, which means you can attribute cost to a resource, a namespace, or a customer — do this before negotiating with a team about their logging.

The optimisations, in the order they pay off:

1. Stop collecting what nobody queries. Audit categories against actual query history. The Usage query above versus your saved searches usually reveals something collected for two years and never read.
2. Move verbose tables to the Basic plan, with long-term retention for the compliance case. Check first that no alert rule targets them.
3. Add DCR transformations to drop rows and columns at ingestion — health-check requests, chatty CRON syslog lines, Info-level everything.
4. Buy a commitment tier once daily volume is stable. It is a property change, applies immediately, and is the least-effort saving available.
5. Tune Application Insights sampling. Adaptive sampling on the SDK preserves statistical accuracy while discarding a large fraction of telemetry; the default is often more aggressive or less aggressive than you assume.
6. Shorten interactive retention, lengthen long-term retention. Most investigation happens within days; most compliance requirements are about years. Paying interactive prices for month-old data satisfies neither.

What keeps billing when idle. Azure Monitor is mostly usage-priced, which is good — but three things bill regardless of whether anyone looks at them: retained data (every GB, every month, until it ages out), alert rules (evaluating on schedule whether or not they fire), and web tests (pinging on schedule). A decommissioned application whose workspace still holds two years of data is a recurring bill for nothing.

Use a daily cap only as a circuit breaker. It stops collection when hit, which means it blinds you at the worst moment. If you set one, always set a metric alert on the cap being reached.

Scaling and limits

You do not scale Azure Monitor; you stay inside it. The limits worth memorising, with the scope each is counted at — a number without a scope is useless in Azure:

Limit	Scope	Notes
Ingestion volume rate	Per workspace	Exceeding it throttles and delays data. Raisable via support ⚠️ verify current default
Daily cap	Per workspace	Self-imposed; stops collection
Query timeout and result size	Per query	Hit by unbounded time ranges and large joins
Concurrent queries	Per workspace, per user	A heavy dashboard on a short refresh is the usual cause
Diagnostic settings	Per monitored resource	A small number; teams routing to several destinations run out ⚠️ verify
Alert rules	Per subscription	Usually hit by a template creating one rule per resource — use multi-resource metric alerts instead
Action group notifications	Per action group, per channel	Rate-limited during storms by design
Application Insights ingestion	Per component, and against the parent workspace	Sampling is the intended relief valve
Data Collection Rules	Per subscription per region; associations per machine	⚠️ verify current values
Platform metric retention and granularity	Per subscription platform store	Not raisable — export to a workspace if you need longer

⚠️ Every number behind these rows varies by region, subscription type, and service version — verify against current Azure docs before designing to one. Azure quotas vary by region and subscription type considerably more than AWS's do; treat any figure you find in a blog post as a hint, not a fact.

Raising them: ingestion rate and most workspace limits go through a support request; some quotas are visible and adjustable via az quota. The design-level answer is usually split the workspace — by region for residency, by business unit for RBAC and chargeback, or by security-versus-operations if Sentinel is in play. The cost of splitting is cross-workspace queries; the benefit is that the workspace is the blast radius, and smaller blast radii are better.

Observability — monitoring the monitor

The observer needs observation, and this is the pillar people skip.

Diagnostic settings are not on by default — including for Azure Monitor's own resources. The workspace itself emits audit logs about who queried what, and you have to route them somewhere. The categories worth enabling on the workspace: audit (query activity) and, for anything regulated, the summary logs ⚠️ verify current category names.

The four checks that catch a broken monitoring system:

// 1. Is anything arriving at all? Run this before you believe a "no results" answer.
union withsource=T *
| where TimeGenerated > ago(1h)
| summarize rows = count(), latest = max(TimeGenerated) by T
| order by rows desc

// 2. Ingestion latency - the difference between when it happened and when you could see it
AzureDiagnostics
| where TimeGenerated > ago(2h)
| extend latency = ingestion_time() - TimeGenerated
| summarize p50 = percentile(latency, 50), p95 = percentile(latency, 95) by ResourceType
| order by p95 desc

// 3. Have agents stopped reporting? The absence alert everyone needs and few configure.
Heartbeat
| where TimeGenerated > ago(24h)
| summarize LastSeen = max(TimeGenerated) by Computer
| where LastSeen < ago(30m)

// 4. Is the pipeline itself unhealthy?
Operation
| where TimeGenerated > ago(24h)
| where OperationStatus != "Succeeded"
| summarize count() by OperationCategory, Detail

Alert on absence, not just on presence. A log search alert that fires when a query returns more than N rows will never tell you that logging stopped. Configure the no-data behaviour explicitly on every rule that guards something critical, and back it with a Heartbeat staleness alert.

Alert hygiene, which is an ongoing job rather than a setting:

• Every alert should have a runbook link in its description. An alert nobody knows how to action is noise with a pager attached.
• Use dynamic thresholds for metrics with daily or weekly seasonality rather than guessing a static number.
• Use multi-resource metric alerts rather than one rule per resource — cheaper, fewer objects, and one notification instead of two hundred during a shared-cause incident.
• Use alert processing rules for maintenance windows. Disabling rules manually is how monitoring gaps outlive the maintenance that caused them.
• Review fired-alert volume monthly. Anything that fires weekly and is always ignored should be deleted or re-tuned; it is actively training people to ignore the pager.

Reliability

The workspace is regional, and so is your ability to investigate. In a regional outage you may lose both the service and the telemetry about it. Options, in increasing order of cost:

• Accept it for most workloads — the outage is Microsoft's and the post-mortem can wait for data to become available again.
• A secondary diagnostic setting to an Event Hub or storage account in another region for the small set of signals you cannot afford to lose. This is the pragmatic middle.
• A second workspace in a paired region with duplicate diagnostic settings for critical resources. Doubles the ingestion cost for those resources; do it selectively.

Availability zones. Workspaces in supported regions are zone-redundant for the service itself ⚠️ verify current regional availability. This protects against a zone failure, not a regional one.

Backup and restore, honestly. There is no backup operation for a workspace. Your recovery options are the soft-delete window if the workspace was deleted, and whatever you exported to a storage account. If a regulator will one day ask for logs from three years ago, the answer is a diagnostic setting to immutable storage, configured today — not workspace retention.

Data residency. Data lives in the workspace's region. There is no move. Choose the region on purpose, and for multi-region estates with residency rules, accept a workspace per region and query across them.

The failure drill. Run it at least once:

1. Delete a non-critical alert rule and confirm the nightly drift job notices.
2. Remove a diagnostic setting and confirm the "no data received" alert fires — and note how long the gap in data is when you restore it.
3. Take a workspace to private-only network access in a test subscription and confirm your agents were correctly enrolled in the AMPLS.
4. Run a search job against long-term retention data and time it. Everyone assumes archived data is a click away; the first time you do it under incident pressure should not be the first time.
5. Simulate an alert storm and confirm your action group behaves — rate limits during a storm are expected, and you want to discover the deduplication gap in a drill rather than at 3 a.m.

---

Next: Interview Questions →

← Back to the Azure Monitor overview · ← Previous: Integrations