3. Architecture

This page traces the machinery: what actually happens between a resource emitting an event and you reading it in a query, why the two stores behave so differently, where the control plane stops and the data plane starts, and which failure modes you will meet in production.

The ingestion path, end to end

Follow one Key Vault secret read from the moment it happens.

1. The resource emits. The Key Vault service writes an audit event internally. At this point it exists only inside the service. Nothing you configured has been consulted yet.
2. The diagnostic setting is consulted. Azure checks whether a Microsoft.Insights/diagnosticSettings extension resource exists on that vault and whether the AuditEvent category is selected. If not, the event is discarded and there is no record that it ever existed. This is the step everyone forgets.
3. The event is fanned out. One diagnostic setting can name several destinations; several diagnostic settings can exist on one resource. Each destination gets its own copy and its own bill — routing to both a workspace and an Event Hub costs you both.
4. The pipeline reshapes it. For a workspace destination, the event is mapped into either the legacy wide AzureDiagnostics table or a resource-specific table (AZKVAuditLogs), depending on the setting's destination type. Where a DCR is involved — agent data, custom ingestion, and an increasing number of platform paths — the DCR's transformation runs here, and rows dropped by the transformation are never billed and never recoverable.
5. Ingestion commits. The row lands in the workspace with a TimeGenerated (when the event happened) and, internally, an ingestion time. These are not the same, and the gap between them is your ingestion latency ⚠️ verify current latency targets against current Azure docs.
6. Indexing and availability. After commit, the row is queryable via the workspace query API. Anything that reads it — the portal, a workbook, a log search alert, Grafana, Sentinel — is a client of that same API.
7. Alert rules evaluate on their own clock. A log search alert with a five-minute frequency runs its KQL every five minutes over a lookback window. Your end-to-end "time to notice" is therefore ingestion latency plus up to one evaluation interval plus action-group delivery. Design your expectations around minutes.
8. Retention ages the row. It stays interactively queryable for the table's interactive retention, then moves to long-term retention, where reaching it costs a search job or a restore.

Metrics take a shorter road. Platform metrics are written by the platform directly into the time-series store with no configuration, no diagnostic setting, and no ingestion charge; they are available with much lower latency, which is precisely why metric alerts fire faster than log alerts.

[Image Prompt: 2D minimalistic numbered sequence diagram tracing one audit event from an Azure resource through its diagnostic setting, a fan-out to multiple destinations, a Data Collection Rule transformation, ingestion into a Log Analytics workspace table, and finally a KQL query and an alert rule evaluation, flat design, clean vector art style, white background]

Control plane vs. data plane

Azure Monitor has one of the sharpest control/data splits in the platform, and it produces one of the most common permission puzzles on Azure: I am Owner on the subscription and I still cannot read that workspace's logs the way my colleague can.

Control plane — ARM, management.azure.com. Creating the workspace, changing its pricing tier or retention, creating and modifying diagnostic settings, DCRs, alert rules, and action groups. Governed by Azure RBAC on the ARM resource IDs.

Data plane — the query and ingestion endpoints. Reading rows (api.loganalytics.azure.com for workspace queries, api.applicationinsights.io for Application Insights, and the metrics endpoints for metrics) and writing rows (a Data Collection Endpoint for the Logs Ingestion API). Governed by a different set of roles, plus the workspace's own access-control mode.

The roles that matter, and what each side they cover:

Role	Plane	What it actually lets you do
Monitoring Reader	Both, read-only	Read all monitoring data and settings across the scope. The right role for an on-call engineer who should see everything and change nothing
Monitoring Contributor	Control	Create and edit diagnostic settings, alert rules, action groups, workbooks. Does not imply broad data read on its own
Log Analytics Reader	Data (+ settings read)	Query all data in the workspace, plus read its configuration
Log Analytics Contributor	Control + data	Everything above plus changing the workspace, installing extensions, and configuring collection. Note this is a powerful role — it can turn on collection that costs money and can read all data
Monitoring Metrics Publisher	Data, write	Push custom metrics to a resource. The role a workload identity needs, and nothing more
Application Insights Component Contributor	Control	Manage the Application Insights resource

The classic mistakes. First: granting Contributor on the resource group and assuming an application team can read its own logs — the workspace's access mode decides that, not the resource group. Second: granting Log Analytics Contributor when Log Analytics Reader was meant, which hands over the ability to reconfigure collection and inflate the bill. Third: giving a workload Monitoring Contributor so it can publish a custom metric, when Monitoring Metrics Publisher is the correct, far narrower role.

Workspace access modes. The workspace carries an access-control mode that decides how the two paths combine:

• Require workspace permissions — only workspace-level RBAC grants read. Nothing leaks; nothing is self-service.
• Use resource or workspace permissions — a user with read access to an Azure resource can query that resource's rows in the workspace, without any workspace permission. This is resource-context access and it is how you give twenty application teams their own logs from one shared workspace. It works because rows carry a _ResourceId column; rows that lack one — some custom tables and some agent data — are invisible in resource context, which is a real and under-documented gotcha.
• Table-level RBAC — a custom role granting or denying Microsoft.OperationalInsights/workspaces/query/<Table>/read, for the case where a single table (say, SigninLogs) must be restricted more tightly than the rest.

[Image Prompt: 2D minimalistic diagram of two separate doors into Azure Monitor — a control plane door at management.azure.com governed by Monitoring Contributor and Log Analytics Contributor, and a data plane door at the query and ingestion endpoints governed by Log Analytics Reader, resource-context access, and Monitoring Metrics Publisher — with a Log Analytics workspace behind both, flat design, clean vector art style, white background]

Inside the two stores

Metrics. A pre-aggregated time-series database. The platform writes values at a fixed granularity and rolls them up; you query by choosing a metric, an aggregation (average, total, min, max, count), a time grain, and optionally a dimension split. The consequences of that design:

• It is fast and cheap because the shape is fixed. There is no schema flexibility to pay for.
• Dimensionality is limited. You cannot split by an arbitrary high-cardinality attribute like customer ID. If your question needs that, it is a logs question or a Prometheus question.
• Retention is fixed and comparatively short. If you need a year of CPU history, export the metric to a workspace via a diagnostic setting's metric category — at which point you are paying logs prices for it, which is the correct trade but should be a decision.
• Aggregation is lossy in the way you would expect: a one-minute average hides a five-second spike. Alerting on Maximum rather than Average is often what you actually meant.

Logs. A columnar, schema-on-write store descended from Azure Data Explorer (Kusto). Its behaviour follows from that lineage:

• Queries are time-partitioned. Filtering on TimeGenerated first is not a style preference — it is the difference between scanning a day and scanning a year.
• Columns are typed, and a _CL custom table's column types are set on first ingestion, so a sloppy first payload can lock in a bad schema.
• join is available and genuinely powerful, but the left side is the one held in memory — put the smaller table on the left, and prefer lookup for small dimension tables.
• Basic and Auxiliary plan tables are stored differently and expose a restricted query surface. A query that works on an Analytics table may simply be rejected on a Basic one, and log search alerts cannot target them at all. Re-planning a table from Analytics to Basic will silently break any alert rule on it, which is one of the more painful self-inflicted incidents available here.

Prometheus metrics are a third path: managed Prometheus scrapes your endpoints, stores samples in an Azure Monitor Workspace, and is queried with PromQL. It exists because Kubernetes shops have a large investment in that ecosystem and because Prometheus handles high-cardinality labels that the platform metrics store will not.

Scaling, throttling, and the ceiling

Azure Monitor is a shared multi-tenant platform service, so you do not scale it — you stay inside its limits. The ones that matter, with the scope each is counted at, because a number without a scope is useless:

Limit	Counted at	Why it bites
Ingestion volume rate	Per workspace	A burst above the ingestion rate limit gets throttled and the data is delayed or dropped; there is a raisable quota ⚠️ verify current default
Query concurrency and query rate	Per workspace, per user	A dashboard refreshing many heavy tiles is a self-inflicted denial of service
Query result size and query duration	Per query	Big joins and unbounded time ranges hit this before they hit anything else
Number of alert rules	Per subscription	Rarely hit deliberately, frequently hit by a template that creates a rule per resource
Diagnostic settings per resource	Per monitored resource	A small number — teams that route to a workspace, an Event Hub, and two partner tools run out ⚠️ verify current limit
Action group rate limits	Per action group, per channel	Alert storms get throttled per channel, so SMS/voice suppression during a storm is expected behaviour, not a bug
Daily cap	Per workspace	Self-imposed; when hit, collection stops
Metric retention and granularity	Per subscription platform store	Not raisable — export if you need longer

⚠️ Every number behind these rows varies by region, subscription type, and service version — verify against current Azure docs before designing to one. The durable lesson is the shape: your ceiling is the workspace, and the workspace is the blast radius.

Failure modes

• The silent one: no diagnostic setting. The system appears healthy because the query returns zero rows. Guard against it with Azure Policy DeployIfNotExists and with a "no data received" alert per critical source.
• Ingestion latency mistaken for an outage. A query run 90 seconds after an event legitimately returns nothing. Before declaring data loss, check ingestion_time() against TimeGenerated.
• Throttling at ingestion. Sudden volume — a debug flag left on, a retry storm — exceeds the workspace rate limit and data is delayed. The Operation table in the workspace records ingestion problems; it is the first place to look.
• The daily cap firing. Collection stops mid-incident. Always pair a cap with an alert on reaching it.
• Alert rules that fire on no data — or don't. A log search alert's behaviour when the query returns no rows is configurable, and the default is frequently not what people assume. If your alert is meant to catch absence (a heartbeat stopping), you must configure that explicitly.
• Alert storms. One underlying fault triggering hundreds of resource-scoped rules. Mitigate with multi-resource metric alerts, dynamic thresholds, and alert processing rules rather than by turning rules off.
• Table-plan change breaking alerts. Re-planning a table to Basic disables log alerts on it.
• Regional dependency. The workspace is regional. A regional outage affects both your telemetry and, potentially, your ability to see what is happening — which is the argument for having a secondary workspace or an Event Hub export for the signals you cannot afford to lose.
• Cost as a failure mode. This is the one that actually pages people. Uncontrolled ingestion is the most common Azure Monitor incident by a wide margin, and it always traces to a specific category on a specific resource. Production covers finding it.

---

Next: Getting Started →

← Back to the Azure Monitor overview · ← Previous: Core Concepts