8. Interview Questions

A managed API gateway plus a governance layer. It terminates client calls to your HTTP APIs, runs a declarative policy pipeline on each one (authentication, rate limiting, transformation, caching, logging), and forwards them to backends — while separately controlling which consumers may call which APIs, on what terms, and publishing that in a developer portal.

The problem it kills is the fan-out of cross-cutting concerns. Without a gateway, every backend team reimplements auth, throttling, logging, CORS, key issuance and versioning, slightly differently — so one policy change becomes N pull requests in N languages. APIM makes those concerns one reviewable artifact.

The second problem is presentation: it lets you publish the API you wish you had in front of the one you actually have — a REST facade over SOAP, a clean path structure over an inconsistent one.

Say explicitly what it is not: it is not a WAF, not a CDN, and not a service mesh.

Operation (one method + URL template, e.g. GET /orders/{id}) lives in an API (a set of operations sharing a URL suffix and a backend), which lives in the API Management service instance (Microsoft.ApiManagement/service) — the ARM resource — which lives in a resource group, in a subscription, in a tenant. See the scope hierarchy.

The part that surprises people: APIs, operations, policies, products and named values are all ARM child resources of the service. API configuration is infrastructure here, which is why Terraform genuinely manages your policy XML and why a portal edit is drift.

Crossing the hierarchy sideways: a product groups APIs for commercial packaging, and a subscription is the join between a product (or an API, or all APIs) and a consumer, carrying two keys.

The consequence for RBAC: Azure RBAC's natural grain is the whole instance, so a Contributor on one instance can edit every API in it. Premium workspaces or separate instances are the answers.

• An API is technical: operations and a backend.
• A product is commercial: a bundle of APIs with terms — published or not, subscription required or not, approval required or not, a quota, a portal page.
• A subscription is the entitlement record joining a consumer to a product. It has a state (active, suspended, expired, …) and carries two keys, primary and secondary, so you can rotate without an outage.
• A subscription key is the bearer secret sent in Ocp-Apim-Subscription-Key.

The point that matters in a real interview: a subscription key is not authentication. It identifies and meters a consumer. It's a shared secret that ends up in mobile apps and repositories. Authentication is validate-jwt against Entra ID (or another OIDC issuer), or a client certificate. Use keys for metering and quota, tokens for identity, and don't conflate them.

Quotas usually live at product scope (commercial terms); rate limits often live at API or operation scope (backend protection).

They answer different questions. Front Door answers where should this request go — global anycast, edge caching, WAF, cross-region routing. APIM answers is this caller allowed to make this request, how many more may they make, and what should the request and response look like on the way through.

Front Door alone: you need global routing, edge caching or OWASP WAF, and have no API-level governance requirements.

APIM alone: internal or single-region API estate with governance needs and no untrusted internet exposure.

Both, which is the common enterprise answer: Front Door (or Application Gateway for a single region) in front for WAF, TLS, and global entry; APIM behind it for API semantics. The important detail is closing the side door — if callers can still reach *.azure-api.net directly, the WAF is decorative. Fix it with internal-mode networking or Private Link, with a header check as defence in depth, not as the control.

Also worth naming: APIM has no WAF of its own, which is precisely why this pairing exists.

Every tier except Consumption is a provisioned resource billed per unit-hour, per region, running 24/7. Consumption bills per call after a monthly grant. ⚠️ Verify current pricing.

What keeps billing at zero traffic: the gateway itself, every additional region, every self-hosted gateway, an attached Azure Cache for Redis, and — the one that surprises people — Application Insights and Log Analytics ingestion, which on a busy gateway with 100% sampling can exceed the gateway's own cost.

The comparison worth making: coming from Amazon API Gateway's per-request model, budget APIM like a VM, not like Lambda. The usual cost regrets, in order: Developer instances left running per engineer, App Insights sampling at 100%, an over-provisioned tier bought for one feature, a forgotten second region, and long Log Analytics retention on GatewayLogs.

1. DNS and TLS at the gateway hostname; mTLS negotiation if the API requires a client certificate.
2. Route to an operation — match the API's URL suffix, then the operation's URL template. No match, 404, and none of your policies ever run.
3. Subscription resolution — read the key, resolve the subscription, check state and quota. This is the only built-in check, and it runs before your policies.
4. Inbound pipeline, outermost scope first: global → workspace → product → API → operation, with <base /> marking where the parent's policy executes. validate-jwt, rate-limit, validate-content, header hygiene. A cache-lookup hit or a return-response short-circuits here and the backend never hears about the request.
5. Backend section — the effective backend is chosen (URL, backend entity, or pool with circuit breaker), backend auth is applied, forward-request runs with its timeout and retry.
6. Backend responds (or times out, or the breaker trips).
7. Outbound pipeline, innermost scope first — the mirror image. Transformations, header removal, cache-store, emit-metric.
8. Response returned, telemetry written asynchronously.
9. <on-error> runs instead of the remainder if anything threw, with context.LastError available.

The two facts to state explicitly: inbound is outside-in, outbound is inside-out, and omitting <base /> silently drops the parent scope's policy — which is how a global validate-jwt ends up not running.

Control plane is ARM (management.azure.com), governed by Azure RBAC — API Management Service Contributor, ...Reader Role, ...Operator Role, or a custom role ⚠️ verify the current list. It governs the service resource and its configuration: APIs, policies, products, subscriptions, named values, tier, units, networking.

Data plane is the gateway (*.azure-api.net), governed by whatever the policies demand: subscription keys, JWTs, client certificates. No Azure RBAC role grants the right to call an API through the gateway, and no subscription key grants any ARM permission.

The classic mistake is assuming one implies the other — an engineer who can see and edit the API in the portal assuming they can therefore call it, or the reverse.

The APIM-specific wrinkle worth adding: because configuration is on the control plane, RBAC on the instance is effectively edit rights over every API in it — hence workspaces, separate instances, or a pipeline-only write model. And there is a third surface, the legacy direct management endpoint (<name>.management.azure-api.net), which authenticates outside Entra ID entirely and should be disabled.

Revisions are for non-breaking changes. A revision is an invisible-to-consumers working copy of an API. You edit revision 2 while revision 1 stays current, test it at its own ;rev=2 URL, then make it current — an atomic switch taking seconds. Consumers never change anything.

Versions are for breaking changes. A version is a deliberate, consumer-visible variant at a different URL path, query parameter, or header, grouped in a version set. Both versions run simultaneously and consumers migrate on their own schedule.

Get it wrong in either direction and it hurts: a version where a revision belonged leaves you maintaining two contracts forever; a revision where a version belonged breaks someone's client without warning.

The operational payoff: revisions are APIM's rollback mechanism. Making the previous revision current again is the fastest undo the service has — the closest thing APIM has to a deployment slot swap, and the reason "every change lands as a revision" is a habit worth enforcing.

The tier decides which features exist, not merely how fast they run.

• Consumption — serverless, per-call, near-zero idle cost, instant provisioning. No VNet, no developer portal, no built-in cache, no multi-region, cold starts, stricter limits ⚠️ verify.
• Developer — the full classic feature set at the lowest price, one unit, and no SLA. The classic trap: perfect for building against, unsuitable for production.
• Basic / Standard — production tiers with an SLA but without VNet integration or multi-region in the classic generation ⚠️ verify.
• Premium — multi-region, VNet injection in external or internal mode, self-hosted gateways, workspaces, zone redundancy, the highest unit ceiling. Expensive, per region, always on.
• v2 tiers (Basic v2 / Standard v2 / Premium v2) — a newer platform whose headline benefit is provisioning in minutes rather than tens of minutes, plus a simplified networking model and a lower entry price for network-integrated scenarios. The feature matrix differs from classic in both directions ⚠️ verify current GA status and matrix.

How to choose: does it need a private network? (rules out Consumption) · does it need active-active multi-region? (classic Premium) · is it production? (rules out Developer). Take the cheapest survivor.

What moving costs you: tier changes are online but slow on classic tiers, and scaling down is not always symmetric with scaling up ⚠️ verify current constraints. More importantly, some moves change the architecture — going from a public tier to VNet injection means subnet design, NSG rules, DNS, and possibly a new Application Gateway in front. That's a project, not a property change.

Four layers, none of which is optional:

1. Caller → gateway: validate-jwt against Entra ID's OIDC discovery document, checking issuer, audience and claims (roles or scp). Subscription keys remain for metering and quota, not as the security control. Client certificates where the caller is a system.
2. Gateway → backend: APIM's managed identity (user-assigned, so it survives instance re-creation) with authentication-managed-identity, plus an Azure RBAC role assignment on the backend. No shared secret exists. Where the backend can't do Entra, a client certificate from Key Vault.
3. Secrets that must exist: named values as Key Vault references, resolved by the managed identity, versionless so rotation is automatic — and therefore not in Terraform state. The gateway certificate likewise.
4. Network: internal-mode VNet injection or Private Link so the gateway has no public endpoint, private endpoints to backends, and a WAF in front for public exposure.

Then the least-privilege part on the control plane: nobody has standing write access to production configuration — changes go through a pipeline authenticated by workload identity federation (OIDC), with a federated credential scoped to one repo and one GitHub environment, and a role assignment scoped to the resource group.

And the honest caveat: keep authorisation in the backend too. Anything with network reach to the backend bypasses every policy.

The hard part is a property of the service: creating or materially reconfiguring a classic instance takes tens of minutes, while creating an API or policy inside it takes seconds. So split the deployment in two:

• Infrastructure track — Terraform managing the instance, tier, units, VNet, custom domains, identity, loggers, diagnostic settings. Runs rarely, owned by the platform team, with generous provider timeouts blocks.
• Configuration track — APIs, operations, policies, products, named values. Runs constantly, owned by API teams. Terraform for a modest portfolio; the APIOps extract/publish pattern for a large federated one ⚠️ verify current tooling status.

Pipeline specifics: workload identity federation / OIDC against an Entra app registration — never a client secret — with the federated credential's subject pinned to repo:org/repo:environment:prod so the GitHub environment's approval gate is load-bearing. plan on pull request with the output posted to the PR, apply on merge, manual approval for prod, remote state in an Azure Storage backend using native blob leases for locking.

Two APIM-specific pipeline steps most people miss: lint the policy XML for a missing <base /> (its absence silently disables inherited security policies and nothing else will warn you), and purge the soft-deleted instance if a destroy-and-recreate flow needs the name back.

State protection matters more than usual here, because the state file contains your policy XML and any non-Key-Vault named values.

The honest answer starts with the method: read the terraform plan for # forces replacement rather than trusting any written list, including this one — the provider's behaviour changes. ⚠️ Verify against your provider version.

That said, the reliably destructive ones are the identity of the resource: name, location, and resource group. Several networking and identity changes are in-place but slow.

What replacement costs you, and this is the part that separates a real answer from a recited one:

• Tens of minutes of downtime while the new instance provisions.
• New outbound public IP addresses, which break every backend firewall rule and partner allow-list written against the old ones.
• Soft delete blocks the recreate. The old instance keeps its globally unique name for a retention window ⚠️ verify current window, so the create fails with a name conflict until you run az apim deletedservice purge — which is also why an accidental delete should be restored rather than rebuilt.
• A system-assigned managed identity gets a new principal ID, invalidating every role assignment and Key Vault access policy referencing it. This is the argument for a user-assigned identity created outside the module.

Related traps to name: a CanNotDelete resource lock makes the apply fail in a way that reads like an authorisation error, and an ARM deployment in complete mode can remove child entities — APIs, products, policies — that the template doesn't declare.

The hypothesis to state first: capacity is consumed by concurrent in-flight requests, not requests per second. A backend that degraded from 50 ms to 5 s multiplies concurrency by a hundred at unchanged traffic. "Idle" from the backend's perspective can mean "each request is now waiting on a database lock" — low CPU, high latency, and the gateway holding every connection open.

Diagnose:

• Query ApiManagementGatewayLogs for p95 TotalTime and BackendTime by API and operation. TotalTime - BackendTime is the gateway's own overhead; if BackendTime is what grew, it's the backend regardless of what its dashboard says.
• Check whether the 503s are APIM-generated or forwarded — the logs distinguish them.
• Check whether one subscription's traffic changed. A client that started retrying in a tight loop looks exactly like this.
• Check whether a policy was deployed recently: a send-request added to inbound puts a synchronous network call on every request, and an unsampled Application Insights logger costs real latency.

Fix, in order:

1. A backend timeout and a circuit breaker. This is the actual fix. It converts a slow backend into fast failures and stops the concurrency spiral.
2. Rate-limit the offending consumer if one caller is responsible.
3. Add units — but say clearly that this is buying capacity to hold connections open, and only buys time.
4. Response caching on read-heavy operations, which reduces concurrency, not just backend load.

The senior-signal sentence: "adding units to absorb a slow backend is treating the symptom; the gateway's capacity is a concurrency budget."

Tier and topology. Premium (or the appropriate v2 tier ⚠️ verify current capability), VNet injected in internal mode, with Front Door Premium or Application Gateway WAF v2 in front for public exposure. Zone redundancy in each region.

Data residency is the constraint that shapes it. "Two regions" with residency usually means data must stay in each region, which rules out a single multi-region instance routing traffic to whichever region is nearest. Two options, and you should say which and why:

• Two independent instances, one per region, each fronting that region's backends, with routing by hostname or by Front Door rules on a residency-determining attribute. Correct for genuine residency requirements.
• One multi-region Premium instance, correct when the requirement is availability rather than residency. Note the caveats: DNS-based failover latency, and the control plane living in the primary region.

Team isolation for eight teams. Azure RBAC's grain is the whole instance, so name the three options: Premium workspaces (per-team RBAC inside one instance — the intended answer), separate instances per boundary (clean, expensive, and constrained by the instances-per-subscription-per-region quota ⚠️ verify), or a pipeline-only model where nobody has standing write access and isolation is enforced by CODEOWNERS on the repository. In practice a combination: workspaces plus pipeline-only writes.

Configuration at fifty APIs. Terraform for the instance; APIOps extract/publish for the API configuration, because fifty azurerm_api_management_api blocks in one state file is a slow plan and a merge-conflict generator ⚠️ verify current tooling status.

Partners. Products with approval required, subscription keys for metering, Entra External ID or the partner's own OIDC issuer for authentication via validate-jwt, per-product quotas, and the developer portal restricted to signed-in users.

Cross-cutting policy at global scope in a policy fragment: correlation ID, JWT validation, header hygiene, a default rate limit. Per-API policies own only what's genuinely per-API.

Operations. Diagnostic settings to a central Log Analytics workspace, App Insights at low sampling, Azure Policy at the management-group level denying Developer tier and public networking in production, CanNotDelete locks, scheduled drift detection, and revisions as the standard change mechanism.

First, work out which layer failed — infrastructure or configuration — because the two have very different recovery costs.

Configuration failure (the likely case, and the recoverable one):

• If each API change landed as a revision, roll back by making the previous revision current. Seconds, atomic, per API. This is why the revision habit matters.
• Otherwise git revert and re-run the pipeline. Terraform and APIOps are both convergent — a half-applied run followed by a corrected run reaches the right state.
• A likely root cause worth naming: ARM control-plane throttling on a pipeline pushing many API definitions in a loop ⚠️ verify current limits. The symptom is exactly "60% deployed, then errors." The fix is batching and retry with backoff, not a bigger runner.
• az apim restore from a backup exists but reverts everything, including other teams' APIs. It's a disaster tool, not a rollback tool.

Infrastructure failure (the expensive case): if the instance itself was mid-change, you may be waiting tens of minutes for it to settle before anything else will apply. If Terraform timed out mid-create, you may have a real instance the state file doesn't know about — import it rather than re-creating.

The Azure-specific part of the question — what does a re-run in complete deployment mode do? Complete mode deletes every resource in the resource group that the template doesn't declare. For APIM that is unusually dangerous because APIs, products, policies and named values are themselves ARM child resources, so a template that declares only some of them is a plausible way to delete API configuration you never intended to touch ⚠️ verify exactly how complete mode treats child resources for this provider. Recovery means re-provisioning, which means tens of minutes and new outbound IPs. Use incremental mode; use deployment stacks if you need managed deletion.

Blast radius, stated: one instance fronting fifty APIs means one bad global policy is a fifty-API outage. That asymmetry is the argument for policy fragments over copy-paste, staged environments, revision-based change, and — past a certain size — workspaces or multiple instances.

First: that was probably the right call at 3 a.m. The job is detection and reconciliation, not blame. Say that — it's the difference between an answer and a good answer.

Detection:

• Scheduled terraform plan -detailed-exitcode in CI against every environment, nightly. Exit code 2 means drift; open an issue automatically.
• The activity log answers who and when for the instance and its child resources. Change Analysis shows the before and after.
• Azure Policy compliance state catches guardrail drift — tier, networking, diagnostics, tags.
• APIM-specific: revision change logs record who made a revision current; if you use the APIOps pattern, running the extractor against the live instance and diffing it against git is your drift detector, and it's the most complete one available.

Reconciliation:

1. Read the diff before doing anything. Do not blind-apply over production drift.
2. Decide whether the portal change was correct. It usually was — someone was fixing something real.
3. If correct, port it into the repository (policy XML file, .tfvars, or an extracted API definition), review it as a normal pull request, and apply. The next plan is a no-op.
4. If incorrect, apply the repository's version and tell the person what happened and why.
5. Then close the loop structurally: if a human needed portal write access during an incident, either the pipeline is too slow to use under pressure or the break-glass path isn't documented. Fix whichever it was. Just-in-time elevation (Entra PIM) plus a documented break-glass procedure is the sustainable version of "nobody has standing write access."