7. Production

Five pillars, the same five as every topic in this article: security, cost, scaling and limits, observability, and reliability. Foundry adds a sixth concern the other services do not have — whether the output is any good — and that one is discussed under observability, because in production it is a monitoring problem, not a development one.

Security

Turn off keys. disableLocalAuth = true (local_auth_enabled = false in Terraform) is the highest value change you can make on this resource. While keys are enabled, anyone with the control-plane Cognitive Services Contributor role can read them, and any leaked key is an unattributable caller. With local auth off, every request carries an Entra ID identity that appears in the audit log. Enforce it with Azure Policy, not with a wiki page.

Least privilege, concretely. The roles you will actually use, and what each is for:

Role	Plane	Grant it to
Cognitive Services OpenAI User	Data	The application identity that calls models. This is the common case, and it is not implied by Owner
Cognitive Services OpenAI Contributor	Data	Rarely — callers that also manage data-plane objects like fine-tunes
Foundry User (formerly Azure AI User)	Data (project)	People and services that use agents, threads, and evaluations
Foundry Project Manager (formerly Azure AI Project Manager)	Data (project)	People who create agents and publish them
Cognitive Services Contributor	Control	Deployment pipelines. Note it can read keys — which is another reason to disable them
Foundry Account Owner (formerly Azure AI Account Owner)	Control	Platform owners of the account

Where the built-in role is too broad, the custom role worth writing is a read-only inference auditor: Microsoft.CognitiveServices/accounts/read plus the diagnostic and metric read actions, and nothing that can call a model or read a key. Compliance and FinOps people ask for access and are usually given Reader on the subscription, which is both too much and not enough.

Managed identity everywhere. Foundry to AI Search, Foundry to Storage, Foundry to Cosmos DB, your app to Foundry, APIM to Foundry. There is no leg of a well-built Foundry architecture that needs a stored secret. Remember that the portal auto-assigns some data-plane roles and automation does not.

Network isolation. Private endpoint on the account sub-resource, publicNetworkAccess disabled, networkAcls.defaultAction set to Deny, and the Private DNS zones linked to every VNet that must resolve it — including the one your build agents run in. Also consider restricting the account's outbound access, so a connection cannot be pointed at an arbitrary internet endpoint.

Encryption. Data at rest is encrypted with platform-managed keys by default. Customer-managed keys in Key Vault are supported and require the vault to have purge protection enabled — which in turn means that vault cannot be cleanly torn down, so put it somewhere your demo teardown will not try to delete. Transit is TLS; enforce a modern minimum.

Content safety is a security control, not a nicety. The default filter catches the obvious categories. Add prompt shields for injection attempts, protected-material detection where copyright matters, and blocklists for domain-specific terms. Then handle the filter error distinctly in your application — telling a user "something went wrong" when the real answer is "that request was blocked" produces both a bad experience and an unexplainable incident.

The prompt-injection reality. No filter makes an LLM safe to give unrestricted tool access. If an agent can read untrusted content (a web page, an inbound email, a user-uploaded document) and call a tool with real effect, assume the untrusted content can influence the tool call. Design for it: authorise tools against the end user rather than the agent, keep destructive actions behind human confirmation, and keep the blast radius of any single tool small.

Cost

What you pay for:

• Tokens, on standard deployments — input and output priced differently, output typically costing several times input. Cached input, where supported, is discounted, which rewards a byte-stable system prompt.
• PTU-hours, on provisioned deployments — from creation to deletion, traffic or not. Reservations cut the rate for a term commitment.
• Batch — discounted tokens for asynchronous work.
• The dependencies — AI Search (priced by tier and replica, running continuously), Cosmos DB (provisioned throughput or serverless, running continuously), Storage, and Log Analytics ingestion.
• Log Analytics ingestion, which on a chatty application with request/response logging on can rival the token bill. Prompts and completions are large.

⚠️ All pricing shapes are conceptual — verify current rates, free grants, and reservation terms against current Azure docs.

The biggest cost trap is an under-utilised provisioned deployment. PTU makes sense when you have steady, predictable, latency-sensitive traffic; it is the most expensive possible way to serve a bursty workload, because you size for the peak and pay for it at 3 a.m. Measure with a standard deployment for several weeks first, and size the reservation from the measured p95, not from the launch forecast.

The second trap is the agent's tail: a project with a standard agent setup is running a Cosmos DB account and an AI Search service continuously. On a low-traffic internal agent, those two can cost more than every token the agent ever generates. Know that before you choose standard over basic.

Three concrete optimisations, in order of payoff:

1. Route by task. Not every call needs the flagship model. A cheap model behind classify-cheap and the flagship behind chat-default — chosen by your code, not by the model — frequently halves the bill with no measurable quality change. Named deployments make this a config change.
2. Control the context. Token cost is dominated by input on RAG workloads. Retrieving ten chunks where three would do is a permanent 3x on every call. Tune retrieval before you tune the model.
3. Cap the consumers. APIM token-limit policies per consumer turn "someone's test loop cost us £4,000 overnight" into "someone's test loop got throttled". Combine with budget alerts on the subscription and a metric alert on token consumption.

Tag everything, and put each environment in its own subscription — which you are doing anyway for quota reasons — so the invoice answers "which team" without a spreadsheet.

Scaling and limits

There is no autoscale. Scaling on this service means one of four things, and you should know which one you are doing:

1. Increase the deployment's capacity — possible only up to your remaining quota pool.
2. Request more quota — a portal request or support ticket, with lead time, which can be partially granted or refused in a hot region.
3. Add a region — another account, matching deployment names, a router in front.
4. Buy provisioned throughput — deterministic capacity, at the cost of paying for the peak.

The quota scopes, because a number without a scope is useless:

Limit	Counted at
TPM (tokens per minute) for a standard deployment	Per subscription, per region, per model, per deployment type
RPM (requests per minute)	Derived alongside TPM, same scope; binds first on many-small-call workloads
PTU quota	Per subscription, per region, per deployment type
Number of Foundry/Cognitive Services accounts	Per subscription, per region
Deployments per account, projects per account, connections per project	Per account / per project
Agent, thread, file, and vector store limits	Per project, and dependent on basic vs. standard setup

⚠️ Every number behind these limits varies by region, model, and subscription type, and changes often. Verify against current Azure docs and your own subscription's quota page rather than trusting any figure you read anywhere — including here.

Handle 429 properly. Honour Retry-After — do not implement your own fixed backoff and ignore the header. Add jitter. Decide in advance whether a throttled request should fail fast (interactive UI) or queue (batch), because those want opposite behaviours, and route them to different deployments so they cannot starve each other.

Observability

Diagnostic settings are off by default. Nothing is logged until you create one. The categories that matter on a Foundry account:

• Audit — control-plane and access events. Always on.
• RequestResponse — request metadata and, if you opt in, prompt and completion content. High volume and high sensitivity: it contains whatever your users typed. Decide deliberately, and set retention accordingly.
• Trace — agent and orchestration traces.
• AllMetrics — token consumption, request counts, latency, and throttling.

⚠️ Verify the current log category names against current Azure docs; these have changed as the endpoints consolidated.

Route to a Log Analytics workspace and, for application-level tracing, wire the SDK's OpenTelemetry output to Application Insights so a single trace spans your application, the agent run, each tool call, and the model call. Without that, "the agent was slow" is unanswerable — you cannot tell whether it was six model calls or one slow tool.

Alarm on these four:

• Throttled request rate — your leading indicator that capacity is running out.
• Token consumption rate against your quota — so you see it coming before the throttling does.
• p95 latency, per deployment — a shared standard deployment degrades quietly under contention.
• Content-filter block rate — a spike is either an attack or a broken prompt, and both matter.

A starting KQL query — where are the tokens going?

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Category == "RequestResponse"
| summarize
    Calls        = count(),
    PromptTokens = sum(toint(properties_promptTokens_d)),
    Completion   = sum(toint(properties_completionTokens_d)),
    p95_ms       = percentile(DurationMs, 95)
  by DeploymentName = tostring(properties_modelDeploymentName_s), bin(TimeGenerated, 1h)
| order by PromptTokens desc

⚠️ Column and property names in the diagnostic schema differ by table and API version — check your own workspace's schema before relying on this verbatim.

Monitor quality, not just health. This is the pillar unique to AI workloads. Run the evaluation suite on a schedule against production-shaped traffic, not just in the pipeline. Track groundedness and relevance as time series next to latency and cost. Capture user feedback signals — thumbs, corrections, abandonment — and treat a drop as an incident, because a model that is up and wrong looks perfectly healthy on every infrastructure dashboard you own.

Reliability

A Foundry account is regional and there is no failover switch. Plan accordingly.

• What Azure gives you: the endpoint's availability SLA, and — with GlobalStandard — routing around fleet-level capacity problems. Preview features and preview models carry no SLA at all. ⚠️ Verify current SLA terms against current Azure docs.
• What you must build: two or more accounts in different regions with identical deployment names, behind API Management or Front Door, failing over on 429 and 5xx. Identical names are what make the router a configuration rather than a code path — another argument for naming deployments by job.
• Provisioned + spillover: reserved capacity for the steady load with overflow to a standard deployment, so bursts degrade in cost rather than in availability. ⚠️ Verify current spillover support.
• Agent state is only as durable as its store. Basic setup: Microsoft's problem. Standard setup: your Cosmos DB, your replication choice, your backup policy, your restore test.
• Model deprecation is a scheduled outage. Every deployed model has a retirement date. Track them, decide the upgrade policy per deployment, and rehearse a version change in staging before the calendar forces one in production.
• Graceful degradation belongs in the application. A cached answer, a smaller model, or an honest "this feature is temporarily unavailable" is a better experience than a spinner. AI features are usually enhancements — design them so the product still works when the model does not.

The failure drill worth running once a quarter: disable the primary region's deployment, confirm the router fails over, confirm latency and cost stay acceptable on the secondary, and confirm your alerts fired. Then do the version-change drill: point chat-default at a different model version in staging and see what your evaluation suite says. The second drill finds more problems than the first.

---

Next: Interview Questions →

← Back to the Azure AI Foundry overview · ← Previous: Integrations