8. Interview Questions

Object storage: a managed service that keeps arbitrarily large sequences of bytes under string keys inside containers, addressable over HTTPS at https://<account>.blob.core.windows.net/<container>/<blob>.

The problem it kills is durable file storage at unpredictable scale. Doing it yourself means disks, RAID, replication, a serving fleet, and a capacity forecast. Blob Storage replaces all of that with an HTTP endpoint, capacity you never provision, durability from three synchronous replicas, and a bill measured in gigabyte-months.

The trade you accept: you give up the filesystem. No partial in-place writes, no directory semantics (unless you enable the hierarchical namespace), no rename, no file locking, no query by content. In exchange you get everything else.

subscription → resource group → storage account → blob service → container → blob, with snapshots and versions hanging off a blob.

The level that matters and that people skip is the storage account. It's a regional ARM resource (Microsoft.Storage/storageAccounts) with a globally unique DNS name — 3–24 characters, lowercase alphanumeric only — and it owns the redundancy setting, the firewall and network rules, the encryption configuration, the access keys, the four service endpoints, and most of the scale limits.

The practical consequence: the account, not the container, is the throughput and blast-radius boundary. Designing one giant shared account is how an analytics burst throttles a customer-facing API, and the diagnosis takes a week because the metric that shows it is off by default.

The container is the finest scope at which you can assign a data-plane RBAC role, which is why least-privilege designs assign at container level rather than account level.

Consistency is strong, always. A successful write is immediately visible to every subsequent read and list within the primary region. There is no eventual-consistency window to design around — unlike the pre-2020 S3 folklore people sometimes carry over. Writes are atomic per blob; there is no cross-blob transaction.

Durability starts with three synchronous replicas within the primary location before any write is acknowledged. That floor is the same for every redundancy option. Redundancy determines what failure domain those copies span:

• LRS — three copies in one datacentre. Survives disk/node/rack failure.
• ZRS — three copies across availability zones in the region. Survives losing a whole zone.
• GRS — LRS primary plus an async copy in the paired region. Survives regional loss, with a manual failover and a non-zero RPO.
• GZRS — ZRS primary plus the async geo copy.
• RA- variants add a readable secondary endpoint.

Two clarifications worth volunteering: geo-replication is asynchronous (check the last-sync-time metric — everything after it is lost in an unplanned failover), and redundancy is not backup. Every option faithfully replicates a deletion. Soft delete, versioning, and point-in-time restore are what protect against mistakes.

Choose by the interface the consumer needs:

• Blob — the consumer speaks HTTPS and reads by key. Large, mostly-immutable objects: images, video, backups, parquet, model artifacts. Cheapest per GB, effectively unlimited scale.
• Azure Files — an existing application expects a mounted filesystem with POSIX or Windows semantics and you can't change it. Costs more per GB, lower scale ceiling, per-share throughput limits. The honest answer for lift-and-shift.
• Managed Disks — block I/O attached to one VM: OS disks, database data files. Not URL-addressable, mostly single-attach, billed on provisioned size whether used or not.

The tell for a wrong choice: if you're mounting blob storage with BlobFuse so a legacy app can use a path, you probably wanted Files. If you're listing blobs to find one matching a condition, you wanted a database.

Five meters: capacity (GB-month), transactions (per 10,000 operations), data retrieval (per GB from Cool/Cold/Archive), egress, and early deletion penalties.

The critical relationship: capacity rates fall Hot → Cool → Cold → Archive while transaction and retrieval rates rise. Frequently-read data in Cool costs more than in Hot. That inversion is why tiering needs measured access patterns, not a blanket age rule.

What keeps billing when idle: everything stored — including things that don't show in a container listing. Soft-deleted blobs, old versions, snapshots, and uncommitted blocks from failed uploads all bill at the full rate. Versioning with no lifecycle rule expiring old versions is the most common runaway cost in the service, and it's invisible in the portal.

Unlike compute-plan services, there's no always-on charge for the resource existing — an empty storage account is free. The bill is entirely a function of bytes and requests.

1. Token. The client's managed identity gets an access token for https://storage.azure.com/ from Entra ID. (The legacy path computes an HMAC signature with the account key — no round trip, which is why it's faster and more dangerous.)
2. DNS. account.blob.core.windows.net resolves to a public front-end VIP — unless a private endpoint exists, in which case the privatelink.blob.core.windows.net zone overrides it with a private IP. That DNS override is Private Link.
3. Front-end layer. Terminates TLS, enforces minimum TLS version, evaluates the account firewall (IP rules, VNet rules, private endpoint, publicNetworkAccess), validates the token, and asks RBAC whether this principal has the data action at a scope covering this blob.
4. Partition layer. Routes to the partition server owning this blob's key range. The partition key is account/container/blobname, lexicographically ranged, split and merged automatically by load.
5. Stage blocks. The SDK splits the file and issues parallel Put Block calls. Each is written to the stream layer — an append-only distributed filesystem — and acknowledged only once durable on three replicas within the stamp.
6. Commit. Put Block List names the blocks in order; the partition server updates the index atomically. Readers see nothing, then the whole blob. There is no half-written state.
7. Async geo-replication if the account is GRS/GZRS, lagging by the last-sync-time gap.

The bonus point: uncommitted blocks are billed and expire after about a week ⚠️ verify current window. That's where "capacity used but no blobs visible" comes from.

Both, at different levels — and naming the scope is the answer they're testing.

• Per storage account: ingress/egress bandwidth and total request rate. This is the ceiling that bites first, and it's why the account is a throughput boundary rather than a folder.
• Per blob: a single blob has its own throughput limit far below the account's. One hot file cannot absorb the account's bandwidth, no matter how much headroom the account has.
• Per partition range: because partitioning is lexicographic on the blob name, a monotonic prefix (timestamps, sequential IDs) concentrates load on one partition server. This caps you well below the account limit and looks like throttling for no reason.
• Per subscription, per region: number of storage accounts. A soft limit, raisable by support — and the reason sharding across accounts needs planning rather than improvisation.

Fixes in order: fix key distribution, parallelise across blobs rather than within one, shard across accounts, move to Premium block blob if the problem is request rate and latency rather than bandwidth, and put a CDN in front for public reads.

Actual numbers vary by region, redundancy, and account type ⚠️ verify against current Azure docs — and saying that is a better answer than quoting a figure confidently.

They trade storage price against transaction price, retrieval cost, and latency, in that order:

What moving costs you:

• Early deletion. Delete, overwrite, or re-tier before the minimum retention elapses and you're billed pro-rata for the remainder. A lifecycle rule that archives at 30 days and deletes at 90 pays this on every single object.
• Rehydration. An archived blob cannot be read at all. You rehydrate in place (hours; standard or high priority) or copy it to an online tier. Any synchronous request path that could hit Archive is broken by design.
• The transaction inversion. Tiering data that's still read regularly increases the bill.

Also worth saying: tier is per blob, redundancy is per account. People conflate them because in S3 the storage class covers both. Wanting geo-redundancy for one dataset and not another means two accounts, not two storage classes.

Six settings and one habit.

Settings: allowSharedKeyAccess = false, allowBlobPublicAccess = false, publicNetworkAccess = Disabled with a private endpoint, minimumTlsVersion = TLS1_2, supportsHttpsTrafficOnly = true, and encryption with a customer-managed key in Key Vault where the regulator asks (platform-managed otherwise, since data is always encrypted at rest either way).

Habit: managed identity + a data-plane role assignment at container scope. The consumer gets a system- or user-assigned identity; you assign Storage Blob Data Reader or Data Contributor scoped to the container it needs, not the account, not the subscription. No credential exists to leak or rotate.

If you must hand out a link, use a user-delegation SAS — signed with a key obtained from Entra ID rather than the account key, bounded by the signing identity's permissions, short expiry, revocable by removing the role assignment. Never a service SAS signed with the account key without a stored access policy, because the only way to revoke that is rotating the key, which breaks everything at once.

Enforce it with Azure Policy at management-group scope so it isn't a convention someone forgets.

Control plane is ARM at management.azure.com — creating and configuring the account: SKU, firewall, redundancy, encryption, keys. Roles: Owner, Contributor, Storage Account Contributor, Reader. Logged in the activity log.

Data plane is <account>.blob.core.windows.net — reading, writing, listing, tiering, and leasing blobs. Roles: Storage Blob Data Reader / Contributor / Owner, plus Storage Blob Delegator for minting user-delegation SAS. Logged in storage diagnostic logs, which are off by default.

The classic mistake: an engineer who is Owner opens the container in the portal, sees the blobs, and concludes Owner grants data access. It doesn't. The portal defaults to authenticating with the account key, which Owner is allowed to read via the control plane. Switch the portal's authentication method to "Microsoft Entra user account" and the same Owner gets AuthorizationPermissionMismatch.

The follow-up worth volunteering: a Contributor is effectively a data owner as long as shared-key access is enabled, because they can list the keys. Setting allowSharedKeyAccess = false is what actually separates the two planes, and that's the real argument for the setting.

The practical version: azurerm_storage_container is a data-plane operation. A pipeline identity with Contributor but no blob data role creates the account fine and then fails on the container — one of the most common Terraform-on-Azure failures there is.

Replacement of a storage account means destroy and recreate — every blob gone. The properties that force it:

• name — including a case or prefix change. And because names are globally unique, the new one may already be taken.
• is_hns_enabled — flipping ADLS Gen2 on or off. This is the big one: the hierarchical namespace is set at creation and is irreversible.
• location, always.
• account_kind on some transitions, and creation-only flags like nfsv3_enabled and is_sftp_enabled.

What it costs: all data, every SAS and URL anyone holds, every role assignment scoped to the account or its containers, the private endpoint, the diagnostic settings, and any downstream service with the endpoint hard-coded.

Mitigation: lifecycle { prevent_destroy = true } on the account resource, so a replacement plan becomes an error rather than an outage; read the plan output for # forces replacement before every apply; and — the underlying lesson — make the HNS decision deliberately at creation, because it is the one Azure storage decision you genuinely cannot undo.

Hypothesis first: storage throttling. Azure Storage signals overload with 503 ServerBusy (and sometimes 500 OperationTimedOut), not the 429 you see from ARM. Every SDK retries those with exponential backoff by default, which is exactly why it surfaces as latency rather than errors — the application never sees the failure, just the delay.

Diagnose:

1. Transactions metric split by ResponseType, filtered to ServerBusyError. That confirms or kills the hypothesis in about thirty seconds.
2. Compare SuccessE2ELatency with SuccessServerLatency. A large gap means the delay is in your network or client, not the service.
3. Check whether the traffic is concentrated: is one blob being hammered (per-blob ceiling), or is the key space monotonic (per-partition ceiling), or is the whole account near its bandwidth or request ceiling?
4. Check whether something else shares the account — an analytics job's 3 a.m. burst throttling the API tier is the classic.
5. StorageBlobLogs in Log Analytics for the per-operation picture, assuming diagnostics were on beforehand. If they weren't, turn them on now and note that for the postmortem.

Fix, in order: fix key distribution if it's a hot partition; parallelise across blobs rather than within one; split workloads onto separate accounts (the real fix at scale); consider Premium block blob if it's a latency-and-request-rate problem rather than bandwidth; add a CDN for read-heavy public content. And check nobody has replaced the SDK's backoff with a tight retry loop, which turns a throttle into an outage.

Accounts. Multiple, not one — 50 TB/day will approach account bandwidth ceilings. Shard by source system or by date range, with the sharding key chosen up front. All accounts in EU regions (westeurope / northeurope — note that Azure's region pairing determines where GRS replicates to, so verify the pair stays in the EU; this is exactly the kind of thing to check rather than assume ⚠️). Enable the hierarchical namespace — this is a data lake, and directory rename plus ACLs matter.

Ingestion. Write to a raw container, partitioned by a path that spreads the key space rather than concentrating it: prefix with a hash or a source ID before the date, not a bare timestamp. Batch small events into large files — parquet, ideally — because ten million tiny blobs cost more in transactions than the storage itself and analytics engines read them far slower.

Processing. Event Grid on BlobCreated → queue → Databricks or Data Factory, with idempotent handlers keyed on blob name plus ETag, since Event Grid is at-least-once. Medallion layout: raw → curated → published.

Retention and cost. A lifecycle policy: Cool at 30 days, Archive at ~365, delete at 2555 days (seven years) — with the archive threshold set well past the minimum retention period so early-deletion charges don't apply. Expire old versions and snapshots at 90 days. Enable last-access-time tracking so the rules act on real usage rather than modification date.

Compliance. Immutable storage with a time-based retention policy on the container if the seven years is a regulatory WORM requirement rather than just a policy. Note that a locked policy will block deletion for anyone, including an Owner, and will make terraform destroy fail — which is the point.

Security. allowSharedKeyAccess = false, publicNetworkAccess = Disabled, private endpoints on both blob and dfs sub-resources, managed identities with container-scoped roles per consumer (ingestion Contributor on raw only; BI Reader on published only), CMK in Key Vault if the regulator asks. Azure Policy at management-group scope enforcing all of it.

Reliability. GZRS for curated and published; LRS or ZRS is defensible for raw if it's regenerable from source, and at 50 TB/day that difference is a large number. Soft delete and versioning on everywhere, with expiry rules.

Observability. Diagnostic settings to Log Analytics from day one, alerting on ServerBusyError, availability, and capacity growth rate.

First, establish what actually changed. terraform plan against current state, plus the activity log filtered to the resource group for the deployment window. Distinguish three cases, because the rollback differs completely:

1. Configuration changed (firewall, redundancy, blob service settings). Low blast radius. Revert the commit, re-apply. These are in-place updates.
2. A resource was replaced or destroyed. High blast radius. If a storage account was replaced, the data is gone and the recovery is soft delete / versioning / point-in-time restore / Azure Backup — all of which had to be enabled before the incident. This is why prevent_destroy = true belongs on the account.
3. The apply is stuck mid-run, holding a blob lease on the state file. az storage blob lease break on the state blob, then verify state integrity against a prior version of the state blob — which is why versioning on the state account is not optional.

The Azure-specific traps to name:

• CanNotDelete resource locks make an apply or destroy fail with what looks like a permissions error. Locks inherit from parent scopes, so a lock on the subscription fails an apply in a resource group where you can see no lock at all. az lock list before assuming RBAC broke.
• Soft delete means a destroyed container's name is still taken. Destroy-then-recreate in dev can fail with a name conflict against something invisible in the portal.
• Immutability policies refuse deletion by design and no role overrides them.

Complete mode. az deployment group create --mode Complete deletes every resource in the resource group that is not in the template. Point it at a resource group containing a storage account you forgot to include and the account — and every blob in it — is gone, with no confirmation beyond the one you skipped. The discipline: always run az deployment group what-if with the same --mode you intend to deploy, because what-if shows the deletions. And never use complete mode against a shared resource group.

Detect. Four mechanisms, and a mature setup runs the first three continuously:

1. Scheduled terraform plan -detailed-exitcode in CI, nightly. Exit code 2 means drift; fail the job and alert. This is the primary control.
2. Azure Policy compliance state — catches violations of the baseline including on resources Terraform doesn't manage, which plan cannot see at all.
3. Activity log for who and when: az monitor activity-log list --resource-id <account-id> --start-time .... Change Analysis gives the same picture with a friendlier diff.
4. Defender for Cloud recommendations, if the change weakened the security posture.

Remediate. Decide whether the manual change was correct:

• It was correct (someone legitimately needed to allow an IP): port it into the module, PR it, apply. The code becomes the truth again and the emergency fix is now documented.
• It wasn't: re-apply and Terraform reverts it. Say plainly that this is why drift detection must be continuous — reverting a two-week-old change is far riskier than reverting a one-day-old one.
• It created a resource Terraform doesn't know about: terraform import or an import block brings it under management rather than leaving an orphan that the next complete-mode deploy will delete.

Never edit the state file by hand.

Prevent the recurrence. The honest answer is that the process failed, not the person: during an incident, a portal change is the fastest path and always will be. So make the sanctioned fast path a break-glass PR with an auto-merge, add a CanNotDelete lock on production, enforce the baseline with Azure Policy in Deny mode so the dangerous version of the change is impossible, and run the drift job nightly so the window between change and detection is hours rather than weeks.