8. Interview Questions

EC2 rents virtual machines by the second. You choose the shape (instance type), the disk image (AMI), and the network placement, and you get a server with root access in seconds.

The problem it solved wasn't "servers are hard" — it was capital risk and lead time. Before EC2 you forecast peak demand 18 months out, bought hardware for that peak, waited weeks for it, and paid for it year-round at low utilisation. EC2 turned a capital expense with a fixed ceiling into an operating expense that tracks demand.

The deeper consequence, and the one worth leading with: it made servers disposable. Once a machine can be replaced in 60 seconds, you stop repairing them individually and start treating them as interchangeable. Auto-scaling, immutable AMIs, blue/green deploys, and chaos engineering all descend from that single property.

Strong-answer marker: mention that EC2 is also the substrate under ECS, EKS, EMR, and Batch — so its vocabulary and failure modes recur across the catalogue.

An instance runs in one Availability Zone, inside a Region. It boots from an AMI (an immutable, region-scoped image), takes its shape from an instance type, and sits on a subnet in a VPC via an ENI, firewalled by a security group. It gets permissions from an instance profile wrapping an IAM role, and storage from EBS volumes (network-attached, durable, AZ-scoped) and optionally an instance store (local, ephemeral).

A launch template is the versioned recipe for all of that; an Auto Scaling group maintains N instances from that recipe across multiple subnets; a target group and load balancer put one address in front of them.

The two scoping facts that matter: AMIs and snapshots are region-scoped, while EBS volumes and instances are AZ-scoped. That's why a volume can't follow an instance to another AZ and why hard-coded AMI IDs break in a new region.

Split it three ways:

• A single instance is not highly available. It has a materially lower SLA than the region-level figure, and it will eventually be retired, rebooted, or degraded. Availability comes from the pattern — multiple instances, multiple AZs, behind a load balancer, in an ASG — not from the service.
• EBS is durable within one AZ. Every write is replicated synchronously across multiple servers in that AZ before acknowledgement. Annual failure rate is low but non-zero and varies by volume type.
• Instance store offers no durability guarantee. It survives a reboot; it does not survive a stop/start, a terminate, or a host failure.

The critical distinction: EBS is durable, not backed up. Replication protects against device failure. It does nothing about rm -rf, ransomware, a bad migration, or an AZ-level event. Only snapshots — ideally cross-region with a retention policy — do that.

Say the exact percentages only if you're certain they're current; "region-level and instance-level SLAs differ, and I'd check the current SLA page" is a better answer than a confidently wrong number.

Frame it as a spectrum of how much of the stack you're responsible for: EC2 → ECS on EC2 → Fargate → Lambda, with control and operational burden both decreasing left to right.

Choose EC2 when:

• You need a specific OS, kernel, or driver stack — legacy Windows, GPU drivers, a custom kernel.
• The workload is long-running and steadily busy; above the point where a Lambda would be warm anyway, committed EC2 is dramatically cheaper.
• You need nameable hardware: GPUs, high memory, local NVMe, bare metal.
• You're lifting and shifting something that assumes a real filesystem, a fixed IP, or a resident daemon.
• Licensing is tied to cores or sockets (Dedicated Hosts).

Choose something else when:

• The work is event-driven, short, and spiky → Lambda. An always-on instance serving occasional events is pure waste.
• It's containerised and you don't want hosts to patch → Fargate. You pay a premium per vCPU-hour and delete an entire operational surface.
• You'd be self-managing a database, cache, or queue → RDS / ElastiCache / SQS. Higher hourly cost, far lower annual cost in engineer-hours.

Strong-answer marker: name the hidden cost of EC2 — every instance is an OS you patch, an access surface you secure, alarms you write, and an entry in your CVE process. None of that shows up on the bill.

Instance hours are the obvious one and often not the biggest:

• Instance time — per second while running (60s minimum on Linux; some OSes and Marketplace AMIs bill per hour). Stopped instances cost no compute.
• EBS volumes — per GB-month provisioned, not used. A 500 GB volume that's 3% full bills 500 GB, and keeps billing while the instance is stopped.
• Provisioned IOPS/throughput above the included baseline.
• Snapshots — per GB-month of changed blocks, accumulating forever without a lifecycle policy.
• Data transfer out to the internet, and cross-AZ traffic in both directions.
• Public IPv4 addresses — per address-hour, in use and idle.
• NAT gateway — per hour plus per GB processed. Frequently the biggest surprise.
• Load balancer hours plus capacity units; detailed monitoring per instance.

Data transfer in is free, and intra-AZ traffic over private IPs is free.

Strong-answer marker: name the traps — idle instances, over-provisioning, orphaned volumes and snapshots, NAT data processing, cross-AZ chatter, and t-family unlimited-mode surplus credits.

1. Authorisation and validation at the regional control-plane endpoint — SigV4 auth, IAM check (including iam:PassRole if an instance profile is attached), then validation: does the AMI exist in this region, is the type offered in this AZ, does the subnet exist?
2. Quota check against your vCPU quota for that family group in that region. Fail here and no capacity was ever sought — this is VcpuLimitExceeded, a support-ticket fix, not a retry.
3. Placement — the placement service finds a physical host in the target AZ with free capacity of the right shape, honouring placement group, tenancy, and any capacity reservation. No room → InsufficientInstanceCapacity, which is a different problem: AWS is out of that type in that AZ right now, and retrying elsewhere usually works.
4. Resource attachment — the Nitro controller provisions the slice: an ENI in your subnet bound to the Nitro networking card with your SG rules loaded into it, the root EBS volume created from the AMI snapshot and attached over the network, instance-store NVMe presented.
5. Boot — the Nitro hypervisor allocates vCPU and memory; firmware hands off to the guest kernel.
6. cloud-init — fetches metadata from IMDS at 169.254.169.254, injects the SSH key, sets hostname, grows the root filesystem, executes user data as root.
7. Status checks begin. running means the VM booted — not that your application is up.

Strong-answer marker: distinguish VcpuLimitExceeded (your quota) from InsufficientInstanceCapacity (AWS's capacity), because the responses are completely different. Second marker: root volumes are hydrated lazily from S3, so first-read of each block can be slow.

EC2 doesn't scale — you do, by running more instances. Vertically (bigger type) needs a stop/start, so it's disruptive and capped by the largest size in the family; right for a single-writer database, wrong as a growth strategy. Horizontally via an ASG is the real answer.

The ASG control loop: compare desired to actual, launch/terminate to close the gap, replace instances failing health checks, and adjust desired capacity per scaling policy — target tracking by default.

Four real ceilings:

And the thing people underestimate: scaling takes minutes, not seconds — alarm evaluation, launch, boot, configure, health checks, target registration. If traffic can double in 90 seconds, reactive scaling won't save you; you need headroom, scheduled scaling, or a queue.

Strong-answer marker: scale on a leading metric like ALBRequestCountPerTarget rather than CPU, and pay for detailed (1-minute) monitoring — a 5-minute metric means a 5-minute-late decision.

Two independent axes: how you pay and what capacity you're promised.

The headline: Savings Plans and RIs are billing constructs, not capacity guarantees. If the AZ is out of c6i.4xlarge, your launch fails whether or not you hold a Savings Plan. Only a Capacity Reservation (or a Zonal RI) reserves the machine. You can combine them: a Capacity Reservation for the guarantee, a Savings Plan for the discount.

Spot isn't cheap On-Demand — it's a different reliability contract. Great for stateless, interruptible, horizontally scalable work: CI runners, batch, big-data workers, fault-tolerant web tiers in an ASG with a mixed instances policy. Wrong for anything with local state or a hard deadline that can't absorb eviction. And handle the interruption notice via EventBridge or IMDS — drain, checkpoint, deregister, exit.

Strong-answer marker: Spot vCPU quota is separate from On-Demand vCPU quota, so a mixed instances policy can be blocked by a quota you didn't know you had. Second marker: commit to your floor, not your average — unused commitment is pure waste.

Access: delete SSH. Use SSM Session Manager — no inbound port (the agent dials out), IAM decides who may connect to which instances (tag-scoped), every session in CloudTrail and optionally recorded to S3, no private key to distribute or leak. The cost is a dependency on the SSM agent and on reachability, which in private subnets means three interface endpoints (ssm, ssmmessages, ec2messages) or a NAT.

Credentials: an instance profile wrapping one role per service. Never access keys on disk, in user data, or baked into an AMI. Scope resources specifically — s3:GetObject on one bucket prefix, not *. Add condition keys (aws:SourceVpce, aws:RequestedRegion, ec2:ResourceTag/*). Use permission boundaries so role-creators can't escalate.

IMDSv2, required, hop limit 1. The threat is concrete: an SSRF bug in your app becomes a path to your role's credentials at 169.254.169.254. IMDSv2 requires a PUT for a session token first and the hop limit stops containers/proxies relaying the request.

Network: instances in private subnets, ALB in public. Security groups referencing other security groups rather than CIDRs. No 0.0.0.0/0 except the LB's public ports.

Data: EBS encryption on by default at the account level so it can't be forgotten; TLS at the ALB via ACM; secrets from Secrets Manager or Parameter Store fetched at startup, authorised by the instance role.

Patching: prefer immutable — rebuild the AMI, roll the ASG via instance refresh. Keep Patch Manager as the break-glass path. Inspector for continuous CVE scanning.

Guardrails: Config rules (restricted-ssh, ec2-imdsv2-check, encrypted-volumes), GuardDuty for behavioural detection, and SCPs to make the worst actions impossible rather than merely detected.

Strong-answer marker: mention publicly-shared EBS snapshots and AMIs as a real, recurring breach vector — they contain whole filesystems, credentials included.

Don't quote specific default numbers — they change and vary by account. Know the shapes:

• Running On-Demand vCPUs per instance-family group, per region. Counted in vCPUs, not instances, with separate quotas for standard families (A/C/D/H/I/M/R/T/Z) and for G, P, Inf, Trn, X, DL, HPC.
• Running Spot vCPUs per family group — a completely separate quota.
• EBS storage per volume type per region, snapshots per region, Elastic IPs per region (low by default — the classic early-account wall), instances per ASG, launch template versions.

Soft vs hard: all of the above are soft (raisable via Service Quotas). What isn't a quota is instance type availability in an AZ — that's InsufficientInstanceCapacity, a momentary hard limit no support ticket fixes.

Raising them: Service Quotas console or API (request-service-quota-increase). Review takes hours to days, longer for large GPU asks. Quotas are per region per account — a new region starts at defaults, which is a classic expansion-day outage. Request headroom before the launch event, and alarm on quota utilisation where supported.

Strong-answer marker: distinguish VcpuLimitExceeded (your quota — ticket) from InsufficientInstanceCapacity (AWS capacity — diversify AZ/type). Different problem, different fix.

On a standalone aws_instance:

That user_data row causes real incidents: you change the bootstrap script, apply reports success, and nothing changed on any running instance.

Rolling a new AMI safely — the ASG pattern: changing the launch template creates a new version; existing instances are untouched until an instance refresh rolls through — launch a replacement, wait for warmup, wait for the ELB health check, terminate an old one, repeat, respecting min_healthy_percentage throughout. Set auto_rollback = true so a failed refresh reverts itself.

This is precisely why the ASG pattern beats managing aws_instance directly: every deploy produces genuinely new instances, so user data actually runs, and rollback is "point at the previous launch template version."

Also worth saying:

• ignore_changes = [desired_capacity] on the ASG — otherwise a routine apply resets your fleet to the declared count, silently scaling prod down mid-traffic.
• create_before_destroy on security groups and target groups, or ordinary changes deadlock on dependencies.
• git revert is not automatically safe. Terraform's reverse of "created a resource" is "destroy that resource" — reverting the commit that added a volume deletes the volume and its data. Read the plan on a revert with the same care as a forward change.

First, get the data. Cost Explorer grouped by usage type and by service, month over month — that immediately tells you whether it's compute, storage, transfer, or something adjacent. Then group by tag (Service, Environment) to localise it. Check Cost Anomaly Detection findings for the date the step change started.

Then work the candidates, cheapest check first:

Prevention as part of the answer: enforced cost-allocation tags, Budgets per team, Cost Anomaly Detection, a scheduled orphan-resource sweep, and calendar reminders before Savings Plan expiry.

Strong-answer marker: naming Savings Plan / RI expiry. It's the classic "nothing changed but the bill" cause and most candidates never mention it.

The architecture:

Route 53 alias → (CloudFront + WAF if it's public web) → ALB with an ACM certificate in public subnets across 3 AZs → target group with a real /health → ASG across 3 private subnets → launch template (looked-up AMI, gp3 encrypted, IMDSv2 required, instance profile, no key pair) → RDS Multi-AZ and ElastiCache behind SG-to-SG rules → S3 via a gateway endpoint for artifacts.

Surviving the AZ failure — and this is the part to lead with: static stability. Provision across three AZs with enough headroom that losing one requires launching nothing. Three AZs each at ~50% utilisation absorbs an AZ loss instantly. The alternative — "we'll auto-scale into the survivors" — depends on the control plane and on spare capacity at exactly the moment both are most contested, and everyone else in the AZ is scrambling for the same instances. That headroom costs money and it is cheaper than the outage.

Also: health_check_type = "ELB", ASG spanning all three subnets, and no instance-local state — sessions in ElastiCache or a cookie, uploads to S3, logs shipped off-box immediately.

Handling 10x — layered, because a single mechanism won't do it:

1. Headroom for the first slice, absorbed with no action.
2. Target tracking on ALBRequestCountPerTarget, not CPU — it's proportional to the load you're distributing and reacts sooner. Detailed 1-minute monitoring.
3. Fast launches: a golden AMI with everything baked, so boot-to-healthy is tens of seconds, not minutes. Warm pools for pre-initialised instances if the ramp is still too slow.
4. Scheduled or predictive scaling if the spike is predictable (a sale, a batch window).
5. A queue in front of anything asynchronous, to convert a spike into a backlog rather than errors.
6. Load shedding and rate limiting at the ALB/WAF as the honest last resort — degrade deliberately rather than collapsing.

Trade-offs I'd state out loud:

• Three AZs with headroom costs roughly 50% more compute than a tight single-AZ fleet. That's the price of the availability, stated plainly.
• Cross-AZ traffic is billed per GB both ways — so spreading for availability has an ongoing cost, and I'd keep genuinely hot internal paths intra-AZ where correctness allows.
• Spot for a fraction of the fleet via a mixed instances policy cuts cost substantially, at the price of building interruption handling — and remember Spot vCPU quota is separate.
• vCPU quota must cover peak plus a replacement AZ, requested well in advance.
• Reactive scaling cannot handle a 90-second doubling. If that's the requirement, the answer is headroom and queues, not a cleverer scaling policy. Say so rather than promising it.

Strong-answer marker: volunteering the cost of your own design before being asked.

Stop the bleeding first, diagnose second.

1. Halt the refresh. aws autoscaling cancel-instance-refresh. Already-replaced instances stay replaced, but no more get cycled — the failure stops spreading.
2. Assess. describe-instance-refreshes for StatusReason; target group health; are the errors from new instances only (HTTPCode_Target_5XX correlated with the new AMI) or both?
3. Roll back. If the refresh is still active, rollback-instance-refresh — or auto_rollback = true should already have done it. If it completed, revert the commit and re-apply so the launch template points at the previous contents, then refresh again. Read that plan — the reverse of a create is a destroy.
4. If the bad change is application-level, not AMI-level, re-run the Ansible playbook pinned to the previous artifact version. Far faster than replacing instances.
5. Verify recovery on the service metrics — p99 latency, 5XX rate, healthy host count — not on "the apply succeeded."
6. Then diagnose properly, with a preserved instance if useful (detach it from the ASG rather than terminating, so you keep the evidence and the ASG launches a replacement).

Blast-radius reasoning, stated as the interviewer wants to hear it:

• min_healthy_percentage bounded the damage: at 90%, only a small slice was ever in flight, which is why this is a degradation and not an outage.
• The failure mode I'd worry about most is a refresh that passes health checks but is functionally broken — a shallow /health that returns 200 while the app can't reach the database. A health check that only proves the process is listening will happily certify a broken deploy.
• Anything stateful is the real blast radius. Replacement means a new root volume; local state is gone. If a schema migration ran, rolling the code back may not be safe — which is why migrations must be backwards-compatible for one release.
• Prod deletion protection and prevent_destroy on stateful resources exist so that a panicked apply can't escalate an incident into data loss.

Strong-answer marker: saying you knew the rollback command before the deploy, and that you'd deploy in a window when the people who understand the change are awake.

Finding it:

• terraform plan -detailed-exitcode — exit 2 means drift. Run this nightly in CI and alert on it; that's how you catch this in a day rather than three weeks.
• CloudTrail for the AuthorizeSecurityGroupIngress call: who, when, from where. AWS Config gives you the resource's configuration timeline, which is the friendlier view.
• CloudFormation shops use detect-stack-drift for the same purpose.

Why it's urgent: the change itself may be fine. The danger is that the next unrelated apply silently reverts it — restoring a rule someone needed, at a moment nobody connects to the deploy. Drift converts a past fix into a future incident with no obvious cause.

Resolving it — decide which source of truth wins:

Preventing recurrence — this is the part that matters:

• Nightly drift detection in CI, alerting on exit code 2.
• Remove console write access in prod via SCPs or permission boundaries, so the pipeline is the only mutation path.
• Provide a legitimate fast path for emergencies — a break-glass role with time-limited elevated access that fires an alert on assumption. People click in the console because the pipeline is too slow during an incident; if you don't give them a sanctioned fast path, they'll keep making an unsanctioned one. Then require a follow-up PR within 24 hours.

Strong-answer marker: treating the human factor as a design problem rather than a discipline problem. Blaming the engineer is the weak answer.