Secrets Management

9 min read

In a Nutshell

Secrets are the credentials your system needs to function but must never expose: database passwords, API keys, encryption keys, TLS certificates, OAuth client secrets, tokens. Secrets management is the discipline of storing, distributing, rotating, and auditing these safely — so they aren't hard-coded in source, committed to Git, printed in logs, or left readable to anyone who shouldn't have them. It's a deceptively common source of breaches: leaked secrets in public repositories, config files, and CI logs cause a huge share of real-world incidents. A dedicated secrets-management system replaces the ad-hoc "just put it in an env var / config file" approach with a secure, centralized, auditable one.

2D minimalistic diagram showing a central secrets manager (vault) that securely stores credentials (DB password, API key, TLS cert) encrypted; applications request secrets at runtime with authenticated, least-privilege access, and every access is logged, contrasted with an insecure hard-coded secret sitting in plaintext in a source file marked with a warning

How It Actually Works

The Anti-Patterns (What Not to Do)

❌ Hard-coded in source:   API_KEY = "sk_live_abc123"   → leaks via Git forever
❌ Committed to Git:       even if deleted later, it's in the HISTORY
❌ Plaintext config files:  readable by anyone with file access
❌ In environment variables (as the ONLY control): visible in process
   listings, crash dumps, child processes, and often logged
❌ Printed in logs:        secrets end up in log aggregation, searchable
❌ Shared over Slack/email: unencrypted, permanent, uncontrolled copies

The most notorious: a secret committed to Git lives in the history forever, even after deletion — and bots scan public GitHub for exposed keys within seconds of a push.

What a Secrets Manager Provides

Capability	Why It Matters
Centralized secure storage	One encrypted source of truth, not scattered files
Encryption at rest & in transit	Secrets never stored or sent in plaintext
Access control (least privilege)	Each service reads only the secrets it needs
Audit logging	Every access recorded — who read what, when
Rotation	Change secrets regularly, ideally automatically
Dynamic secrets	Generate short-lived, on-demand credentials
Versioning	Roll back to previous secret versions

Runtime Retrieval, Not Build-Time Embedding

The key pattern: applications fetch secrets at runtime from the secrets manager (authenticating with their own identity), rather than having secrets baked into code, images, or config:

App startup / runtime:
  1. App authenticates to the secrets manager using its workload identity
     (IAM role, Kubernetes service account, mTLS) — NOT a secret-to-get-secrets
  2. Secrets manager checks the app's permissions (least privilege)
  3. Returns only the secrets that app is allowed to read
  4. App holds them in memory, uses them, never writes them to disk/logs
  5. Every request is logged for audit

This means the container image and repo contain no secrets — they're injected only to the running, authenticated workload.

Secret Rotation

Secrets should change periodically so a leaked-but-undetected secret has a limited useful life, and a known leak can be remediated fast:

Static secret leaked 6 months ago, unrotated → attacker has had access
for 6 months and you may never know.

With rotation (e.g., every 30 days) → the window is bounded; and
dynamic secrets (generated per-session, expire in minutes) shrink it to
near-zero — even a captured credential is useless moments later.

Dynamic secrets (e.g., Vault generating a fresh, short-lived database credential per app instance) are the gold standard: there's no long-lived secret to steal.

Common Tools

Tool	Type
HashiCorp Vault	Full-featured secrets manager (dynamic secrets, rotation)
AWS Secrets Manager / Parameter Store	Managed cloud secrets
Google Secret Manager / Azure Key Vault	Managed cloud secrets
Kubernetes Secrets	Built-in (but base64, not encrypted by default — enable encryption/KMS)
Sealed Secrets / SOPS	Encrypt secrets safely for GitOps

Note: plain Kubernetes Secrets are only base64-encoded, not encrypted by default — a common misconception. Enable encryption-at-rest (KMS) or use an external manager.

Preventing Leaks

Secret scanning — tools (GitHub secret scanning, gitleaks, trufflehog) detect secrets in commits/PRs before they're exposed.
Pre-commit hooks — block commits containing secret-like strings.
.gitignore config/secret files; never commit them.
Rotate immediately if a secret is ever exposed — treat any leaked secret as compromised.
Never log secrets — scrub them from logs and error reports.

2D minimalistic diagram showing dynamic secrets: an application requests database access from a secrets manager, which generates a fresh, unique, short-lived credential on demand (valid for minutes), the app uses it, and it auto-expires — so there is no long-lived password to steal, contrasted with a static shared password that persists indefinitely

Seeing It in Action

Scenario: Migrating from hard-coded secrets to a managed secrets workflow.

Before (insecure, common):
  # config.py committed to Git
  DB_PASSWORD = "prod_p@ssw0rd"
  STRIPE_KEY  = "sk_live_51H..."
  → in Git history forever; readable by everyone with repo access;
    bots scan public repos and find keys in seconds.

After (secrets manager + runtime retrieval):
  1. Store secrets in AWS Secrets Manager / Vault (encrypted, access-controlled).
  2. Remove ALL secrets from code/config; repo contains only references:
       db_password = secrets.get("prod/db/password")
  3. The app authenticates via its WORKLOAD IDENTITY (IAM role / K8s service
     account) — no secret needed to fetch secrets.
  4. Least privilege: the orders-service role can read "prod/db/password"
     but NOT "prod/stripe/key" — each service sees only its own secrets.
  5. Audit: every secret access logged (who, what, when) for compliance.
  6. Rotation: DB password rotated every 30 days automatically; better,
     use DYNAMIC secrets — Vault issues a fresh short-lived DB credential
     per instance that expires in an hour.

Guardrails against re-introduction:
  - Pre-commit hook + CI secret scanning (gitleaks) block any commit that
    contains a secret-like string.
  - .gitignore for local .env files.
  - Incident runbook: any exposed secret is rotated IMMEDIATELY and treated
    as compromised.

The historical cleanup:
  - The old committed secrets are STILL in Git history → they must be
    ROTATED (changed), not just deleted, because the old values remain
    recoverable from history and may already be compromised.

Why runtime retrieval and rotation are the crux: the fundamental shift is that secrets stop living with the code and start living in a dedicated, encrypted, access-controlled, audited system that hands them to authenticated workloads at runtime. This eliminates the entire class of "secret leaked via source control / config / image" breaches, because the artifacts developers handle contain no secrets at all. Least-privilege access means a compromised service can't read every credential in the company. Audit logging turns secret access into something observable and reviewable. And rotation — especially dynamic, short-lived secrets — ensures that even a secret that does leak has a tightly bounded useful life, ideally minutes, so a captured credential is worthless before an attacker can use it. The critical, often-missed detail is that once a secret has been exposed (e.g., committed to Git), deleting it isn't enough — it must be rotated, because the exposed value persists in history and must be assumed compromised. Secrets management done right makes leaks rare, contained, and quickly recoverable rather than catastrophic and permanent.

Interview Questions

Q: Why are environment variables and config files insufficient for secrets? Hint: Config files in source control leak the secret (and it persists in Git history forever); env vars are visible in process listings, crash dumps, child processes, and are often accidentally logged. Neither provides encryption at rest, fine-grained access control, audit logging, or rotation. A dedicated secrets manager provides centralized encrypted storage, least-privilege access, auditing, and rotation that these ad-hoc methods lack.
Q: What does a secrets manager provide over storing secrets yourself? Hint: Centralized encrypted storage (at rest and in transit), least-privilege access control (each service reads only its own secrets), audit logging (who accessed what, when), rotation (change secrets regularly), dynamic/short-lived secrets (generated on demand), and versioning/rollback. It replaces scattered plaintext with one secure, auditable source of truth that hands secrets to authenticated workloads at runtime.
Q: What's the runtime-retrieval pattern and why is it more secure? Hint: Apps fetch secrets at runtime from the secrets manager by authenticating with their own workload identity (IAM role, K8s service account, mTLS) rather than embedding secrets in code, images, or config. This means repos and container images contain no secrets, so the entire class of leaks via source control/images disappears; access is authenticated, least-privilege, and audited.
Q: Why are dynamic (short-lived) secrets better than static ones? Hint: Static secrets are long-lived — if leaked and undetected, an attacker has extended access you may never notice. Dynamic secrets are generated on demand, unique per instance, and expire in minutes, so there's no long-lived credential to steal and even a captured one is useless almost immediately. They shrink the leaked-secret exposure window to near-zero and remove the shared-password problem.
Q: A secret was accidentally committed to Git and then deleted. Is that sufficient? Why or why not? Hint: No — deleting it from the current code doesn't remove it from Git history, where the value remains recoverable, and bots may have already scraped it. You must rotate (change) the secret and treat the exposed value as compromised. Also purge history if feasible, and add secret scanning/pre-commit hooks to prevent recurrence. Deletion hides it; rotation actually neutralizes it.

References

OWASP Secrets Management Cheat Sheet — comprehensive guidance
HashiCorp Vault — dynamic secrets and rotation
AWS Secrets Manager — managed secrets in the cloud

Dive Deeper

GitGuardian: State of Secrets Sprawl — the scale of leaked-secret incidents
Vault dynamic secrets tutorial — generating short-lived DB credentials
Kubernetes Secrets encryption at rest — fixing the base64 misconception