The Keys to the Kingdom#

Application secrets, API keys, database passwords, encryption keys, tokens and certificates, are the credentials that unlock everything your systems protect. Secrets management is the practice of storing, distributing, rotating and auditing them so that when one leaks, and one eventually will, the damage is contained instead of catastrophic. A well-managed secret is short-lived, narrowly scoped and revocable; a badly managed one is a decade-old admin password sitting in a Git repository.

The Secret Sprawl Problem#

Secrets multiply faster than teams track them. A single service might hold a database password, a cache password, two or three third-party API keys, a signing key, SMTP credentials and a cloud access key, each duplicated across development, staging and production. Without discipline they scatter into .env files, CI variables, container image layers, wiki pages, tickets, chat messages and shell history. Every copy is an independent chance to leak, and the attack surface grows silently.

Store Secrets in a Dedicated System#

The foundation is a purpose-built secrets manager: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or Kubernetes secrets integrated with one of these through an external provider. What they give you beyond "a place to put strings":

  • Encryption at rest and fine-grained access control per secret.
  • Audit logging of every read, so you know who accessed what and when.
  • Dynamic secrets: the manager generates a short-lived database credential on demand and revokes it automatically, so there is no long-lived password to steal.
  • Leasing and revocation: access expires on a schedule, and you can revoke it immediately during an incident.

The application fetches secrets at runtime, authenticating with an identity it already has (a cloud role, a Kubernetes service account, a Vault auth method), rather than reading them from a file baked into the deployment.

Fetch at Runtime, Not Build Time#

A common anti-pattern is loading secrets from a .env file committed alongside the code, or baking them into the image at build time. Fetch them at process start instead. With AWS Secrets Manager and boto3:

import json
import boto3
from botocore.exceptions import ClientError

_client = boto3.client("secretsmanager")

def get_secret(name: str) -> dict:
    try:
        resp = _client.get_secret_value(SecretId=name)   # this call is audit-logged
    except ClientError as e:
        # Fail closed: never fall back to a hardcoded default credential
        raise RuntimeError(f"could not load secret {name}: {e.response['Error']['Code']}")
    return json.loads(resp["SecretString"])

db = get_secret("prod/billing/db")     # {"username": "...", "password": "...", "host": "..."}

The important choices: the workload authenticates as its attached role (no key in the code to fetch the secret in the first place), the fetch is logged, and a failure raises rather than silently using a default. If you cache the value in memory, cache it briefly so rotation takes effect without a redeploy.

Never Hardcode#

The cardinal rule: no secret in source code, ever. Git history is permanent, and a key committed even briefly, then "removed" in a later commit, is still there and must be treated as compromised. Two layers catch leaks:

# Block secrets before they are committed, and scan history for ones already there
gitleaks protect --staged        # pre-commit: refuse to commit a detected secret
gitleaks detect --source .       # scan the full history
trufflehog git file://.          # verifies some findings against live services

Wire gitleaks protect into a pre-commit hook and run gitleaks detect in CI. The response to a real leak is fixed order:

  1. Rotate the credential first. Rewriting history does not un-leak a key that automated scrapers may already have pulled from a public repo.
  2. Then investigate what the key could access and whether it was used.
  3. Then clean the repository if needed.

Rotate-then-investigate, never delete-and-hope.

Rotation#

Secrets should expire. Rotation replaces a credential on a schedule or on demand, and the design decisions matter more than the cadence:

  • Prefer short-lived credentials (minutes to hours) generated on demand over long-lived static ones. A stolen 15-minute credential is far less useful than a stolen permanent key.
  • Overlap validity during rotation: issue the new secret, let both work for a window, deploy, then retire the old one, so a rotation does not cause an outage.
  • Automate it. Manual rotation does not happen under pressure, and secrets nobody documented are never rotated at all.
  • Design for rotation from the start. Retrofitting it into a system that assumes a static password is painful, because code, config and running processes all cached the old value in different ways.

Least Privilege and Scoping#

Each secret should grant the minimum access needed and be scoped to one service and one environment:

  • A key that can read one bucket is vastly safer than one that can administer the whole cloud account. See cloud security.
  • Separate credentials per environment mean a staging leak cannot touch production.
  • Split by function: a service that only needs to read a queue should not hold credentials that can also delete it.

Scoping is what turns a leaked secret from a breach into a contained incident.

CI/CD and Build Systems#

Pipelines are a frequent leak point because they hold deployment credentials and run third-party code:

  • Do not print secrets in build logs; use the platform's masking, but do not rely on masking alone.
  • Never expose production secrets to builds triggered by pull requests from forks.
  • Use short-lived, environment-scoped credentials injected by the CI platform's own secret store, and prefer OIDC federation so the pipeline exchanges its identity token for temporary cloud credentials with no stored key at all.
  • Review pipeline definitions for accidental echo $SECRET and for secrets passed as command-line arguments (which appear in process listings).

The supply-chain security lesson covers hardening the pipeline more broadly.

Detection and Response#

Assume leaks will happen and watch for them:

  • Automated scanning of public repositories and paste sites for your key patterns. Providers such as GitHub and cloud vendors run their own secret scanning and will sometimes auto-revoke a key they detect in public.
  • Unusual use of a known key: calls from unexpected IPs or regions, at unusual times, or actions the key has never performed before.
  • Keys still active long after they should have rotated, and keys that have never been used (delete them).

When a leak is confirmed, the sequence is the same as above: rotate first to close the window, then use the secrets manager's audit log to see exactly what was accessed while the key was valid. That audit log is the reason a managed secrets store beats a .env file, it turns "we think this key leaked" into "here is precisely what it touched". See incident response.

Practice#

Run gitleaks detect against a repository you own and count the findings; investigate any real one by rotating it. Then take one hardcoded credential in a personal project, move it into a secrets manager (or even a local Vault dev server), fetch it at runtime as in the example above, and confirm the app still starts with no secret in the source tree. Finally, add a pre-commit hook so the next accidental key never lands.