Securing the New Operating System#

Containers and Kubernetes are the default way to run software, and they bring a distinct security model. A container is not a virtual machine: it shares the host kernel, so a container escape is a host compromise. Kubernetes adds a control plane that, misconfigured, hands an attacker the whole cluster. Securing this environment means covering four layers, the image, the runtime, the network and the orchestrator, because a weakness in any one undoes the others.

The recurring theme: the defaults are permissive for convenience, and security is something you deliberately switch on.

Image Security#

Everything a container can do starts with what is in the image:

  • Minimal base images. Start from distroless or slim bases. Fewer packages means fewer vulnerabilities and no shell for an attacker to use after a breakout.
  • Scan in CI with Trivy, Grype or Clair, and fail the build above a severity threshold. See supply-chain security.
  • Pin base images by digest, not by a mutable tag, so a tampered or updated upstream cannot silently change your build: python:3.12-slim@sha256:....
  • Sign and verify images (Sigstore/Cosign), checking the signature at admission time.
  • Never bake secrets into images. Anyone who pulls the image can extract them, and a secret added then deleted in a later layer is still present in the layer history. Fetch secrets at runtime instead (see secrets management).
  • No latest in production; it is non-deterministic and defeats rollback and reproducibility.

Runtime Hardening#

A container should run with the least privilege it needs. The single most effective change is not running as root, because a root process that escapes the container is root on the node. A hardened pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault        # filter dangerous syscalls
  containers:
    - name: api
      image: registry.example/api@sha256:abc123...
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]           # then add back only what is genuinely needed
      resources:
        limits:                   # a compromised pod cannot starve the node
          cpu: "500m"
          memory: "256Mi"

Line by line, the security value:

  • runAsNonRoot plus runAsUser means the process is unprivileged; the container refuses to start if the image would run as root, catching mistakes early.
  • allowPrivilegeEscalation: false blocks setuid binaries from gaining more rights.
  • readOnlyRootFilesystem: true stops an attacker writing a web shell or tool to disk; give writable paths as explicit emptyDir volumes.
  • capabilities: drop: ["ALL"] removes Linux capabilities like NET_ADMIN and SYS_ADMIN; most apps need none.
  • seccompProfile: RuntimeDefault restricts the syscalls available, shrinking the kernel attack surface a breakout would use.
  • Resource limits bound the damage a compromised or runaway pod can do to its neighbours.

What to forbid entirely: privileged containers, hostPID/hostNetwork, and mounting the Docker/container socket or host paths like /. Each is effectively root on the host. AppArmor profiles add another syscall/file restriction layer where available.

RBAC and the Control Plane#

Kubernetes role-based access control is the primary authorisation boundary, and it applies to workloads as well as people:

  • Most application pods need no Kubernetes API access at all. Set automountServiceAccountToken: false unless the pod genuinely calls the API.
  • Never grant cluster-admin to an application service account.
  • Audit who can create pods, get secrets, or exec into containers, these are cluster-takeover primitives. Anyone who can create a pod can usually mount a host path or run privileged and escape; anyone who can read secrets often holds the keys to everything.
  • Protect the API server: strong authentication, no anonymous access, network isolation, and audit logging enabled.

You can find over-broad access before an attacker does:

# Who can read secrets in the prod namespace?
kubectl auth can-i --list --as=system:serviceaccount:prod:default -n prod

# Find bindings that grant cluster-admin
kubectl get clusterrolebindings -o json \
  | jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .metadata.name + " -> " + (.subjects // [] | tostring)'

Network Security#

By default, every pod can reach every other pod, and often the cloud metadata endpoint too. Network policies restrict east-west traffic to only the paths services actually use. Start with default-deny per namespace, then allow specific flows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: prod
spec:
  podSelector: {}                 # every pod in the namespace
  policyTypes: ["Ingress", "Egress"]
# With no ingress/egress rules listed, everything is denied; add allow rules separately.

Then explicitly allow, say, the API to reach the database on 5432 and nothing else. This is zero trust micro-segmentation applied inside the cluster: a compromised front-end cannot reach the database, the backups or the metadata endpoint. Note that network policies require a CNI plugin that enforces them (Calico, Cilium and others); with a plugin that ignores them, the YAML applies but nothing is enforced, a dangerous false sense of safety worth verifying with a connectivity test.

Add mTLS between services (often via a service mesh) so traffic is authenticated and encrypted, not just filtered by IP.

Secrets in Kubernetes#

Native Kubernetes secrets are only base64-encoded, not encrypted, unless you enable encryption at rest for etcd. Anyone who can read the secret object, or read etcd, sees the plaintext. Practical guidance:

  • Enable etcd encryption at rest at minimum.
  • Prefer an external secrets manager mounted via the Secrets Store CSI driver or an operator, so secret material never lives in etcd.
  • Lock down get/list on secrets with RBAC; read access to secrets is frequently equivalent to full cluster privilege.

Admission Control Turns Guidance Into Gates#

All of the above is advice until something enforces it. Admission controllers reject non-compliant workloads at deploy time. Kubernetes ships Pod Security Admission with baseline and restricted levels you can enforce per namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: prod
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted

The restricted level blocks privileged pods, host namespaces, running as root and more, exactly the runtime rules above. For custom policy (require signed images, ban specific registries, mandate resource limits), use OPA Gatekeeper or Kyverno. Policy as an admission gate means a bad manifest is refused, not merely frowned upon in a wiki.

The Hardening Checklist#

  • Run as non-root; read-only root filesystem; allowPrivilegeEscalation: false; drop all capabilities.
  • No privileged containers, host namespaces, host-path or socket mounts.
  • Resource limits on every container.
  • Images: minimal base, scanned, signed, digest-pinned, no latest, no baked secrets.
  • Default-deny network policy plus explicit allows, enforced by a capable CNI; mTLS between services.
  • Least-privilege RBAC; automountServiceAccountToken: false where unused; no app-owned cluster-admin.
  • Secrets externalised or at least encrypted at rest; tight RBAC on get secrets.
  • Audit logging on; admission policy (Pod Security Admission or Gatekeeper/Kyverno) enforcing all of it.

Practice#

On a throwaway cluster (kind or minikube), deploy a pod that runs as root with a writable filesystem, then apply the restricted Pod Security Admission label to its namespace and watch the redeploy get rejected, reading the exact reason in the error. Next, apply a default-deny network policy and confirm from one pod that you can no longer reach another, then add a single allow rule and confirm the intended path works. The cluster networking side is covered in container and Kubernetes networking.