Container & Kubernetes Networking

A Network Per Pod#

Containers changed how networks are used inside a single machine. Each container needs its own IP address and ports, containers on one host must be isolated from each other, and in an orchestrator like Kubernetes, thousands of short-lived pods spread across many nodes must reach each other as if they were on one flat network. All of it is built from a few Linux primitives plus a layer of automation, and knowing those primitives is what makes debugging tractable.

The Building Blocks#

• A network namespace gives a process its own interfaces, routing table, and firewall rules. Each container (or each Kubernetes pod) gets one.
• A veth pair is a virtual cable: one end sits inside the container's namespace as eth0, the other end sits on the host.
• A bridge (such as docker0 or cni0) connects the host ends together, acting as a virtual switch.
• iptables/nftables rules on the host provide NAT for outbound traffic and the port publishing for inbound traffic. Some platforms use eBPF programs instead.

You can see these on any Docker host: ip link shows the veth... interfaces and the bridge, and sudo iptables -t nat -L -n shows the NAT rules Docker created.

Docker Networking#

Docker's network drivers:

Two behaviours cause most Docker networking confusion:

# 1. Name resolution only works on user-defined networks
docker network create app-net
docker run -d --name db --network app-net -e POSTGRES_PASSWORD=example postgres:16
docker run --rm --network app-net postgres:16 pg_isready -h db     # "db" resolves
docker run --rm postgres:16 pg_isready -h db                        # fails: default bridge has no DNS

# 2. Published ports bind to all interfaces unless you say otherwise
docker run -d -p 8080:80 nginx              # reachable from the whole network
docker run -d -p 127.0.0.1:8080:80 nginx    # reachable only from this host

User-defined networks get Docker's embedded DNS server (at 127.0.0.11 inside the container), so containers can find each other by name. The default bridge network does not. On the second point, Docker writes its own iptables rules for published ports, and those rules are evaluated before rules managed by host firewall front-ends such as ufw. A port published with -p 8080:80 can therefore be reachable from the internet even though ufw says it is blocked. Bind to 127.0.0.1 when only a local reverse proxy needs access.

The Kubernetes Networking Model#

Kubernetes sets out a few rules and leaves the implementation to a plugin:

1. Every pod gets its own IP address, unique across the cluster.
2. Pods can reach every other pod without NAT, on any node.
3. Agents on a node can reach all pods on that node.
4. Containers in the same pod share one network namespace and talk over localhost.

The implementation is a CNI (Container Network Interface) plugin. It allocates pod IPs (usually from a per-node range), creates the veth pairs, and makes pod IPs routable between nodes. Plugins differ in how they do that last part: Flannel uses a VXLAN overlay, Calico can route pod prefixes natively with BGP or use an overlay, and Cilium implements the datapath, policy, and observability in eBPF. Overlays work on any underlay but add encapsulation overhead and reduce MTU; native routing is faster but needs the underlying network (or cloud route tables) to know the pod prefixes. The mechanics of overlays are covered in virtual networks and overlays.

Services: Stable Names for Moving Targets#

Pod IPs change every time a pod is rescheduled. A Service gives a set of pods a stable virtual IP (the ClusterIP) and DNS name, and load-balances across the pods whose labels match its selector:

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: shop
spec:
  selector:
    app: api            # must match the pod template's labels exactly
  ports:
    - port: 80          # the port clients use on the ClusterIP
      targetPort: 8080  # the port the container actually listens on

Service types build on each other:

• ClusterIP (default): a virtual IP reachable only inside the cluster.
• NodePort: also opens a port (30000–32767 by default) on every node's IP.
• LoadBalancer: also asks the cloud provider to create an external load balancer that forwards to those node ports or directly to pods.
• ExternalName: just a DNS CNAME to an outside hostname.
• Headless (clusterIP: None): no virtual IP; DNS returns the individual pod IPs, which StatefulSets and client-side load balancers use.

The ClusterIP does not belong to any pod or physical interface. kube-proxy (using iptables or IPVS) or an eBPF datapath intercepts traffic to it on every node and rewrites the destination to one of the ready pods listed in the Service's EndpointSlices. That is why you often cannot ping a ClusterIP (only the service's ports are translated) but can curl it. For client IPs to survive the hop to the pod, a LoadBalancer or NodePort Service needs externalTrafficPolicy: Local, which also avoids an extra hop between nodes.

DNS Inside the Cluster#

CoreDNS resolves Service names. A pod's /etc/resolv.conf typically looks like this:

nameserver 10.96.0.10
search shop.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

So from a pod in the shop namespace, api resolves through the search list to api.shop.svc.cluster.local, and a Service in another namespace is reached as api.payments. The ndots:5 option means any name with fewer than five dots is tried against every search domain first, so resolving api.stripe.com can trigger several failed lookups inside the cluster before the real one. For latency-sensitive external calls, use a fully qualified name with a trailing dot (api.stripe.com.).

Ingress and the Gateway API#

External HTTP traffic usually enters through an Ingress resource, or the newer and more expressive Gateway API, implemented by a controller such as NGINX, Traefik, or Envoy. This layer does TLS termination, routing by host and path, and often rate limiting and authentication: the cluster's equivalent of the reverse proxy and API gateway.

Network Policy#

By default every pod can reach every other pod, so one compromised pod can probe the whole cluster. NetworkPolicy objects restrict this. The standard pattern is default-deny, then explicit allows:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: shop
spec:
  podSelector: {}               # every pod in the namespace
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-from-frontend
  namespace: shop
spec:
  podSelector:
    matchLabels: { app: api }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels: { app: frontend }
      ports:
        - { protocol: TCP, port: 8080 }

Three things to know. First, a default-deny egress policy also blocks DNS, so every application breaks with name-resolution errors until you add the allow-dns policy. Second, policies are only enforced if the CNI supports them: Flannel on its own accepts NetworkPolicy objects and silently ignores them. Third, in the allow-dns policy the namespaceSelector and podSelector sit in the same list item, meaning "pods labelled kube-dns in kube-system"; as two separate items they would mean "anything in kube-system, or any kube-dns pod anywhere". Because egress from the frontend is also denied, it needs its own egress rule towards the API pods. For the wider security picture, see container and Kubernetes security.

Debugging "Service A Cannot Reach Service B"#

Work through these in order; most problems are found in the first three steps.

# 1. Start a throwaway pod with network tools in the caller's namespace
kubectl run -it --rm netdebug -n shop --image=nicolaka/netshoot -- bash
#    then inside it:
nslookup api                 # DNS: does the name resolve to a ClusterIP?
curl -sv http://api/healthz  # Service: does the ClusterIP forward to a pod?

# 2. Does the Service have ready endpoints?
kubectl get endpointslices -n shop -l kubernetes.io/service-name=api

# 3. Compare the selector with the pods' labels
kubectl get svc api -n shop -o jsonpath='{.spec.selector}'
kubectl get pods -n shop --show-labels

# 4. Is a policy blocking it?
kubectl get networkpolicy -n shop

An empty EndpointSlice list means no pod is both matched by the selector and Ready: a label typo, a failing readiness probe, or pods in a different namespace. A connection that is refused usually means a wrong targetPort, or an application listening on 127.0.0.1 instead of 0.0.0.0, so it is unreachable from outside its own pod. A timeout points to a NetworkPolicy or a node-level problem.

The selector check is easy to automate with the official Python client:

from kubernetes import client, config

config.load_kube_config()          # or config.load_incluster_config() inside a pod
v1 = client.CoreV1Api()

def explain_service(name, namespace):
    svc = v1.read_namespaced_service(name, namespace)
    selector = svc.spec.selector or {}
    if not selector:
        # an empty label_selector would list every pod in the namespace
        if svc.spec.type == "ExternalName":
            print(f"service {name}: DNS alias for {svc.spec.external_name}, no pods")
        else:
            print(f"service {name}: no selector, so its endpoints are managed manually "
                  "(EndpointSlices/Endpoints you create), not chosen by pod labels")
        return
    label_query = ",".join(f"{k}={v}" for k, v in selector.items())
    pods = v1.list_namespaced_pod(namespace, label_selector=label_query).items
    print(f"service {name}: selector {selector}, {len(pods)} matching pod(s)")
    for pod in pods:
        ready = any(c.type == "Ready" and c.status == "True"
                    for c in (pod.status.conditions or []))
        print(f"  {pod.metadata.name:<32} ip={pod.status.pod_ip}  ready={ready}")

explain_service("api", "shop")

A Service without a selector is valid (it is how you point a Service at a database outside the cluster, for example), but no pod labels are involved, so check its EndpointSlices directly with kubectl get endpointslices -l kubernetes.io/service-name=<name> instead. Zero matching pods means the selector is wrong; matching pods with ready=False means the readiness probe is failing and the fault lies in the application, not the network.