Networking Without Wires

Modern infrastructure rarely connects machines with physical cables alone. Cloud providers, data centers, and container platforms build virtual networks—logical topologies that run on top of a physical underlay. The overlay gives operators the illusion of a private, flexible network while the underlay just moves packets from host to host.

The Underlay vs. the Overlay

The underlay is the physical network: switches, routers, and links that carry real Ethernet frames and IP packets. The overlay is a logical network constructed by tunneling packets across the underlay. Overlay packets are encapsulated inside underlay packets, so the physical network only sees outer headers and never inspects the inner addressing.

This separation buys enormous flexibility. An overlay can assign any IP scheme, isolate tenants completely, and move a workload's address to a different physical host without the physical network knowing or caring.

VXLAN

VXLAN (Virtual Extensible LAN) is the dominant overlay encapsulation. It wraps a full Layer 2 Ethernet frame inside a UDP packet with a 24-bit VNI (VXLAN Network Identifier), supporting roughly 16 million isolated segments—far beyond the 4,094 VLANs of 802.1Q. Physical routers see plain UDP; the ends of the tunnel (VTEPs, VXLAN Tunnel Endpoints) encapsulate and decapsulate.

VXLAN solves three problems at once: scale (millions of segments), layer-2 extension across layer-3 boundaries, and tenant isolation. It is the foundation of most cloud and data-center overlay designs.

SDN: Separating Control from Data

Software-Defined Networking (SDN) splits the network into a control plane (decides where traffic goes) and a data plane (forwards it). A central controller programs forwarding rules into the data-plane devices. This centralization makes networks programmable: policy changes propagate automatically, and the network becomes an API-driven resource rather than a stack of per-device CLI configurations.

OpenFlow was the classic SDN protocol; modern SDN appears as cloud VPCs, Kubernetes network policy, and service mesh—all of which program forwarding behavior in software rather than on physical switches.

Network Function Virtualization

NFV moves network functions—firewalls, load balancers, NAT gateways, deep-packet inspection—from dedicated hardware appliances into software running on commodity servers or containers. An NFV firewall is just a VNF (virtual network function) that can be scaled, versioned, and rolled out like any other service. Combined with SDN, NFV lets a provider build entire networks out of software.

Why It Matters in Practice

Understanding overlays explains everyday cloud behavior:

  • Two VMs with 10.0.0.5 in different VPCs are truly isolated even though the addresses collide.
  • A "same subnet" in Kubernetes may span nodes across a data center, stitched by an overlay.
  • Encapsulation overhead reduces usable MTU; overlay networks often require adjusting MTU to avoid fragmentation.
  • The physical distance between two "adjacent" virtual hosts may be thousands of kilometers.

For anyone building or debugging distributed systems, the mental model is essential: the addresses you see are often virtual, and the path a packet takes is determined by software, not by the physical layout.

Overlay Limits and Trade-offs

Overlays are powerful but not free. Encapsulation adds header overhead, which reduces the effective MTU and can force fragmentation unless it is tuned correctly. One virtual hop may hide several physical hops, so latency and failure domains become harder to reason about from the virtual topology alone. Troubleshooting also changes character: you often need visibility into both the overlay (what the application sees) and the underlay (what actually carries the packets). Tools like traceroute measure the physical path, while overlay-aware instrumentation is required to see the logical one. Understanding these limits is what separates an operator who can debug a cloud network from one who can only describe it.