Virtual Networks & Overlays

Networking Without Wires#

Cloud providers, data centres, and container platforms rarely give a workload a "real" network. They build virtual networks: logical topologies with their own addressing and isolation, running on top of a physical network that knows nothing about them. The result is that two virtual machines can share a subnet while sitting in different buildings, and two customers can both use 10.0.0.0/16 without ever seeing each other's traffic.

Underlay and Overlay#

The underlay is the physical network: switches, routers, and links carrying ordinary IP packets between hosts. It is usually a simple, routed fabric whose only job is fast and reliable delivery between host IPs.

The overlay is built by encapsulation: a tenant's packet is wrapped inside an outer packet addressed from one host to another. The underlay routes the outer packet; the receiving host strips the outer headers and delivers the inner packet to the right VM or container. The underlay never inspects inner addresses, which is exactly what lets the overlay use any addressing scheme, isolate tenants, and move a workload to another physical host without touching the physical network.

Outer Ethernet | Outer IP  192.0.2.10 -> 192.0.2.20 | UDP dport 4789 | VXLAN VNI 100 | Inner Ethernet | Inner IP 10.100.0.1 -> 10.100.0.2 | payload
<------------------------- underlay sees this ------------------------>  <------------- tenant's original frame -------------->

VXLAN#

VXLAN (Virtual Extensible LAN) is the most widely used overlay encapsulation. It wraps a complete Layer 2 Ethernet frame in UDP (destination port 4789) with an 8-byte header carrying a 24-bit VNI (VXLAN Network Identifier). That gives about 16 million isolated segments, against the 4,094 usable IDs of 802.1Q VLANs. The devices that encapsulate and decapsulate are VTEPs (VXLAN Tunnel Endpoints), which can be hypervisors, Linux hosts, or hardware switches.

The header is simple enough to build and parse by hand, which makes it clear what a VTEP actually does:

import struct

def vxlan_header(vni: int) -> bytes:
    # 8 bytes: flags (I bit = 0x08) + 24 reserved bits, then 24-bit VNI + 8 reserved bits
    return struct.pack("!II", 0x08 << 24, vni << 8)

def parse_vxlan(payload: bytes):
    flags_word, vni_word = struct.unpack("!II", payload[:8])
    if not flags_word & (0x08 << 24):
        raise ValueError("VNI-present flag not set")
    return vni_word >> 8, payload[8:]          # (VNI, inner Ethernet frame)

inner_frame = bytes.fromhex("02000a640002" "02000a640001" "0800") + b"...inner IP packet..."
udp_payload = vxlan_header(100) + inner_frame
print(udp_payload[:8].hex(" "))
vni, frame = parse_vxlan(udp_payload)
print("VNI", vni, "| inner dst MAC", frame[:6].hex(":"), "| inner src MAC", frame[6:12].hex(":"))

08 00 00 00 00 00 64 00
VNI 100 | inner dst MAC 02:00:0a:64:00:02 | inner src MAC 02:00:0a:64:00:01

0x64 is 100, the VNI. A receiving VTEP uses the VNI to decide which virtual network (which bridge, which tenant) the inner frame belongs to. Two details make VXLAN work well on real fabrics. The UDP source port is derived from a hash of the inner packet's headers, so different inner flows get different outer source ports and the underlay's ECMP spreads them across all links. And because the outer packet is plain UDP/IP, the underlay can be an ordinary routed network with no VLANs stretched across it.

Build a Two-Host Overlay on Linux#

With two Linux machines (VMs are fine) whose underlay addresses are 192.0.2.10 and 192.0.2.20:

# Host A
sudo ip link add vxlan100 type vxlan id 100 dstport 4789 \
     local 192.0.2.10 remote 192.0.2.20 dev eth0
sudo ip addr add 10.100.0.1/24 dev vxlan100
sudo ip link set vxlan100 up

# Host B (mirror image)
sudo ip link add vxlan100 type vxlan id 100 dstport 4789 \
     local 192.0.2.20 remote 192.0.2.10 dev eth0
sudo ip addr add 10.100.0.2/24 dev vxlan100
sudo ip link set vxlan100 up

# From host A
ping -c 3 10.100.0.2

While the ping runs, watch the underlay on host B with sudo tcpdump -ni eth0 udp port 4789. You will see UDP packets between the two host IPs; Wireshark decodes port 4789 as VXLAN and shows the inner ICMP packet with its 10.100.0.x addresses. bridge fdb show dev vxlan100 shows the forwarding table: which inner MAC addresses live behind which remote VTEP.

Pitfalls this lab exposes, all of which occur in production too:

• Leaving out dstport 4789. The Linux kernel's default VXLAN port is 8472, which predates the IANA assignment. Two implementations that disagree on the port silently fail to talk. (Some CNIs, such as Flannel, use 8472 deliberately.)
• A firewall on the underlay dropping UDP 4789 between hosts. The overlay looks "down" while the underlay pings fine.
• MTU. The vxlan100 interface is created with an MTU of 1450 on a 1500-byte underlay, because VXLAN adds 50 bytes (outer IP 20, UDP 8, VXLAN 8, inner Ethernet 14). If something inside the overlay assumes 1500, large packets are dropped and you get the classic "ping works, file transfers hang" pattern. Fix it by raising the underlay MTU (data-centre fabrics commonly use jumbo frames of about 9,000 bytes) or by making sure everything in the overlay uses the lower MTU. The VPN and tunneling lesson walks through the MTU arithmetic and how to test for it.

The Control Plane: How VTEPs Learn Where MACs Live#

In the lab, each host had exactly one remote. With hundreds of VTEPs, something must tell each one which remote VTEP hosts a given MAC or IP address.

• Flood and learn: unknown and broadcast traffic is sent to every VTEP in the VNI (by underlay multicast or by replicating to a list), and VTEPs learn MAC locations from the traffic they receive, just like a physical switch. Simple, but broadcast-heavy at scale.
• BGP EVPN: VTEPs advertise their MAC and IP bindings to each other over BGP, so every VTEP knows where every address lives before traffic flows, and ARP can be answered locally. This is the standard in modern data-centre fabrics.
• Central controller: a software controller (as in cloud VPCs and OVN) programs the mappings into every hypervisor directly.

Cloud VPCs go furthest. They are not really Layer 2 networks at all: broadcast is generally not delivered, and the virtualisation layer answers ARP itself because the controller already knows which VM holds each IP. This explains why protocols that depend on broadcast or multicast, and some failover designs that move an IP by sending gratuitous ARP, do not work in a VPC without provider-specific features.

Comparing Segmentation and Encapsulation Options#

VXLAN and Geneve provide isolation but no encryption. Anyone who can capture underlay traffic can read the inner packets. Across untrusted links, run the overlay over IPsec or WireGuard, or pick a CNI that encrypts node-to-node traffic.

SDN: Separating Control From Data#

Software-Defined Networking splits the network into a control plane, which decides where traffic should go, and a data plane, which forwards packets at high speed. A central controller computes policy and programs forwarding rules into the data-plane elements through an API. OpenFlow was the early standard protocol for this. Today the same idea shows up as cloud VPC APIs, Open vSwitch programmed by OVN, Kubernetes NetworkPolicy enforced by a CNI, and service meshes. In each case, a change is an API call that propagates to every relevant host, not a CLI session on each device.

Network Function Virtualisation#

NFV moves network functions such as firewalls, load balancers, NAT gateways, and VPN concentrators off dedicated appliances and into software on commodity servers or containers. A virtual firewall can then be deployed, scaled, and versioned like any other service. Combined with SDN, which steers traffic through those functions, a provider can assemble a complete network in software, and this is exactly what happens when you create a VPC, a NAT gateway, and a load balancer from a cloud console.

Debugging Across Two Layers#

Every overlay problem has two possible homes, so check both:

1. Is the underlay healthy? Ping between host (VTEP) IPs, verify UDP 4789 or 6081 is allowed, and check the underlay MTU.
2. Is the overlay mapping correct? Check bridge fdb show, the EVPN routes, or the cloud route tables and security groups for the VNI or VPC in question.
3. Capture on the underlay with tcpdump -ni eth0 udp port 4789. If encapsulated packets leave host A but never reach host B, the problem is the physical path; if they reach B but the inner packet is never delivered, it is VTEP configuration or MTU.
4. Remember what traceroute shows. Inside the overlay, a traceroute between two VMs may show one hop even when the packet crosses several physical routers. Latency and failures follow the physical path, not the virtual diagram.

The same model underlies pod networking, covered in container and Kubernetes networking.

Practice#

Build the two-host VXLAN lab, then create a second VNI (id 200) on both hosts, move each new interface into its own network namespace (sudo ip netns add tenant2 and sudo ip link set vxlan200 netns tenant2), and give it the same 10.100.0.0/24 addresses. Confirm with tcpdump that both virtual networks share the underlay yet stay isolated, with the VNI as the only thing telling them apart. Finally, lower the underlay interface MTU to 1400 and reproduce the large-packet failure with ping -M do -s 1400 10.100.0.2.