🛣 Underlay
The physical routed IP network that carries the wrapped packets. It only needs to deliver IP packets between switches, and knows nothing about tenants or VLANs.
Start from zero. In nine short chapters you will go from “what is a VLAN?” to following a real VXLAN packet hop by hop, and spotting the mistakes that break overlays in production. Every idea has a picture, a plain-English explanation and something to click.
VXLAN builds on three old ideas. If you know them already, skim this and move on. If not, this is all you need.
Data on a LAN travels in frames. A frame has a destination MAC, a source MAC and the data. A MAC address is a hardware name like 00:AA:AA:AA:AA:01. Think of it as the name on an envelope.
A switch learns which MAC lives behind which port by watching the source of every frame. Next time, it sends frames for that MAC out of just that port. It stores this in a MAC table.
A VLAN splits one physical switch into separate virtual LANs. Hosts in VLAN 10 cannot hear hosts in VLAN 20, even on the same switch. A VLAN is also called a broadcast domain.
A broadcast is a frame addressed to everyone (FF:FF:FF:FF:FF:FF). Click a host to make it shout, and see who is allowed to hear it.
VLANs are still extremely useful. The challenge appears when many separate tenant segments must be extended across racks while the physical network needs flexible Layer 3 routing.
A VLAN ID is only 12 bits. After the reserved values 0 and 4095, only 4,094 usable VLANs remain. A very large environment may need more isolated segments than that single VLAN ID space provides.
To stretch a VLAN across many racks, classic networks used Layer 2 everywhere, and Spanning Tree may need to block redundant links to prevent loops in spanning-tree-based designs. Poorly controlled broadcasts or loops can affect a large domain.
Layer 3 (IP) fabrics can use multiple equal-cost paths (ECMP), support scalable routing and offer useful failure isolation. Data centers prefer to build the physical network as routed IP.
Virtual machines want to move between racks and keep their IP and MAC. Some workloads need to remain in the same Layer 2 segment across racks, although many modern applications can use Layer 3 mobility instead.
This compares number spaces only. It is not a hardware limit. Real devices support far fewer segments than 16 million.
VXLAN does not change the frame. It wraps it inside a normal IP packet, sends it across the routed network, and unwraps it at the other side. This is called encapsulation.
| Post office | VXLAN network |
|---|---|
| Original letter (Alice → Bob) | Ethernet frame from Host A to Host B |
| Outer envelope addressed to the branch office | Outer IP/UDP/VXLAN headers addressed to the remote switch (VTEP) |
| Post office reads only the envelope | Underlay routers read only the outer IP address |
| Branch office opens the envelope | Remote VTEP decapsulates and delivers the original frame |
| Alice and Bob never see the envelope | Hosts never know VXLAN is used |
Everything in VXLAN uses these four terms. Learn them once and the rest is easy.
The physical routed IP network that carries the wrapped packets. It only needs to deliver IP packets between switches, and knows nothing about tenants or VLANs.
The virtual Layer 2 network that hosts believe they are on. It exists only as a logical layer on top of the underlay.
VXLAN Tunnel EndPoint. The device that wraps and unwraps frames, usually a leaf switch. It owns an IP address (often on a loopback) used as the tunnel endpoint.
VXLAN Network Identifier. A 24-bit number in the VXLAN header naming the overlay segment, like a VLAN ID with 16.7 million possible values.
Start with the original frame and add the four wrappers a VTEP adds. Click Add next layer, then click any layer to see its fields.
Overhead added: 0 bytes
On the wire, a VXLAN-encapsulated Ethernet frame adds 50 bytes (outer Ethernet 14 + outer IPv4 20 + UDP 8 + VXLAN 8), excluding FCS and any outer VLAN tag. When checking the underlay IP MTU for an inner IP packet, count 14 bytes of inner Ethernet + 20 outer IPv4 + 8 UDP + 8 VXLAN = 50 bytes; the outer Ethernet header is not included in IP MTU. IPv6 outer IP headers add 20 bytes to either calculation.
Host A sends a frame to Host B. Watch the packet travel step by step. Notice what changes at every hop, and what never changes. Follow the moving green frame and watch the outer headers appear at Leaf 1. Assumption: Leaf 1 already knows where Host B lives (chapter 7 explains how it learns).
Amber = changed at this step · Teal = what the current device reads to decide where to send it.
In chapter 6 Leaf 1 already knew “Host B is behind VTEP 10.0.0.2”. In real life it starts with an empty table. With plain VXLAN, switches flood and learn, just like an ordinary switch.
The VXLAN wrapper makes packets larger. If the resulting outer IP packet exceeds the path MTU, some packets may be dropped or fragmented, depending on platform settings and fragmentation behavior. This can make small tests succeed while larger traffic fails. Try it.
Fragmentation behavior is platform- and configuration-dependent. Outer IPv4 DF handling and Path MTU Discovery determine whether oversized encapsulated packets are dropped or fragmented. Do not assume every oversized packet silently disappears; check underlay MTU, ICMP Packet Too Big / fragmentation-needed messages, tunnel counters, and platform behavior.
This is an advanced preview. Beginners can skip it and take the quiz. Actual vendor syntax depends on NOS release, platform, and whether flood-and-learn or EVPN is used. The three concepts from this lesson appear in configuration as: a loopback IP (the VTEP address), a VLAN-to-VNI mapping, and a tunnel interface toward other VTEPs. Syntax differs by vendor, but the ideas are the same.
| Concept | Cisco NX-OS | Juniper Junos | Arista EOS | Cumulus Linux |
|---|---|---|---|---|
| VTEP address | interface loopback0 | lo0.0 + vtep-source-interface | Loopback0 | lo address / nve vxlan source |
| VLAN → VNI | vn-segment | vxlan vni | vxlan vlan … vni | vlan … vni |
| Tunnel interface | interface nve1 | (implicit) | interface Vxlan1 | nve vxlan |
| Remote VTEP (static) | peer-ip | EVPN (later) | vxlan flood vtep | head-end-replication |
Eight questions. Every answer comes with an explanation. This quiz runs in your browser.
Tick each one when you can explain it in your own words.
RFC 7348 — Virtual eXtensible Local Area Network (VXLAN). This lab models a simple IPv4 underlay with the standard header layout.