VXLAN Mastery · Lesson 02A · Deep dive

Ingress replication: the source makes every copy

In Lesson 2 you saw that BUM traffic must reach every VTEP in a VNI. This page zooms into ingress (head-end) replication: how the flood list works, how BUM copies move, which headers matter, what it costs, and how to configure and debug it.

📘 Builds on Lesson 2⏱ ~35 minutes🖱 Animated packet walk🧠 Quiz and checklist
Progress is saved in this browser
1Starter⏱ 4 min

Ingress replication: the source does all the copying

Ingress replication is also called head-end replication (HER). When a leaf receives a BUM frame (broadcast, unknown unicast, multicast) from a local host, that same leaf makes one separate VXLAN unicast packet for every other VTEP in the VNI. The underlay never replicates anything. It forwards the resulting unicast packets; underlay ECMP may put different replicas on different uplinks.

📬 Plain-English analogy

You want to send the same flyer to 9 colleagues in other offices. Instead of asking the post office to copy it, you photocopy it yourself and mail 9 separate envelopes, each addressed to a different office. The post office only delivers envelopes. It never makes copies.

🧩 In VXLAN terms

One broadcast arrives at Leaf 1. Leaf 1 wraps it 9 times: same inner frame, different outer destination IP (one per remote VTEP). The spine routes each packet by its outer IP like any other unicast traffic.

The shape of the traffic

1 broadcast in → N − 1 packets out
Leaf 1makes every copy to 10.0.0.2to 10.0.0.3to 10.0.0.4to 10.0.0.5 …
Remember: ingress = “where the frame enters the fabric”. For this VNI and BUM event, the ingress leaf creates the remote VXLAN copies; the routed underlay forwards them as unicast.
2Starter⏱ 5 min

The flood list: who gets a copy?

In this replication design, each participating leaf maintains a remote replication list for the VNI: the IP addresses of the remote VTEPs that also have that VNI. A BUM frame in VNI 10100 is copied only to the VTEPs on that VNI’s list, not to every VTEP in the fabric.

Build a flood list

Choose a VNI and a source leaf

VNI membership: Leaf 1 = 10100, 20200 · Leaf 2 = 10100 · Leaf 3 = 10100, 20200, 30300 · Leaf 4 = 10100, 30300

Static list

You configure relevant remote VTEP IPs for each VNI. This avoids a BGP EVPN dependency but is easy to get wrong as the fabric grows.

EVPN Type 3 (automatic)

Each VTEP advertises an Inclusive Multicast Ethernet Tag route (Type 3) for each VNI, with the tunnel type “ingress replication”. Every leaf builds the list from the routes it receives. No typing.

3Intermediate⏱ 8 min

Packet-by-packet: one ARP broadcast across four leaves

Host A (behind Leaf 1) ARPs for Host B. All four leaves share VNI 10100. Press Next step and watch every packet. Colored chips on the packets tell you what they are, and the list below the diagram says whether each one is BUM.

Follow one broadcast through ingress replication

Ethernet frame on a host linkVXLAN BUM copy (ingress replication)VXLAN unicast
Routed underlay SpineIP routing only (no multicast) Leaf 1VTEP 10.0.0.1Leaf 2VTEP 10.0.0.2Leaf 3VTEP 10.0.0.3Leaf 4VTEP 10.0.0.4Host A00:AA…:01Host B00:BB…:02Host C00:CC…:03Host D00:DD…:04

Packets in this step

Spot the pattern: the same broadcast crosses Leaf 1’s uplink three times, once per remote VTEP. The spine performs no BUM replication. It routes each already-created unicast VXLAN packet by its outer destination IP.
4Intermediate⏱ 5 min

Inside the copies: what is the same, what is different?

Here are the three packets Leaf 1 sends for one ARP request, field by field. The key field that identifies each remote VTEP is the outer destination IP. Other derived header fields can also differ.

FieldCopy 1 (to Leaf 2)Copy 2 (to Leaf 3)Copy 3 (to Leaf 4)
Outer Ethernet dst (next hop = spine)0C:50:50:50:50:500C:50:50:50:50:500C:50:50:50:50:50
Outer Ethernet src (Leaf 1)0C:01:01:01:01:010C:01:01:01:01:010C:01:01:01:01:01
Outer IP src10.0.0.110.0.0.110.0.0.1
Outer IP dst10.0.0.210.0.0.310.0.0.4
UDP src → dst52814 → 478952814 → 478952814 → 4789
VNI101001010010100
Inner Ethernet dst (broadcast)FF:FF:FF:FF:FF:FFFF:FF:FF:FF:FF:FFFF:FF:FF:FF:FF:FF
Inner Ethernet src (Host A)00:AA:AA:AA:AA:0100:AA:AA:AA:AA:0100:AA:AA:AA:AA:01
PayloadARP requestARP requestARP request

The amber cells highlight the outer destination IP, the essential change in this simplified example. Each copy also requires a valid IPv4 header checksum; next-hop MACs can differ across ECMP paths, and implementations may choose different UDP source ports. The inner frame and VNI remain the same.

Why the outer MAC is the same here

This example sends every copy through one uplink to the same next-hop spine, giving them the same outer Ethernet MACs. With ECMP, replicas may use different physical next hops and therefore different outer MACs. Routed devices rewrite the outer Ethernet header at each hop.

How receivers learn

A remote leaf decapsulates and sees: inner source MAC 00:AA…:01 arrived from outer source IP 10.0.0.1. In this flood-and-learn example it records “Host A is behind VTEP 10.0.0.1”, allowing a subsequent known-unicast response. With EVPN, the remote MAC may instead be installed from a Type 2 route.

Split horizon reminder: the receiving leaf delivers the frame to local ports only. It never re-replicates it to other VTEPs, because Leaf 1 already sent each of them their own copy.
5Mastery⏱ 5 min

The cost: uplink bandwidth grows with the number of VTEPs

Ingress replication trades simplicity for bandwidth on the source leaf’s uplinks. For a fully populated head-end replication list, every extra remote VTEP adds another copy for each BUM frame; policy and optimization can alter actual replication.

Replication load calculator

Change the numbers
Copies the source leaf sends per second
Estimated VXLAN frame load, excluding FCS (ingress replication)
Estimated source frame load, excluding FCS (underlay multicast)

Uplink load as the VNI grows (same BUM rate and frame size)

Why it is usually acceptable: BUM traffic is a small slice of total traffic when ARP suppression, storm control and sensible VNI sizes are in place. Modern leaf uplinks are 100 Gbps and faster. The cost shows up in very large, flat VNIs with chatty hosts.
6Mastery⏱ 4 min

How ingress replication looks in configuration

Every vendor needs two things: (1) tell the leaf to use unicast replication for the VNI, and (2) get the peer VTEP list, either by typing it or from EVPN.

    Illustrative only: these snippets show the replication lines, not a complete working configuration. Syntax and defaults differ across software versions, so check your vendor’s documentation.
    7Mastery⏱ 6 min

    Strengths, limits and troubleshooting

    ✅ Strengths

    • Underlay needs only unicast IP routing
    • No PIM, RPs or multicast groups to design
    • Easy to reason about: one packet, one destination
    • Can cross routed IP or WAN infrastructure when reachability, MTU, security policies and UDP 4789 are supported; VXLAN alone does not encrypt traffic
    • Default choice in most EVPN fabrics

    ⚠️ Limits

    • Source uplink load grows with N − 1
    • Switch ASIC must replicate many copies per frame
    • Static lists are error prone and tedious
    • Very large VNIs with heavy BUM can saturate uplinks
    Ways to cut BUM cost: ARP/ND suppression, IGMP snooping, storm control (rate limiting), smaller VNIs, and avoiding unknown-unicast flooding by learning MACs through EVPN.

    What’s wrong? Pick a symptom

    Troubleshooting helper
    Click a symptom to see the most likely cause.
    ✓Check yourself⏱ 5 min

    Test what you learned

    Eight questions, each with an explanation.

    Mastery checklist

    Tick each one when you can explain it in your own words.

    Cheat sheet

    • The ingress leaf makes N − 1 unicast copies
    • One flood list per VNI (static or EVPN Type 3)
    • The outer destination IP identifies each copy; checksums and next-hop MACs may also differ
    • Receivers deliver locally only (split horizon)
    • In flood-and-learn mode, remote VTEPs can learn inner source MAC → outer source VTEP IP
    • Underlay needs no multicast

    Next

    Compare with the underlay multicast companion page, where the source sends just one packet and the network copies it. Then continue to Lesson 03 (EVPN control plane).