17 Why Not VXLAN/EVPN

17.1 Overview

This appendix explains a fundamental design choice: where to terminate the overlay network—at the switches (ToR-to-ToR) or at the hosts (Host-to-Host).

The traditional approach used VXLAN/EVPN with VTEPs at ToR switches. We chose GENEVE with TEPs at hypervisors. Here’s why.

17.2 What Is an Overlay Network?

An overlay network is a virtual network built on top of the physical network (underlay). It lets you create isolated tenant networks, move VMs between physical servers, and provide L2/L3 connectivity that’s independent of the physical topology.

The key question: Where does the overlay start and end?

17.3 Two Approaches: Where to Terminate the Overlay

17.3.1 Approach 1: Switch-Centric Overlay (ToR-to-ToR)

VXLAN/EVPN at the ToR switches:

     ┌──────────────┐                              ┌──────────────┐
     │  ToR (VTEP)  │◄─────VXLAN Tunnel───────────►│  ToR (VTEP)  │
     │  Encap/Decap │   IP Routing (L3 Fabric )    │  Encap/Decap │
     └──────┬───────┘                              └───────┬──────┘
            │                                              │
            │                                              │
            │                                              │
            │ L2 (VLAN trunk)                              │ L2 (VLAN trunk)
            │                                              │
     ┌──────▼─────────┐                            ┌───────▼────────┐
     │  Hypervisor-A  │                            │  Hypervisor-B  │
     │  ┌──────────┐  │                            │  ┌──────────┐  │
     │  │   VM-A   │  │                            │  │   VM-B   │  │
     │  └──────────┘  │                            │  └──────────┘  │
     └────────────────┘                            └────────────────┘
     
     Tunnel: ToR-to-ToR (VXLAN)
     ToR-to-Host: L2 (VLAN)

How it works: - VMs send simple Ethernet frames over VLANs - ToR switch (VTEP) encapsulates into VXLAN - Fabric routes VXLAN packets (UDP) between ToR switches - EVPN (BGP-based control plane) tells ToRs where VMs are located

Why this was popular: Switches had specialized hardware for VXLAN. Hosts were weak and couldn’t handle overlay processing without performance impact.

17.3.2 Approach 2: Host-Centric Overlay (Host-to-Host) — Our Choice

GENEVE/OVN at the Hypervisors:

     ┌──────────────┐                              ┌──────────────┐
     │     ToR      │◄──────IP Routing────────────►│     ToR      │
     │  (L3 Router) │   no overlay awareness       │  (L3 Router) │
     └──────┬───────┘    IP Routing (L3 Fabric)    └───────┬──────┘                
            │                                              │
            │                                              |
            │                                              │
            │ L3 (/31 P2P routed)                          │ L3 (/31 P2P routed)
            │ (BGP/ECMP)                                   │ (BGP/ECMP)
            │                                              │
     ┌──────▼─────────────┐                        ┌───────▼────────────┐
     │  Hypervisor-A      │◄────GENEVE Tunnel─────►│  Hypervisor-B      │
     │  (OVS/TEP)         │                        │  (OVS/TEP)         │
     │  Encap/Decap       │                        │  Encap/Decap       │
     │  ┌──────────┐      │                        │  ┌──────────┐      │
     │  │   VM-A   │      │                        │  │   VM-B   │      │
     │  │ (OVN net)│      │                        │  │ (OVN net)│      │
     │  └──────────┘      │                        │  └──────────┘      │
     └────────────────────┘                        └────────────────────┘
     
     Tunnel: Host-to-Host (GENEVE)
     ToR-to-Host: L3 everywhere

How it works: - VMs connect to OVN logical networks - Hypervisor’s OVS (Open vSwitch) encapsulates into GENEVE - Fabric routes GENEVE packets (UDP) between host loopbacks - OVN control plane manages VM locations (not EVPN) - ToRs are pure L3 routers—no overlay awareness

17.4 Why the Industry Shifted from ToR-to-ToR to Host-to-Host

Historical context: When VXLAN was developed (~2012), hosts were weak. Switches had specialized ASICs for VXLAN processing, so putting VTEPs at ToRs made sense—offload the work to the switches.

Modern reality: Hosts are now powerful, and NICs have hardware offload:

More CPU cores: Modern servers have 32-64+ cores; overlay processing is trivial
Smart NICs with hardware offload: ConnectX-6 DX, Intel E810 have GENEVE hardware offload—zero CPU overhead for encapsulation/decapsulation
DPUs emerging: Data Processing Units can handle entire networking stack in hardware

The shift: Overlay work moved from switches (scarce, expensive) to hosts (abundant, commodity).

17.5 Why We Chose GENEVE Over VXLAN

Both are UDP-based tunneling protocols, but:

GENEVE is more extensible: Variable-length options field allows carrying metadata (security group IDs, QoS tags, etc.)
OpenStack default: OVN uses GENEVE natively; switching to VXLAN would require reconfiguration
Modern NICs support both: ConnectX-6 DX offloads both VXLAN and GENEVE, so hardware support isn’t a limitation
TEP terminology: GENEVE uses “TEP” (Tunnel Endpoint), VXLAN uses “VTEP”—same concept, cleaner naming

Result: OVN/OVS with GENEVE creates thousands of virtual networks at the hypervisor, all hardware-offloaded by smart NICs. The underlay stays pure L3.

17.6 What Is EVPN (And Why We Don’t Need It)

EVPN (Ethernet VPN) is a BGP-based control plane for VXLAN that distributes MAC/IP reachability information. It tells VTEPs where endpoints are so they don’t have to flood traffic.

We don’t need it because: - OVN is already our overlay control plane - OVN’s Southbound DB tracks where VMs are (which hypervisor/TEP) - Running EVPN would create a second overlay control plane—duplication and conflicts

17.7 What Is EVPN-MH (And Why We Avoid It)

EVPN Multihoming (EVPN-MH) provides L2 active-active dual-homing across two ToRs without proprietary MLAG/vPC.

DF (Designated Forwarder) election: When a server is dual-homed to two VTEPs, EVPN picks one switch as the DF for BUM (Broadcast/Unknown-unicast/Multicast) traffic to prevent loops. This state can shift during failures or upgrades.

Our approach: We use routed host uplinks (/31 point-to-point) with L3 ECMP, not L2 dual-homing. No DF election, no BUM complexity, no L2 state.

17.8 Why We’re Not Using VXLAN/EVPN in the Fabric

OVN is already our overlay control plane: EVPN would duplicate this
Switches stay simple: Just BGP/ECMP/BFD—no VNI management, no endpoint learning, no DF election
Simpler switches, lower cost: Pure L3 switches like Tomahawk ASICs provide high bandwidth at lower cost. We don’t need expensive switches with VXLAN/EVPN features.
Less operational state: No tenant VNI mappings to troubleshoot
No L2 dual-homing complexity: L3 ECMP is simpler than EVPN-MH
Vendor-neutral: SONiC/FRR BGP works everywhere; EVPN has vendor quirks
OpenStack alignment: Neutron with OVN expects host-based overlay
Host CPU impact mitigated: Modern NICs (ConnectX-6 DX) hardware-offload OVS and GENEVE overlay functions—zero CPU overhead

17.9 What We May Lose (Fair Assessment)

There are some potential trade-offs with our host-based approach:

Lesser complexity at hosts: Fabric-based EVPN means hosts don’t run OVS/GENEVE. However, we’re running OVN/OVS anyway because OpenStack requires it, so this isn’t really “additional” complexity for us.
Network team control: Some organizations prefer the network team to own overlay configuration (via EVPN) rather than the platform team (via OVN). This is an organizational choice.
GENEVE maturity: VXLAN has been around longer (~2012) and is more battle-tested. GENEVE is newer (~2015), though it’s now mature and widely deployed in OpenStack/Kubernetes environments.
Standardized L2 extension: EVPN provides L2 extension across the fabric, useful for appliances requiring L2 adjacency.
Active-active multihoming: EVPN-MH provides standards-based L2 dual-homing without vendor-specific MLAG/vPC. (we don’t need MH.. but in case some device needs in the future)

17.10 When EVPN/VXLAN Would Make Sense

Non-hypervisor appliances needing LACP and L2 across dual ToRs
Network team owns overlay rather than platform/OVN team
ToR gateway offload for performance (moving TEPs back to ToRs)
No OpenStack/OVN in the stack

17.11 Comparison

Aspect	EVPN/VXLAN (ToR<>ToR)	Our Design (Host<>Host)
Overlay Terminates at	ToR switches (VTEP)	Hypervisors (TEP)
Who knows VMs	ToR, Fabric via BGP EVPN	Underlay doesn’t know VMs
Dual-homing	EVPN-MH + DF election	No L2, L3 ECMP multi-path
Fabric complexity	High (EVPN state)	Low (pure BGP)
Control plane	EVPN in fabric	OVN for overlay

17.12 Conclusion

OVN/GENEVE on hosts + simple L3 fabric is the modern approach for OpenStack.

The fabric just routes IP packets between host loopbacks. All tenant networking complexity lives in OVN (software, flexible, debuggable). The physical network stays fast, simple, and vendor-neutral.

EVPN/VXLAN made sense when switches were powerful and hosts were weak. Today, with smart NICs and abundant host CPU, the overlay moved to where it’s most flexible: the hypervisor.

17.13 References

RFC 7432: BGP MPLS-Based Ethernet VPN (EVPN): https://tools.ietf.org/html/rfc7432
RFC 7348: Virtual eXtensible Local Area Network (VXLAN): https://tools.ietf.org/html/rfc7348
RFC 8365: EVPN Multihoming: https://tools.ietf.org/html/rfc8365
RFC 8926: GENEVE (our choice): https://tools.ietf.org/html/rfc8926

--- title: "Why Not VXLAN/EVPN" --- ## Overview This appendix explains a fundamental design choice: where to terminate the overlay network—at the switches (ToR-to-ToR) or at the hosts (Host-to-Host). The traditional approach used [VXLAN](./glossary.qmd#vxlan)/[EVPN](./glossary.qmd#evpn-ethernet-vpn) with [VTEPs](./glossary.qmd#vtep-vxlan-tunnel-endpoint) at ToR switches. We chose GENEVE with TEPs at hypervisors. Here's why. ## What Is an Overlay Network? An overlay network is a virtual network built on top of the physical network (underlay). It lets you create isolated tenant networks, move VMs between physical servers, and provide L2/L3 connectivity that's independent of the physical topology. The key question: **Where does the overlay start and end?** ## Two Approaches: Where to Terminate the Overlay ### Approach 1: Switch-Centric Overlay (ToR-to-ToR) **VXLAN/EVPN at the ToR switches**: ``` ┌──────────────┐ ┌──────────────┐ │ ToR (VTEP) │◄─────VXLAN Tunnel───────────►│ ToR (VTEP) │ │ Encap/Decap │ IP Routing (L3 Fabric ) │ Encap/Decap │ └──────┬───────┘ └───────┬──────┘ │ │ │ │ │ │ │ L2 (VLAN trunk) │ L2 (VLAN trunk) │ │ ┌──────▼─────────┐ ┌───────▼────────┐ │ Hypervisor-A │ │ Hypervisor-B │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ │ VM-A │ │ │ │ VM-B │ │ │ └──────────┘ │ │ └──────────┘ │ └────────────────┘ └────────────────┘ Tunnel: ToR-to-ToR (VXLAN) ToR-to-Host: L2 (VLAN) ``` **How it works**: - VMs send simple Ethernet frames over VLANs - ToR switch (VTEP) encapsulates into VXLAN - Fabric routes VXLAN packets (UDP) between ToR switches - EVPN (BGP-based control plane) tells ToRs where VMs are located **Why this was popular**: Switches had specialized hardware for VXLAN. Hosts were weak and couldn't handle overlay processing without performance impact. ### Approach 2: Host-Centric Overlay (Host-to-Host) — Our Choice **GENEVE/OVN at the Hypervisors**: ``` ┌──────────────┐ ┌──────────────┐ │ ToR │◄──────IP Routing────────────►│ ToR │ │ (L3 Router) │ no overlay awareness │ (L3 Router) │ └──────┬───────┘ IP Routing (L3 Fabric) └───────┬──────┘ │ │ │ | │ │ │ L3 (/31 P2P routed) │ L3 (/31 P2P routed) │ (BGP/ECMP) │ (BGP/ECMP) │ │ ┌──────▼─────────────┐ ┌───────▼────────────┐ │ Hypervisor-A │◄────GENEVE Tunnel─────►│ Hypervisor-B │ │ (OVS/TEP) │ │ (OVS/TEP) │ │ Encap/Decap │ │ Encap/Decap │ │ ┌──────────┐ │ │ ┌──────────┐ │ │ │ VM-A │ │ │ │ VM-B │ │ │ │ (OVN net)│ │ │ │ (OVN net)│ │ │ └──────────┘ │ │ └──────────┘ │ └────────────────────┘ └────────────────────┘ Tunnel: Host-to-Host (GENEVE) ToR-to-Host: L3 everywhere ``` **How it works**: - VMs connect to OVN logical networks - Hypervisor's OVS (Open vSwitch) encapsulates into GENEVE - Fabric routes GENEVE packets (UDP) between host loopbacks - OVN control plane manages VM locations (not EVPN) - ToRs are pure L3 routers—no overlay awareness ## Why the Industry Shifted from ToR-to-ToR to Host-to-Host **Historical context**: When VXLAN was developed (~2012), hosts were weak. Switches had specialized ASICs for VXLAN processing, so putting VTEPs at ToRs made sense—offload the work to the switches. **Modern reality**: Hosts are now powerful, and NICs have hardware offload: - **More CPU cores**: Modern servers have 32-64+ cores; overlay processing is trivial - **Smart NICs with hardware offload**: ConnectX-6 DX, Intel E810 have GENEVE hardware offload—zero CPU overhead for encapsulation/decapsulation - **DPUs emerging**: Data Processing Units can handle entire networking stack in hardware **The shift**: Overlay work moved from switches (scarce, expensive) to hosts (abundant, commodity). ## Why We Chose GENEVE Over VXLAN Both are UDP-based tunneling protocols, but: - **GENEVE is more extensible**: Variable-length options field allows carrying metadata (security group IDs, QoS tags, etc.) - **OpenStack default**: OVN uses GENEVE natively; switching to VXLAN would require reconfiguration - **Modern NICs support both**: ConnectX-6 DX offloads both VXLAN and GENEVE, so hardware support isn't a limitation - **TEP terminology**: GENEVE uses "TEP" (Tunnel Endpoint), VXLAN uses "VTEP"—same concept, cleaner naming **Result**: OVN/OVS with GENEVE creates thousands of virtual networks at the hypervisor, all hardware-offloaded by smart NICs. The underlay stays pure L3. ## What Is EVPN (And Why We Don't Need It) **EVPN** (Ethernet VPN) is a BGP-based control plane for VXLAN that distributes MAC/IP reachability information. It tells VTEPs where endpoints are so they don't have to flood traffic. **We don't need it because**: - OVN is already our overlay control plane - OVN's Southbound DB tracks where VMs are (which hypervisor/TEP) - Running EVPN would create a second overlay control plane—duplication and conflicts ## What Is EVPN-MH (And Why We Avoid It) **EVPN Multihoming (EVPN-MH)** provides L2 active-active dual-homing across two ToRs without proprietary MLAG/vPC. **DF (Designated Forwarder) election**: When a server is dual-homed to two VTEPs, EVPN picks one switch as the DF for BUM (Broadcast/Unknown-unicast/Multicast) traffic to prevent loops. This state can shift during failures or upgrades. **Our approach**: We use routed host uplinks (/31 point-to-point) with L3 ECMP, not L2 dual-homing. No DF election, no BUM complexity, no L2 state. ## Why We're Not Using VXLAN/EVPN in the Fabric - **OVN is already our overlay control plane**: EVPN would duplicate this - **Switches stay simple**: Just BGP/ECMP/BFD—no VNI management, no endpoint learning, no DF election - **Simpler switches, lower cost**: Pure L3 switches like Tomahawk ASICs provide high bandwidth at lower cost. We don't need expensive switches with VXLAN/EVPN features. - **Less operational state**: No tenant VNI mappings to troubleshoot - **No L2 dual-homing complexity**: L3 ECMP is simpler than EVPN-MH - **Vendor-neutral**: SONiC/FRR BGP works everywhere; EVPN has vendor quirks - **OpenStack alignment**: Neutron with OVN expects host-based overlay - **Host CPU impact mitigated**: Modern NICs (ConnectX-6 DX) hardware-offload OVS and GENEVE overlay functions—zero CPU overhead ## What We May Lose (Fair Assessment) There are some potential trade-offs with our host-based approach: - **Lesser complexity at hosts**: Fabric-based EVPN means hosts don't run OVS/GENEVE. However, we're running OVN/OVS anyway because OpenStack requires it, so this isn't really "additional" complexity for us. - **Network team control**: Some organizations prefer the network team to own overlay configuration (via EVPN) rather than the platform team (via OVN). This is an organizational choice. - **GENEVE maturity**: VXLAN has been around longer (~2012) and is more battle-tested. GENEVE is newer (~2015), though it's now mature and widely deployed in OpenStack/Kubernetes environments. - **Standardized L2 extension**: EVPN provides L2 extension across the fabric, useful for appliances requiring L2 adjacency. - **Active-active multihoming**: EVPN-MH provides standards-based L2 dual-homing without vendor-specific MLAG/vPC. (we don't need MH.. but in case some device needs in the future) ## When EVPN/VXLAN Would Make Sense - **Non-hypervisor appliances** needing LACP and L2 across dual ToRs - **Network team owns overlay** rather than platform/OVN team - **ToR gateway offload** for performance (moving TEPs back to ToRs) - **No OpenStack/OVN** in the stack ## Comparison | Aspect | EVPN/VXLAN (ToR<>ToR) | Our Design (Host<>Host) | |--------|-------------------------|---------------------------| | Overlay Terminates at | ToR switches (VTEP) | Hypervisors (TEP) | | Who knows VMs | ToR, Fabric via BGP EVPN | Underlay doesn't know VMs | | Dual-homing | EVPN-MH + DF election | No L2, L3 ECMP multi-path | | Fabric complexity | High (EVPN state) | Low (pure BGP) | | Control plane | EVPN in fabric | OVN for overlay | ## Conclusion **OVN/GENEVE on hosts + simple L3 fabric** is the modern approach for OpenStack. The fabric just routes IP packets between host loopbacks. All tenant networking complexity lives in OVN (software, flexible, debuggable). The physical network stays fast, simple, and vendor-neutral. EVPN/VXLAN made sense when switches were powerful and hosts were weak. Today, with smart NICs and abundant host CPU, the overlay moved to where it's most flexible: the hypervisor. ## References - RFC 7432: BGP MPLS-Based Ethernet VPN (EVPN): https://tools.ietf.org/html/rfc7432 - RFC 7348: Virtual eXtensible Local Area Network (VXLAN): https://tools.ietf.org/html/rfc7348 - RFC 8365: EVPN Multihoming: https://tools.ietf.org/html/rfc8365 - RFC 8926: GENEVE (our choice): https://tools.ietf.org/html/rfc8926