OpenStack DC Network

L3 Underlay, GENEVE Overlay, and Kubernetes Integration

0.1 Overview

This book documents the design and implementation of a complete datacenter network architecture for Canonical’s OpenStack deployment. The architecture covers:

1. L3 Underlay: Pure L3 BGP/ECMP routing with Independent A/B Fabrics
2. GENEVE Overlay: OVN/OVS overlay networks for OpenStack workloads
3. Kubernetes Integration: OVN-Kubernetes using the same GENEVE overlay

The network design provides a unified, scalable foundation for both OpenStack VMs and Kubernetes pods using a single overlay network.

Note: For definitions of technical terms used throughout this book, see the Glossary.

0.2 Key Principles

• Pure L3 Underlay (L3/L4 for ECMP): BGP routing (L3) and ECMP load balancing (L3/L4 5-tuple hashing) - no EVPN/VXLAN at fabric layer
• Host-Based TEPs: GENEVE encapsulation at hypervisors, not at switches
• Dual ToRs per Rack: Redundant uplinks in unified L3 Clos fabric with excellent ECMP path diversity
• Mesh to Leaf-Spine Evolution: Start with mesh topology (5-6 racks), evolve to leaf-spine (7+ racks)
• Operational Simplicity: ~50 config lines per switch vs 300+ for EVPN

0.3 Architecture Summary

The fabric layer provides simple, scalable L3 routing while the overlay layer (OVN/OVS) handles all virtualization complexity at the hosts where it belongs.

┌─────────────────────────────────────────────────────────┐
│                    GENEVE OVERLAY                       │
│                   ------------------                    │
│  (Hosts/Hypervisors - TEP Endpoints)                    │
│  • GENEVE encapsulation/decapsulation                   │
│  • Random UDP src port per-flow (enables underlay ECMP) │
│  • OVN control plane (TEP registration, VM learning)    │
│  • Pure L3 multipath (2 × 100G NICs per server)         │
└─────────────────────────────────────────────────────────┘
                          │
                          │ IP/UDP packets
                          │
┌─────────────────────────────────────────────────────────┐
│                    L3/L4 UNDERLAY                       │
│                   ------------------                    │
│  (ToR/Spine Switches - Pure L3 Routers)                 │
│  • BGP routing (L3 - route advertisement)               │
│  • ECMP load balancing (L3/L4 - 5-tuple hashing)        │
│  • Pure L3 forwarding (no overlay awareness)            │
└─────────────────────────────────────────────────────────┘

0.4 Quick Navigation

• Foundations: Understanding L3 routing trends, network underlay, and overlays
  • L3 & Routing Trends
  • Network Underlay
  • Network Overlay for VMs
  • Network Overlay for Kubernetes
• Architecture & Implementation: Complete network design from principles to deployment
  • Network Architecture Overview
  • Network Design & IP Addressing
  • BGP & Routing Configuration
  • Packet Flows & Load Balancing
  • Hardware & Future Evolution
  • Storage Networking
• Operations: Day-to-day operations, troubleshooting, and maintenance
  • Operations & Maintenance
  • Quick Reference
• Configuration Examples: Ready-to-use configs for FRR, OVN, and network setup
  • Configuration Examples

0.5 Target Audience

This documentation is designed for: - Network engineers implementing modern datacenter networks for OpenStack - Platform engineers deploying Canonical’s OpenStack with OVN/OVS - DevOps teams managing OpenStack networking - Anyone building scalable, vendor-neutral network infrastructure for cloud platforms

0.6 Terminology

Throughout this book, we use standard networking terminology. For definitions of terms like TEP, ECMP, BGP, OVN, and others, see the Glossary.

0.7 References and Standards

This book references several IETF RFCs and industry standards: - RFC 8926: GENEVE (Generic Network Virtualization Encapsulation) - RFC 3021: /31 Point-to-Point Links - RFC 7432: BGP MPLS-Based Ethernet VPN (EVPN - not used in our design)

For complete references, see the References bibliography.