Introduction
This post is part of an ongoing series documenting the build-out of a physical VCF 9 home lab from scratch. Before a single VCF installer OVA is deployed, the physical network layer needs to be correct — VLANs present, MTUs consistent end-to-end, BGP uplinks reachable, and NFS accessible from the management domain. If any of these are wrong at day zero, VCF deployment will fail in ways that are difficult to diagnose after the fact.
This guide covers the complete network pre-requisite configuration applied to an Arista DCS-7050TX-64-R acting as the primary lab leaf switch, including the design decisions behind every choice, the full EOS configuration, and a thorough validation checklist. Everything here reflects a real deployment — including mistakes encountered along the way.
Note: This is not a theoretical design guide. Every command shown was run on real hardware. Where something failed during testing, it is documented here along with the fix.
Lab Hardware Overview
The physical lab consists of the following hardware. Understanding the role of each node informs every design decision that follows.
| Component | Specification | Role in Lab |
|---|---|---|
| Arista DCS-7050TX-64-R | 48x 10GbE RJ45, 4x QSFP+, EOS 4.19.10M | Primary lab leaf switch — all VLANs, BGP, SVIs |
| Cisco Catalyst 3750E-PoE-24 | 24x 1GbE PoE, IOS 15.2 | Core access switch — trunked to Arista Et48 |
| Supermicro SYS-6029TP-HTR (×2) | 2U TwinPro², 4 nodes/chassis, dual Xeon Silver 4214R, 1TB RAM total | 8× VCF compute/management nodes (Site 1 = CHx1 A-D, Site 2 = CHx2 E-H) |
| Dell PowerEdge R630 | 128GB DDR4, 4× 1GbE onboard | Management host: ESXi running Ubuntu 24.04 VM for NFS + Docker services |
| Intel NUC Skull Canyon | 2× NIC, Ubuntu Desktop | Admin jumpbox — SSH gateway, Vaultwarden, HashiCorp Vault |
| HPE 3PAR StoreServ 8000 | 12× 1.2TB SAS + 8× 480GB SAS SSD | Future vSAN drives — require sg_format 520→512 byte sector conversion |
Design Note: The R630 is not a VCF management domain host. It runs ESXi purely to host an Ubuntu Server 24.04 VM which provides NFS storage and Docker-based services (Outline wiki, Gitea, Oxidized, draw.io, HashiCorp Vault). This avoids a circular dependency — the R630 ESXi does not consume the NFS it serves.
VLAN Design & IP Addressing
VCF 9 requires a minimum of five dedicated VLANs per management domain: ESXi Management, vMotion, vSAN, NSX Host TEP, and VM Management. NFS storage and OOB IPMI are additional VLANs added for this lab. A second site VLAN range is pre-provisioned using a completely separate numbering scheme to avoid any ambiguity when both sites are active simultaneously.
Design Decisions
- Site 1 VLANs use the 111x range (1110–1115). Site 2 uses 121x (1210–1215). The leading digit difference makes it immediately obvious from any port config or trunk which site a VLAN belongs to.
- OOB IPMI uses VLAN 100 (Site 1) and VLAN 200 (Site 2) — intentionally low, access-only, never trunked with data VLANs.
- BGP T0 uplink VLANs (60, 70, 160, 170) use dedicated /30 subnets on access-mode ports connecting to NSX Edge uplink vNICs. No other traffic shares these VLANs.
- Native VLAN on blade trunks is set to ESXi Management (1111 for Site 1, 1211 for Site 2) so untagged ESXi management frames are processed correctly.
- Both site VLAN ranges are trunked on all blade port-channels from day one. To isolate a site, remove that site's VLANs from the allowed list — no port mode changes required.
VLAN Reference — Site 1
| VLAN | Name / Purpose | Subnet | Gateway | MTU | Notes |
|---|---|---|---|---|---|
| 100 | OOB IPMI / iDRAC | 10.10.0.0/24 | 10.10.0.1 | 1500 | Access only — Chassis 1+2 IPMI, R630 iDRAC |
| 1110 | VM Management | 10.11.10.0/24 | 10.11.10.1 | 1500 | VCF VM-Mgmt network |
| 1111 | ESXi Management | 10.11.11.0/24 | 10.11.11.1 | 1500 | SDDC Mgr, vCenter, NSX Mgr — Native VLAN on blade trunks |
| 1112 | vMotion | 10.11.12.0/24 | 10.11.12.1 | 9000 | Jumbo MTU — VDS vmkernel port must match |
| 1113 | vSAN | 10.11.13.0/24 | 10.11.13.1 | 9000 | Jumbo MTU — vSAN OSA architecture |
| 1114 | NSX Host TEP | 10.11.14.0/24 | 10.11.14.1 | 9000 | Geneve encapsulation — jumbo MTU mandatory |
| 1115 | NFS Storage | 10.11.15.0/24 | 10.11.15.1 | 9000 | NFS from R630 Ubuntu VM — static IP 10.11.15.10 |
| 60 | NSX T0 Uplink 1 | 10.0.60.0/30 | 10.0.60.1 | 9216 | BGP eBGP: Arista .1 ↔ T0 .2 — ASN 65000 ↔ 65001 |
| 70 | NSX T0 Uplink 2 | 10.0.70.0/30 | 10.0.70.1 | 9216 | BGP eBGP: Arista .1 ↔ T0 .2 — ASN 65000 ↔ 65001 |
VLAN Reference — Site 2
| VLAN | Name / Purpose | Subnet | Gateway | MTU | Notes |
|---|---|---|---|---|---|
| 200 | OOB IPMI / iDRAC S2 | 10.20.0.0/24 | 10.20.0.1 | 1500 | Access only — Chassis 2 IPMI |
| 1210 | VM Management S2 | 10.12.10.0/24 | 10.12.10.1 | 1500 | Site 2 VM-Mgmt |
| 1211 | ESXi Management S2 | 10.12.11.0/24 | 10.12.11.1 | 1500 | Native VLAN on CHx2 blade trunks |
| 1212 | vMotion S2 | 10.12.12.0/24 | 10.12.12.1 | 9000 | Jumbo MTU |
| 1213 | vSAN S2 | 10.12.13.0/24 | 10.12.13.1 | 9000 | Jumbo MTU — vSAN OSA |
| 1214 | NSX Host TEP S2 | 10.12.14.0/24 | 10.12.14.1 | 9000 | Geneve — jumbo MTU mandatory |
| 1215 | NFS Storage S2 | 10.12.15.0/24 | 10.12.15.1 | 9000 | Site 2 NFS |
| 160 | NSX T0 Uplink 1 S2 | 10.0.160.0/30 | 10.0.160.1 | 9216 | BGP eBGP: Arista .1 ↔ T0 .2 — ASN 65000 ↔ 65002 |
| 170 | NSX T0 Uplink 2 S2 | 10.0.170.0/30 | 10.0.170.1 | 9216 | BGP eBGP: Arista .1 ↔ T0 .2 — ASN 65000 ↔ 65002 |
MTU Strategy
MTU misconfiguration is one of the most common causes of silent VCF failures. vSAN, vMotion, and NSX Geneve tunnels all require end-to-end jumbo frame support. A mismatch anywhere in the path causes fragmentation or silent drops that manifest as performance degradation or session instability rather than obvious errors.
| Traffic Type | Required MTU | Applies To |
|---|---|---|
| ESXi / VM Mgmt / OOB | 1500 | VLANs 100, 200, 1110, 1111, 1210, 1211 |
| vMotion | 9000 (inner payload) | VLAN 1112 / 1212 — SVI MTU 9000 |
| vSAN (OSA) | 9000 (inner payload) | VLAN 1113 / 1213 — health check will warn on mismatch |
| NSX Host TEP (Geneve) | 9000 inner / 9216 physical | VLAN 1114 / 1214 — Geneve adds ~50 bytes overhead |
| NFS Storage | 9000 | VLAN 1115 / 1215 — jumbo recommended even on 1GbE |
| NSX T0 BGP Uplinks | 9216 | VLANs 60, 70, 160, 170 — SVIs and access ports |
| Blade Port-Channels (Po1–Po8) | 9216 | Physical MTU headroom for Geneve overhead |
| R630 Trunk Ports | 9000 | 1GbE links — practical ceiling for NFS and mgmt |
Key Rule: Physical port MTU ≥ SVI MTU ≥ VMkernel port MTU. Blade port-channels = 9216. Jumbo SVIs = 9000 or 9216. VDS VMkernel ports for vMotion/vSAN/TEP = 9000. Never set VMkernel MTU higher than its SVI MTU.
Switch Port Allocation
| Ports | Device | LAG / Mode | MTU | Notes |
|---|---|---|---|---|
| Et1–Et2 | CHx1-NodeA (Site 1) | LACP → Po1 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1111 |
| Et3–Et4 | CHx1-NodeB (Site 1) | LACP → Po2 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1111 |
| Et5–Et6 | CHx1-NodeC (Site 1) | LACP → Po3 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1111 |
| Et7–Et8 | CHx1-NodeD (Site 1) | LACP → Po4 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1111 |
| Et9–Et12 | CHx1 IPMI (Nodes A–D) | Access | 1500 | Access VLAN 100 — OOB only |
| Et13–Et14 | CHx2-NodeE (Site 2) | LACP → Po5 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1211 |
| Et15–Et16 | CHx2-NodeF (Site 2) | LACP → Po6 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1211 |
| Et17–Et18 | CHx2-NodeG (Site 2) | LACP → Po7 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1211 |
| Et19–Et20 | CHx2-NodeH (Site 2) | LACP → Po8 | 9216 | Trunk: 100,200,1110-1115,1210-1215 | Native: 1211 |
| Et21–Et24 | CHx2 IPMI (Nodes E–H) | Access | 1500 | Access VLAN 200 — OOB only |
| Et25–Et28 | R630-1 NIC1–4 | Trunk (no LAG) | 9000 | Trunk: all VLANs | Native: 1111 | 1GbE links |
| Et29 | R630-1 iDRAC | Access | 1500 | Access VLAN 100 |
| Et30–Et33 | R630-2 NIC1–4 | Trunk (no LAG) | 9000 | Trunk: all VLANs | Native: 1111 | 1GbE links |
| Et34 | R630-2 iDRAC | Access | 1500 | Access VLAN 100 |
| Et35 | NUC NIC1 | Trunk | 1500 | Admin trunk — all VLANs | Native: 1111 |
| Et36 | NUC NIC2 | Access | 1500 | Access VLAN 1110 — VM-Mgmt |
| Et37 | NSX T0 S1 Uplink1 | Access | 9216 | Access VLAN 60 | BGP peer 10.0.60.2 |
| Et38 | NSX T0 S1 Uplink2 | Access | 9216 | Access VLAN 70 | BGP peer 10.0.70.2 |
| Et39 | NSX T0 S2 Uplink1 | Access | 9216 | Access VLAN 160 | BGP peer 10.0.160.2 |
| Et40 | NSX T0 S2 Uplink2 | Access | 9216 | Access VLAN 170 | BGP peer 10.0.170.2 |
| Et41–Et46 | SPARE | Shutdown | — | Available for future expansion |
| Et47 | Internet Uplink | Routed L3 | 1500 | 192.168.31.2/24 — default route via 192.168.31.1 |
| Et48 | Cisco 3750E Trunk | Trunk | 9216 | All VLANs both sites | Native: 1111 |
| Et49–Et52 | QSFP Reserved | Shutdown | — | 40G uplinks — reserved |
| Management1 | OOB Management | DHCP | 1500 | 192.168.31.x/24 from home AP — out-of-band only |
EOS Configuration
1 Baseline — Hostname, Routing & Credentials
Global — Hostname / Routing / Credentials
hostname VCF-LEAF-SW01 ! spanning-tree mode mstp ! no aaa root username admin privilege 15 role network-admin secret 0 <REPLACE_PASSWORD> ! ip routing ! ! Default route toward home router — internet access for workload VMs via BGP ip route 0.0.0.0/0 192.168.31.1
2 VLAN Database
VLAN Database — Site 1 & Site 2
! ── Site 1 VLANs ────────────────────────────────────────────── vlan 60 name NSX-T0-Uplink1-S1 vlan 70 name NSX-T0-Uplink2-S1 vlan 100 name OOB-IPMI-S1 vlan 1110 name VM-Mgmt-S1 vlan 1111 name ESX-Mgmt-S1 vlan 1112 name vMotion-S1 vlan 1113 name vSAN-S1 vlan 1114 name NSX-TEP-S1 vlan 1115 name NFS-S1 ! ── Site 2 VLANs ────────────────────────────────────────────── vlan 160 name NSX-T0-Uplink1-S2 vlan 170 name NSX-T0-Uplink2-S2 vlan 200 name OOB-IPMI-S2 vlan 1210 name VM-Mgmt-S2 vlan 1211 name ESX-Mgmt-S2 vlan 1212 name vMotion-S2 vlan 1213 name vSAN-S2 vlan 1214 name NSX-TEP-S2 vlan 1215 name NFS-S2
3 LACP Port-Channels
Each Supermicro blade node has two 10GbE NICs bonded as LACP port-channels (active/active) providing link redundancy and 20Gbps aggregate bandwidth. All port-channels trunk both site VLAN ranges from day one.
Port-Channel Configuration (LACP) — Chassis 1 & Chassis 2
! ── Chassis 1 — Nodes A/B/C/D (Site 1 native VLAN 1111) ────── interface Port-Channel1 description CHx1-NodeA-LACP switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 no shutdown ! interface Port-Channel2 description CHx1-NodeB-LACP switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 no shutdown ! interface Port-Channel3 description CHx1-NodeC-LACP switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 no shutdown ! interface Port-Channel4 description CHx1-NodeD-LACP switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 no shutdown ! ! ── Chassis 2 — Nodes E/F/G/H (Site 2 native VLAN 1211) ────── interface Port-Channel5 description CHx2-NodeE-LACP switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1211 mtu 9216 no shutdown ! ! Po6/Po7/Po8 follow identical pattern with native vlan 1211
LACP Member Ports — Physical blade NICs (Node A example)
interface Ethernet1 description CHx1-NodeA-NIC1-LAG1 switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 channel-group 1 mode active spanning-tree portfast no shutdown ! interface Ethernet2 description CHx1-NodeA-NIC2-LAG1 switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 channel-group 1 mode active spanning-tree portfast no shutdown ! ! Pattern repeats: ! Et3/Et4 -> channel-group 2 (Node B) ! Et5/Et6 -> channel-group 3 (Node C) ! Et7/Et8 -> channel-group 4 (Node D) ! Et13/14 -> channel-group 5 native 1211 (Node E) ! Et15/16 -> channel-group 6 native 1211 (Node F) ! Et17/18 -> channel-group 7 native 1211 (Node G) ! Et19/20 -> channel-group 8 native 1211 (Node H)
4 OOB IPMI / iDRAC Ports
IPMI / iDRAC Access Ports
! Chassis 1 IPMI — access VLAN 100 (Et9–Et12) interface Ethernet9 description CHx1-NodeA-IPMI switchport mode access switchport access vlan 100 mtu 1500 spanning-tree portfast no shutdown ! Et10/11/12 — NodeB/C/D IPMI — identical config, VLAN 100 ! ! Chassis 2 IPMI — access VLAN 200 (Et21–Et24) interface Ethernet21 description CHx2-NodeE-IPMI switchport mode access switchport access vlan 200 mtu 1500 spanning-tree portfast no shutdown ! Et22/23/24 — NodeF/G/H IPMI — identical config, VLAN 200 ! ! R630-1 iDRAC — access VLAN 100 (Et29) interface Ethernet29 description R630-1-iDRAC-OOB switchport mode access switchport access vlan 100 mtu 1500 spanning-tree portfast no shutdown
5 Dell R630 — Management / NFS Host
The R630 has 4× onboard 1GbE NICs. All four are trunked with full VLAN ranges. The Ubuntu VM has a static IP of 10.11.15.10 on VLAN 1115 (NFS) and resides on VLAN 1111 (ESXi Management).
R630-1 Management / NFS Host — Et25–Et28
interface Ethernet25 description R630-1-NIC1 switchport mode trunk switchport trunk allowed vlan 100,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9000 spanning-tree portfast no shutdown ! ! Ethernet26/27/28 — R630-1-NIC2/3/4 — identical config
6 NSX T0 BGP Uplink Ports
Four dedicated access ports connect to NSX Edge node uplink vNICs. Each port is an access port on its own /30 VLAN. The Arista SVI acts as the BGP peer endpoint.
NSX T0 BGP Uplink Access Ports — Et37–Et40
! Site 1 interface Ethernet37 description NSX-T0-S1-Uplink1-VLAN60 switchport mode access switchport access vlan 60 mtu 9216 spanning-tree portfast no shutdown ! interface Ethernet38 description NSX-T0-S1-Uplink2-VLAN70 switchport mode access switchport access vlan 70 mtu 9216 spanning-tree portfast no shutdown ! ! Site 2 interface Ethernet39 description NSX-T0-S2-Uplink1-VLAN160 switchport mode access switchport access vlan 160 mtu 9216 spanning-tree portfast no shutdown ! interface Ethernet40 description NSX-T0-S2-Uplink2-VLAN170 switchport mode access switchport access vlan 170 mtu 9216 spanning-tree portfast no shutdown
7 Internet Uplink & Cisco Core Trunk
Internet Uplink / Cisco Core SW / OOB Management
! Et47 — Routed L3 port to home router/firewall ! Workload VMs reach internet via NSX T0 -> Arista Et47 -> 192.168.31.1 interface Ethernet47 description Internet-Uplink-HomeRouter no switchport ip address 192.168.31.2/24 mtu 1500 no shutdown ! ! Et48 — Trunk uplink to Cisco Catalyst 3750E (VCF-CORE-SW01) interface Ethernet48 description Cisco-VCF-CORE-SW01-Trunk switchport mode trunk switchport trunk allowed vlan 60,70,100,160,170,200,1110-1115,1210-1215 switchport trunk native vlan 1111 mtu 9216 no shutdown ! ! Management1 — OOB port to home AP (separate from data plane) interface Management1 description OOB-Management-HomeAP ip address dhcp no shutdown
8 SVIs — Layer 3 Routing Interfaces
Layer 3 SVIs — Site 1 (Site 2 follows same pattern)
! ── Site 1 SVIs ─────────────────────────────────────────────── interface Vlan100 description OOB-IPMI-S1 ip address 10.10.0.1/24 mtu 1500 no shutdown ! interface Vlan1110 description VM-Mgmt-S1 ip address 10.11.10.1/24 mtu 1500 no shutdown ! interface Vlan1111 description ESX-Mgmt-S1 ip address 10.11.11.1/24 mtu 1500 no shutdown ! interface Vlan1112 description vMotion-S1 ip address 10.11.12.1/24 mtu 9000 no shutdown ! interface Vlan1113 description vSAN-S1 ip address 10.11.13.1/24 mtu 9000 no shutdown ! interface Vlan1114 description NSX-TEP-S1 ip address 10.11.14.1/24 mtu 9000 no shutdown ! interface Vlan1115 description NFS-S1 ip address 10.11.15.1/24 mtu 9000 no shutdown ! interface Vlan60 description NSX-T0-Uplink1-S1 ip address 10.0.60.1/30 mtu 9216 no shutdown ! interface Vlan70 description NSX-T0-Uplink2-S1 ip address 10.0.70.1/30 mtu 9216 no shutdown ! ! ── Site 2 SVIs — same structure ────────────────────────────── ! Vlan200/1210/1211 -> mtu 1500, 10.20.x / 10.12.1x.x ! Vlan1212/1213/1214/1215 -> mtu 9000, 10.12.1x.x ! Vlan160 -> ip 10.0.160.1/30 mtu 9216 ! Vlan170 -> ip 10.0.170.1/30 mtu 9216
9 eBGP Configuration — NSX T0 Peering
The Arista runs eBGP ASN 65000. NSX T0 Site 1 uses ASN 65001, Site 2 uses ASN 65002. The Arista advertises all infrastructure subnets plus the default route so workload VMs can reach the internet through the NSX T0 → Arista → Et47 path.
eBGP Configuration — ASN 65000
router bgp 65000
router-id 10.11.11.254
no bgp default ipv4-unicast
maximum-paths 4 ecmp 4
!
! ── Site 1 T0 peers (ASN 65001) ──────────────────────────
neighbor 10.0.60.2 remote-as 65001
neighbor 10.0.60.2 description NSX-T0-S1-Uplink1
neighbor 10.0.60.2 send-community
neighbor 10.0.60.2 maximum-routes 500
neighbor 10.0.60.2 bfd
!
neighbor 10.0.70.2 remote-as 65001
neighbor 10.0.70.2 description NSX-T0-S1-Uplink2
neighbor 10.0.70.2 send-community
neighbor 10.0.70.2 maximum-routes 500
neighbor 10.0.70.2 bfd
!
! ── Site 2 T0 peers (ASN 65002) ──────────────────────────
neighbor 10.0.160.2 remote-as 65002
neighbor 10.0.160.2 description NSX-T0-S2-Uplink1
neighbor 10.0.160.2 send-community
neighbor 10.0.160.2 maximum-routes 500
neighbor 10.0.160.2 bfd
!
neighbor 10.0.170.2 remote-as 65002
neighbor 10.0.170.2 description NSX-T0-S2-Uplink2
neighbor 10.0.170.2 send-community
neighbor 10.0.170.2 maximum-routes 500
neighbor 10.0.170.2 bfd
!
address-family ipv4
neighbor 10.0.60.2 activate
neighbor 10.0.70.2 activate
neighbor 10.0.160.2 activate
neighbor 10.0.170.2 activate
! OOB
network 10.10.0.0/24
network 10.20.0.0/24
! Site 1 infrastructure
network 10.11.10.0/24
network 10.11.11.0/24
network 10.11.12.0/24
network 10.11.13.0/24
network 10.11.14.0/24
network 10.11.15.0/24
! Site 2 infrastructure
network 10.12.10.0/24
network 10.12.11.0/24
network 10.12.12.0/24
network 10.12.13.0/24
network 10.12.14.0/24
network 10.12.15.0/24
! Default route — workload VM internet access
network 0.0.0.0/0
NSX T0 Side Required: Configure matching BGP settings on NSX T0 — Local AS 65001 (Site 1) or 65002 (Site 2), remote-as 65000, neighbour IPs 10.0.60.1 / 10.0.70.1 (Site 1) and 10.0.160.1 / 10.0.170.1 (Site 2). BFD must be enabled on both sides if used.
10 NTP, LLDP, SSH & eAPI
NTP / LLDP / SSH / eAPI
lldp run
!
ntp server 192.168.31.1 prefer
ntp server 0.pool.ntp.org
ntp server 1.pool.ntp.org
!
logging on
logging buffered 65535 informational
! logging host <SYSLOG_SERVER_IP>
!
management ssh
idle-timeout 60
authentication mode password
no shutdown
!
management api http-commands
protocol https
no protocol http
no shutdown
vrf default
no shutdown
Validation — End-to-End Checklist
Run this validation sequence in order. Each phase builds on the previous. Do not proceed to VCF deployment until all checks pass.
Phase 1 — Physical Layer & Port State
| Check | EOS Command | Expected Result |
|---|---|---|
| All active ports are up/up | show interfaces status | Connected ports show connected, correct speed |
| No err-disabled ports | show interfaces status err-disabled | No output (empty) |
| LACP port-channels formed | show port-channel summary | Po1–Po8 show U (in use), member ports show P (bundled) |
| LLDP neighbours visible | show lldp neighbors | R630, NUC, Cisco 3750E, Supermicro nodes visible |
| Correct LLDP port mapping | show lldp neighbors detail | Verify each device on expected interface |
Phase 2 — VLAN & Trunk Verification
| Check | EOS Command | Expected Result |
|---|---|---|
| All 18 VLANs in database | show vlan | VLANs 60,70,100,160,170,200,1110–1115,1210–1215 active |
| VLANs active on correct ports | show vlan id 1111 | Po1–Po4, Et25–28, Et35, Et48 listed |
| Blade trunks carry both site VLANs | show interfaces trunk | Po1–Po8 allowed VLANs include both 111x and 121x ranges |
| Native VLANs correct | show interfaces trunk | Po1–Po4 native=1111, Po5–Po8 native=1211 |
| IPMI ports in correct VLAN | show interfaces Ethernet9 switchport | Access VLAN 100 |
| T0 uplink ports in correct VLAN | show interfaces Ethernet37 switchport | Access VLAN 60 |
Phase 3 — Layer 3 SVI & IP Routing
| Check | EOS Command | Expected Result |
|---|---|---|
| All SVIs are up/up | show ip interface brief | All Vlan interfaces show protocol up |
| SVI IP addresses correct | show ip interface brief | Verify .1 address on each VLAN subnet |
| SVI MTU matches VLAN policy | show interfaces Vlan1112 | MTU 9000 for jumbo VLANs, 1500 for mgmt VLANs |
| Routing table populated | show ip route | Connected routes for all 18 subnets present |
| Default route installed | show ip route 0.0.0.0/0 | Via 192.168.31.1, Ethernet47 |
| Internet reachability | ping vrf default 8.8.8.8 | Success — confirms Et47 uplink and NAT on home router |
Phase 4 — MTU End-to-End Validation
| Check | EOS Command / Test | Expected Result |
|---|---|---|
| SVI MTU — jumbo VLANs | show interfaces Vlan1113 | MTU 9000 |
| SVI MTU — T0 uplinks | show interfaces Vlan60 | MTU 9216 |
| Port-channel MTU | show interfaces Port-Channel1 | MTU 9216 |
| Physical member port MTU | show interfaces Ethernet1 | MTU 9216 |
| Jumbo ping — vSAN VLAN | ping vrf default 10.11.13.1 size 8972 df-bit | Success — 5/5 packets |
| Jumbo ping — TEP VLAN | ping vrf default 10.11.14.1 size 8972 df-bit | Success — 5/5 packets |
Jumbo Frame Ping Tests — Run from Arista
! 8972 byte payload + 28 byte IP/ICMP header = 9000 bytes on wire ! Failure = MTU mismatch somewhere in the path ping vrf default 10.11.12.1 size 8972 df-bit repeat 5 ! vMotion ping vrf default 10.11.13.1 size 8972 df-bit repeat 5 ! vSAN ping vrf default 10.11.14.1 size 8972 df-bit repeat 5 ! NSX TEP ping vrf default 10.11.15.1 size 8972 df-bit repeat 5 ! NFS
Phase 5 — BGP Uplink Verification
| Check | EOS Command | Expected Result |
|---|---|---|
| BGP process running | show bgp summary | BGP process up — peers may show Active/Idle pre-NSX |
| T0 uplink SVIs up (Site 1) | show interfaces Vlan60 | up/up, IP 10.0.60.1/30, MTU 9216 |
| T0 uplink SVIs up (Site 1) | show interfaces Vlan70 | up/up, IP 10.0.70.1/30, MTU 9216 |
| Physical uplink ports up | show interfaces Ethernet37 | connected, 10G full, MTU 9216 |
| [Post-NSX] BGP Established | show bgp summary | Peer 10.0.60.2 state = Established, prefixes > 0 |
| [Post-NSX] Routes received from T0 | show bgp neighbors 10.0.60.2 received-routes | NSX overlay segment routes received |
| [Post-NSX] Routes advertised to T0 | show bgp neighbors 10.0.60.2 advertised-routes | All 16 infra subnets + 0.0.0.0/0 shown |
| [Post-NSX] BFD sessions up | show bfd peers | State = Up for all T0 peers |
Phase 6 — NFS Path Validation
NFS Path Validation Commands
! From Arista — confirm NFS SVI is up and R630 VM is reachable show interfaces Vlan1115 ping vrf default 10.11.15.10 ! From ESXi host (once installed) — confirm NFS mount path esxcli network ip interface list vmkping -I vmk0 -d -s 8972 10.11.15.10 ! jumbo frame test to NFS server ! From Ubuntu NFS server (R630 VM) — verify export is active showmount -e localhost cat /etc/exports systemctl status nfs-kernel-server
NFS Firewall Note: The Ubuntu NFS VM uses UFW. Ensure rules allow traffic from
10.11.11.0/24 (ESXi Management) and 10.11.15.0/24 (NFS VLAN) on ports 111 (rpcbind) and 2049 (NFS). Run: ufw allow from 10.11.11.0/24 to any port 2049
Phase 7 — OOB IPMI Reachability
| Check | EOS Command | Expected Result |
|---|---|---|
| IPMI VLAN 100 SVI up | show interfaces Vlan100 | up/up, 10.10.0.1/24 |
| IPMI VLAN 200 SVI up | show interfaces Vlan200 | up/up, 10.20.0.1/24 |
| Ping Chassis 1 Node A IPMI | ping vrf default 10.10.0.x | Success (DHCP addr from VLAN 100 pool) |
| Ping R630 iDRAC | ping vrf default 10.10.0.y | Success |
| IPMI ports in access VLAN | show interfaces Ethernet9 switchport | Access mode, VLAN 100 |
Phase 8 — Full Connectivity Matrix
Full Connectivity Matrix — Run from Arista before VCF deployment
! ── Layer 3 SVI self-test ────────────────────────────────────── ping vrf default 10.11.11.1 source Vlan1111 ! ESXi Mgmt SVI ping vrf default 10.11.12.1 source Vlan1112 ! vMotion SVI ping vrf default 10.11.13.1 source Vlan1113 ! vSAN SVI ping vrf default 10.11.14.1 source Vlan1114 ! NSX TEP SVI ping vrf default 10.11.15.1 source Vlan1115 ! NFS SVI ping vrf default 10.0.60.1 source Vlan60 ! T0 Uplink1 SVI ping vrf default 10.0.70.1 source Vlan70 ! T0 Uplink2 SVI ! ! ── Device reachability ───────────────────────────────────────── ping vrf default 10.11.15.10 ! R630 NFS VM ping vrf default 10.10.0.x ! Supermicro IPMI ping vrf default 192.168.31.1 ! Home router ping vrf default 8.8.8.8 ! Internet ! ! ── BGP T0 uplinks (after NSX deployment) ────────────────────── ping vrf default 10.0.60.2 ! T0 S1 Uplink1 peer ping vrf default 10.0.70.2 ! T0 S1 Uplink2 peer ping vrf default 10.0.160.2 ! T0 S2 Uplink1 peer ping vrf default 10.0.170.2 ! T0 S2 Uplink2 peer
Issues Encountered & Fixes
Real issues hit during this lab build, documented so others don't waste time on the same problems.
| Symptom | Root Cause | Fix |
|---|---|---|
| LACP port-channel stuck in I (individual) state | ESXi not yet installed — no LACP PDUs sent from host NIC team | Expected pre-ESXi. Port-channels form once ESXi LACP NIC teaming is configured on the VDS. Verify with show lacp neighbor once ESXi is up. |
| vSAN health: MTU check failed | VMkernel vSAN port MTU left at 1500 default while SVI is 9000 | Set VMkernel port MTU to 9000 on the VDS vSAN portgroup. Must match across all ESXi hosts in the cluster. |
| NSX TEP tunnels not forming | MTU mismatch — Geneve needs ~50 bytes overhead on top of 9000 inner payload | Confirm blade port-channel MTU is 9216. Verify: show interfaces Po1 — MTU must show 9216 not 9000. |
| BGP peer stuck in Active state | NSX Edge uplink vNIC not connected to correct port or VLAN mismatch on access port | Verify NSX Edge uplink vNIC is on the correct portgroup, VLAN 60/70 is tagged, and the physical port (Et37/38) shows connected. |
| NFS datastore mount fails in VCF | UFW on R630 Ubuntu VM blocking NFS ports from ESXi management subnet | ufw allow from 10.11.11.0/24 to any port 2049 and ufw allow from 10.11.11.0/24 to any port 111 |
| Management1 and Et47 subnet overlap concern | Both ports on 192.168.31.x — potential routing confusion | Management1 operates in the mgmt VRF, Et47 is in the default VRF. No actual overlap. Confirm with show ip interface brief. |
Next Steps
With the network layer validated, the remaining pre-requisites before launching the VCF 9 SDDC Manager installer OVA are:
- ESXi 9.x installed on all four management domain nodes (CHx1 Nodes A–D) with management vmkernel on VLAN 1111
- VDS configured with portgroups for vMotion (1112), vSAN (1113), NSX TEP (1114), and NFS (1115) with correct MTU settings
- DNS entries created for all VCF components (SDDC Manager, vCenter, NSX Manager ×3, ESXi hosts) before deployment begins
- NTP synchronised across all hosts — VCF deployment fails if time drift exceeds threshold
- VCF 9 Planning and Preparation Workbook completed with all IP and DNS entries populated
- R630 NFS export mounted as a datastore on all management domain hosts for SDDC Manager VM storage
- 3PAR drives sg_formatted 520→512 byte sectors using
sg_format(sg3_utils) via LSI 9211-8i HBA in IT mode for vSAN OSA
VCF 9 Architecture Change: The SDDC Manager OVA is the installer in VCF 9. There is no separate Cloud Builder appliance as in VCF 5.x. This catches many engineers familiar with older documentation — do not reference VCF 5.x deployment guides.
Comments
Post a Comment