VXLAN needs matching tunnel parameters as well as a working underlay

Thu, 28 Oct 2027 00:00:00 +0000

VXLAN carries an Ethernet frame inside a UDP packet so an overlay network can cross an IP underlay. Tunnel endpoints encapsulate and decapsulate frames, while a 24-bit VXLAN Network Identifier, or VNI, identifies the virtual segment. The outer IP addresses route between endpoints; the inner Ethernet and IP addresses serve the overlay. These are separate layers with separate reachability and configuration requirements.

When it breaks

Pinging a tunnel endpoint proves little about a particular overlay segment. Both endpoints may be reachable while using different VNIs, UDP destination ports, or remote-peer configuration. An underlay firewall can also admit ICMP while dropping encapsulated UDP. Diagnose the actual transport and virtual segment instead of assuming that one successful underlay ping validates all tenant traffic above it.

Encapsulation consumes the underlay’s packet budget. With untagged inner Ethernet and ordinary outer IPv4, overlay IP MTU 1450 fits underlay IP MTU 1500: inner Ethernet 14, VXLAN 8, UDP 8, and IPv4 20 bytes. Inner VLAN tags, outer IPv6, or additional encapsulation change that IP budget; an outer Ethernet VLAN tag instead affects link-frame size. RFC 7348 prohibits VTEP fragmentation and permits a destination to discard fragments. A small request does not test path MTU or large packets.

A VNI also is not an encryption key or authenticated tenant identity. Anyone permitted to observe the encapsulated traffic may be able to read its contents, and trust in tunnel endpoints depends on the surrounding network controls. VXLAN’s segmentation role does not establish confidentiality. Protect the underlay and use appropriate authenticated encryption where the threat model requires it.

VXLAN: nested layers, separate checks: conceptual mechanisms and expected checks, not new runtime evidence.

Check it quickly

Save check-vxlan.py for an authorized disposable Linux lab with a verified local Docker daemon and enabled Python assertions. The recipe names Python 3.11.16, iproute2 6.1.0 and tcpdump 4.99.3; verify architecture, packages and kernel support separately. NET_ADMIN/SYS_ADMIN do not override mandatory mount restrictions such as Docker’s default AppArmor policy. Use a compatible approved lab without weakening protections. Runtime isolation does not make builds offline; cleanup return codes are unchecked. No exact runtime capture for this fence was recovered or newly produced here; below are expected observations.

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
 result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
 if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
 return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
 if process.poll() is None:
 process.terminate()
 try:process.wait(timeout=2)
 except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
 assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
 value=process.stdout.readline();assert value,'child exited without a response'
 return value
def spawn(stack,command):
 process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process);return process
@contextmanager
def lab(names):
 with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
 root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
 stack.callback(run,'umount',str(proc))
 for name in names:
 run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
 ip(name,'link','set','lo','up')
 def setting(name,key,value):
 assert name in names
 path=proc/'sys'/key.replace('.','/')
 return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
 yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
 run('ip','link','add',left_if,'type','veth','peer','name',right_if)
 for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
 run('ip','link','set',interface,'netns',name)
 for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
 ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
 data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
 for entry in data['nftables']:
 rule=entry.get('rule',{})
 if 'comment' in rule:
 for expression in rule['expr']:
 if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
 return found

def server(stack,name,address,port=18080,label='server'):
 code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
 def do_GET(self):
 body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
 self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
 def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
 address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
 def server_bind(self):
 if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
 super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
 process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
 assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
 code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
 client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
 assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
 return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

def capture(stack,root,name,interface,expression,label):
 path=root/(label+'.pcap')
 process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
 stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
 stack.callback(stop,process)
 deadline=time.monotonic()+3
 while time.monotonic()<deadline:
 assert process.poll() is None,'tcpdump exited before listening'
 if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
 raise AssertionError('tcpdump did not become ready')
def captured(process,path,ascii=False):
 process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
 assert process.returncode==0,diagnostic
 result=run('tcpdump','-nn','-vv',*(['-A'] if ascii else []),'-r',str(path))
 return result.stdout

with lab(['left','right']) as (root,stack,setting):
 wire('left','ul','right','ur',['192.0.2.1/30'],['192.0.2.2/30'])
 for name,interface in [('left','ul'),('right','ur')]:ip(name,'link','set',interface,'mtu','1500')
 def overlay(name,vni):
 local,remote,dev,address,mac=('192.0.2.1','192.0.2.2','ul','10.42.0.1/24','02:00:00:00:42:01') if name=='left' else ('192.0.2.2','192.0.2.1','ur','10.42.0.2/24','02:00:00:00:42:02')
 ip(name,'link','add','vx0','type','vxlan','id',str(vni),'local',local,'remote',remote,'dev',dev,'dstport','4789','nolearning')
 ip(name,'link','set','vx0','address',mac,'mtu','1450')
 ip(name,'addr','add',address,'dev','vx0');ip(name,'link','set','vx0','up')
 overlay('left',42);overlay('right',42);server(stack,'right','10.42.0.2',label='overlay')
 tap,path=capture(stack,root,'left','ul','udp port 4789','vxlan')
 successful=request('left','10.42.0.2');assert successful['ok'] and successful['body']['server']=='overlay'
 packets=captured(tap,path,ascii=True)
 assert 'vni 42' in packets.lower() and 'lab-visible-message' in packets and '.4789:' in packets,packets
 details=json.loads(ip('left','-d','-j','link','show','vx0').stdout)[0]
 assert details['linkinfo']['info_data']['id']==42 and details['mtu']==1450
 ip('right','link','delete','vx0');overlay('right',43)
 underlay=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert underlay.returncode==0
 mismatch=request('left','10.42.0.2');assert not mismatch['ok'] and mismatch['type']=='TimeoutError'
 ip('right','link','delete','vx0');overlay('right',42)
 recovered=request('left','10.42.0.2');assert recovered['ok']
 print(json.dumps({'healthy_overlay':successful,'underlay_packet_capture':packets,'vni':details['linkinfo']['info_data']['id'],
 'underlay_mtu':1500,'overlay_mtu':details['mtu'],'mismatched_vni_underlay_ping_exit':underlay.returncode,
 'mismatched_vni_overlay':mismatch,'matching_vni_restored':recovered}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
 with TemporaryDirectory(prefix=name+'-') as directory:
 root=Path(directory);(root/'Dockerfile').write_text(recipe)
 build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
 result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
 '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
 assert result.returncode==0,result.stderr
 print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
 docker('rm','-f',name);docker('image','rm',name)

The underlay addresses are 192.0.2.1 and 192.0.2.2. Each endpoint has an explicitly configured remote peer and UDP destination port 4789. Setting the port explicitly avoids depending on historical implementation defaults. Both overlay devices initially use VNI 42, fixed MAC addresses, and addresses 10.42.0.1 and 10.42.0.2. This is a small unicast tunnel, not a complete multipoint fabric or EVPN deployment.

The fixture expects an HTTP response with the overlay server identity and a simultaneous underlay capture. Its capture assertions check VNI 42, UDP port 4789 and the literal path /lab-visible-message, not a complete ARP/TCP decode. If actually observed, the visible path would demonstrate unencrypted application bytes in this configuration. No recovered capture establishes that observation here; VXLAN’s lack of native confidentiality is independently specified by the standard.

A veth capture may show apparently incorrect UDP or TCP checksums even when HTTP succeeds. Kernel partial-checksum and encapsulation-offload processing can explain such observations, but a capture alone does not identify the exact stage or prove wire corruption. This fixture neither disables offloads nor performs a complete checksum diagnosis; its historical capture claim is not newly verified here.

Next, the right device is recreated with VNI 43 and the same address and MAC. The expected underlay ping succeeds while overlay HTTP times out; recreating VNI 42 should restore HTTP. The VNI is the deliberate configuration difference, but recreation also resets device state. A timeout alone is not a universal VNI-mismatch signature or proof that every other condition remained identical.

Match evidence to the layer being changed

Inspect the endpoint’s underlay route, selected interface, configured UDP port, VNI, peer mapping, and overlay address. Capture on both the tunnel device and its transport interface when possible: each exposes a different representation of the same communication. Record where the capture was taken, particularly when interpreting packet sizes or checksum warnings that may reflect kernel offload processing.

This example deliberately reserves encapsulation space with MTUs of 1500 and 1450, but tests only a short application exchange. Validate the real path’s packet-size requirements separately before using it for larger workloads. In a production fabric, also review how remote MAC destinations are learned or distributed and how unknown traffic is handled. Those control-plane choices cannot be inferred from this two-endpoint success.

Sources

RFC 7348 VXLAN, VNI, UDP encapsulation, MTU and security considerations, checked 2026-09-12.
Linux VXLAN documentation, VXLAN devices, destination ports and tunnel offloads, checked 2026-09-12.
Linux checksum offloads, partial and encapsulated checksums at different processing stages, checked 2026-09-12.
Moby default AppArmor template, explicit mount denial in the default profile, checked 2026-10-18.
Docker AppArmor, mandatory policy and mount restrictions, checked 2026-10-18.
Docker run reference, capabilities and capability-aware default seccomp policy, checked 2026-10-18.
Python 3.14: assert statements, optimization removes assertion checks, checked 2026-10-18.
Linux ip-link(8), explicit remote peer, destination port and learning settings, checked 2026-10-18.

Tunnel · XB Field Notes

VXLAN needs matching tunnel parameters as well as a working underlay

When it breaks

Check it quickly

Match evidence to the layer being changed

Sources