VXLAN carries an Ethernet frame inside a UDP packet so an overlay network can cross an IP underlay. Tunnel endpoints encapsulate and decapsulate frames, while a 24-bit VXLAN Network Identifier, or VNI, identifies the virtual segment. The outer IP addresses route between endpoints; the inner Ethernet and IP addresses serve the overlay. These are separate layers with separate reachability and configuration requirements.

When it breaks

Pinging a tunnel endpoint proves little about a particular overlay segment. Both endpoints may be reachable while using different VNIs, UDP destination ports, or remote-peer configuration. An underlay firewall can also admit ICMP while dropping encapsulated UDP. Diagnose the actual transport and virtual segment instead of assuming that one successful underlay ping validates all tenant traffic above it.

Encapsulation consumes the underlay’s packet budget. With untagged inner Ethernet and ordinary outer IPv4, overlay IP MTU 1450 fits underlay IP MTU 1500: inner Ethernet 14, VXLAN 8, UDP 8, and IPv4 20 bytes. Inner VLAN tags, outer IPv6, or additional encapsulation change that IP budget; an outer Ethernet VLAN tag instead affects link-frame size. RFC 7348 prohibits VTEP fragmentation and permits a destination to discard fragments. A small request does not test path MTU or large packets.

A VNI also is not an encryption key or authenticated tenant identity. Anyone permitted to observe the encapsulated traffic may be able to read its contents, and trust in tunnel endpoints depends on the surrounding network controls. VXLAN’s segmentation role does not establish confidentiality. Protect the underlay and use appropriate authenticated encryption where the threat model requires it.

Separate VTEP underlay and overlay addresses, nested encapsulation and a qualified 1500-to-1450 MTU budget precede expected VNI comparison states and security and evidence limits.
VXLAN: nested layers, separate checks: conceptual mechanisms and expected checks, not new runtime evidence.

Check it quickly

Save check-vxlan.py for an authorized disposable Linux lab with a verified local Docker daemon and enabled Python assertions. The recipe names Python 3.11.16, iproute2 6.1.0 and tcpdump 4.99.3; verify architecture, packages and kernel support separately. NET_ADMIN/SYS_ADMIN do not override mandatory mount restrictions such as Docker’s default AppArmor policy. Use a compatible approved lab without weakening protections. Runtime isolation does not make builds offline; cleanup return codes are unchecked. No exact runtime capture for this fence was recovered or newly produced here; below are expected observations.

from pathlib import Path
from tempfile import TemporaryDirectory
import json,subprocess,uuid
recipe='FROM python@sha256:528257d48c1da0dcecc2e725d1ae34498d60c965f1241e39cd6a85a8859bdf84\nRUN apt-get update && apt-get install -y --no-install-recommends iproute2=6.1.0-3 iputils-ping=3:20221126-1+deb12u1 nftables=1.0.6-2+deb12u2 tcpdump=4.99.3-1 traceroute=1:2.1.2-1 procps=2:4.0.2-3 && rm -rf /var/lib/apt/lists/*\nCMD ["python3"]\n'
program=r"""
from contextlib import ExitStack,contextmanager
from pathlib import Path
from tempfile import TemporaryDirectory
import json,os,select,signal,subprocess,time

def run(*args,check=True,input=None,timeout=8):
    result=subprocess.run(args,input=input,capture_output=True,text=True,timeout=timeout)
    if check:assert result.returncode==0,(args,result.returncode,result.stdout,result.stderr)
    return result
def ns(name,*args,**kwargs):return run('ip','netns','exec',name,*args,**kwargs)
def ip(name,*args,**kwargs):return ns(name,'ip',*args,**kwargs)
def stop(process):
    if process.poll() is None:
        process.terminate()
        try:process.wait(timeout=2)
        except subprocess.TimeoutExpired:process.kill();process.wait(timeout=2)
def line(process,timeout=3):
    assert select.select([process.stdout],[],[],timeout)[0],'child response deadline exceeded'
    value=process.stdout.readline();assert value,'child exited without a response'
    return value
def spawn(stack,command):
    process=subprocess.Popen(command,stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
    stack.callback(stop,process);return process
@contextmanager
def lab(names):
    with TemporaryDirectory(prefix='net-path-') as directory,ExitStack() as stack:
        root=Path(directory);proc=root/'proc';proc.mkdir();run('mount','-t','proc','proc',str(proc))
        stack.callback(run,'umount',str(proc))
        for name in names:
            run('ip','netns','add',name);stack.callback(run,'ip','netns','delete',name,check=False)
            ip(name,'link','set','lo','up')
        def setting(name,key,value):
            assert name in names
            path=proc/'sys'/key.replace('.','/')
            return ns(name,'python3','-c',f'from pathlib import Path; p=Path({str(path)!r}); p.write_text({str(value)!r}); print(p.read_text().strip())').stdout.strip()
        yield root,stack,setting
def wire(left,left_if,right,right_if,left_addresses,right_addresses):
    run('ip','link','add',left_if,'type','veth','peer','name',right_if)
    for name,interface,addresses in [(left,left_if,left_addresses),(right,right_if,right_addresses)]:
        run('ip','link','set',interface,'netns',name)
        for address in addresses:ip(name,'address','add',address,'dev',interface,*(['nodad'] if ':' in address else []))
        ip(name,'link','set',interface,'up')
def nft(name,script):return ns(name,'nft','-f','-',input=script)
def counters(name):
    data=json.loads(ns(name,'nft','-j','list','ruleset').stdout);found={}
    for entry in data['nftables']:
        rule=entry.get('rule',{})
        if 'comment' in rule:
            for expression in rule['expr']:
                if 'counter' in expression:found[rule['comment']]=expression['counter']['packets']
    return found

def server(stack,name,address,port=18080,label='server'):
    code=r'''
from http.server import BaseHTTPRequestHandler,ThreadingHTTPServer
import json,socket,sys
address,port,label=sys.argv[1],int(sys.argv[2]),sys.argv[3]
class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        body=json.dumps({'server':label,'peer':self.client_address,'path':self.path}).encode()
        self.send_response(200);self.send_header('Content-Length',str(len(body)));self.end_headers();self.wfile.write(body)
    def log_message(self,*args):pass
class Server(ThreadingHTTPServer):
    address_family=socket.AF_INET6 if ':' in address else socket.AF_INET
    def server_bind(self):
        if self.address_family==socket.AF_INET6:self.socket.setsockopt(socket.IPPROTO_IPV6,socket.IPV6_V6ONLY,1)
        super().server_bind()
service=Server((address,port),Handler);print('ready',flush=True);service.serve_forever()
'''
    process=spawn(stack,['ip','netns','exec',name,'python3','-u','-c',code,address,str(port),label])
    assert line(process).strip()=='ready';return process
def request(name,address,port=18080):
    code=r'''
import http.client,json,sys
client=http.client.HTTPConnection(sys.argv[1],int(sys.argv[2]),timeout=.7)
try:
    client.request('GET','/lab-visible-message');response=client.getresponse();body=json.loads(response.read())
    assert response.status==200;print(json.dumps({'ok':True,'status':response.status,'body':body}))
except OSError as error:print(json.dumps({'ok':False,'type':type(error).__name__,'errno':error.errno}))
finally:client.close()
'''
    return json.loads(ns(name,'python3','-c',code,address,str(port)).stdout)

def capture(stack,root,name,interface,expression,label):
    path=root/(label+'.pcap')
    process=subprocess.Popen(['ip','netns','exec',name,'tcpdump','--immediate-mode','-Z','root','-U','-n','-i',interface,'-w',str(path),expression],
        stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
    stack.callback(stop,process)
    deadline=time.monotonic()+3
    while time.monotonic()<deadline:
        assert process.poll() is None,'tcpdump exited before listening'
        if select.select([process.stderr],[],[],.1)[0] and 'listening on' in process.stderr.readline():return process,path
    raise AssertionError('tcpdump did not become ready')
def captured(process,path,ascii=False):
    process.send_signal(signal.SIGINT);_,diagnostic=process.communicate(timeout=3)
    assert process.returncode==0,diagnostic
    result=run('tcpdump','-nn','-vv',*(['-A'] if ascii else []),'-r',str(path))
    return result.stdout

with lab(['left','right']) as (root,stack,setting):
    wire('left','ul','right','ur',['192.0.2.1/30'],['192.0.2.2/30'])
    for name,interface in [('left','ul'),('right','ur')]:ip(name,'link','set',interface,'mtu','1500')
    def overlay(name,vni):
        local,remote,dev,address,mac=('192.0.2.1','192.0.2.2','ul','10.42.0.1/24','02:00:00:00:42:01') if name=='left' else ('192.0.2.2','192.0.2.1','ur','10.42.0.2/24','02:00:00:00:42:02')
        ip(name,'link','add','vx0','type','vxlan','id',str(vni),'local',local,'remote',remote,'dev',dev,'dstport','4789','nolearning')
        ip(name,'link','set','vx0','address',mac,'mtu','1450')
        ip(name,'addr','add',address,'dev','vx0');ip(name,'link','set','vx0','up')
    overlay('left',42);overlay('right',42);server(stack,'right','10.42.0.2',label='overlay')
    tap,path=capture(stack,root,'left','ul','udp port 4789','vxlan')
    successful=request('left','10.42.0.2');assert successful['ok'] and successful['body']['server']=='overlay'
    packets=captured(tap,path,ascii=True)
    assert 'vni 42' in packets.lower() and 'lab-visible-message' in packets and '.4789:' in packets,packets
    details=json.loads(ip('left','-d','-j','link','show','vx0').stdout)[0]
    assert details['linkinfo']['info_data']['id']==42 and details['mtu']==1450
    ip('right','link','delete','vx0');overlay('right',43)
    underlay=ns('left','ping','-n','-c','1','-W','1','192.0.2.2');assert underlay.returncode==0
    mismatch=request('left','10.42.0.2');assert not mismatch['ok'] and mismatch['type']=='TimeoutError'
    ip('right','link','delete','vx0');overlay('right',42)
    recovered=request('left','10.42.0.2');assert recovered['ok']
    print(json.dumps({'healthy_overlay':successful,'underlay_packet_capture':packets,'vni':details['linkinfo']['info_data']['id'],
        'underlay_mtu':1500,'overlay_mtu':details['mtu'],'mismatched_vni_underlay_ping_exit':underlay.returncode,
        'mismatched_vni_overlay':mismatch,'matching_vni_restored':recovered}))
"""
name='xbcatm-netpath-'+uuid.uuid4().hex[:10]
def docker(*args,**kwargs):return subprocess.run(['docker',*args],capture_output=True,text=True,timeout=300,**kwargs)
try:
    with TemporaryDirectory(prefix=name+'-') as directory:
        root=Path(directory);(root/'Dockerfile').write_text(recipe)
        build=docker('build','-t',name,str(root));assert build.returncode==0,build.stderr
        result=docker('run','--rm','-i','--name',name,'--network','none','--cap-add','NET_ADMIN','--cap-add','SYS_ADMIN',
            '--memory','256m','--pids-limit','128',name,'python3','-',input=program)
        assert result.returncode==0,result.stderr
        print(json.dumps(json.loads(result.stdout),sort_keys=True))
finally:
    docker('rm','-f',name);docker('image','rm',name)

The underlay addresses are 192.0.2.1 and 192.0.2.2. Each endpoint has an explicitly configured remote peer and UDP destination port 4789. Setting the port explicitly avoids depending on historical implementation defaults. Both overlay devices initially use VNI 42, fixed MAC addresses, and addresses 10.42.0.1 and 10.42.0.2. This is a small unicast tunnel, not a complete multipoint fabric or EVPN deployment.

The fixture expects an HTTP response with the overlay server identity and a simultaneous underlay capture. Its capture assertions check VNI 42, UDP port 4789 and the literal path /lab-visible-message, not a complete ARP/TCP decode. If actually observed, the visible path would demonstrate unencrypted application bytes in this configuration. No recovered capture establishes that observation here; VXLAN’s lack of native confidentiality is independently specified by the standard.

A veth capture may show apparently incorrect UDP or TCP checksums even when HTTP succeeds. Kernel partial-checksum and encapsulation-offload processing can explain such observations, but a capture alone does not identify the exact stage or prove wire corruption. This fixture neither disables offloads nor performs a complete checksum diagnosis; its historical capture claim is not newly verified here.

Next, the right device is recreated with VNI 43 and the same address and MAC. The expected underlay ping succeeds while overlay HTTP times out; recreating VNI 42 should restore HTTP. The VNI is the deliberate configuration difference, but recreation also resets device state. A timeout alone is not a universal VNI-mismatch signature or proof that every other condition remained identical.

Match evidence to the layer being changed

Inspect the endpoint’s underlay route, selected interface, configured UDP port, VNI, peer mapping, and overlay address. Capture on both the tunnel device and its transport interface when possible: each exposes a different representation of the same communication. Record where the capture was taken, particularly when interpreting packet sizes or checksum warnings that may reflect kernel offload processing.

This example deliberately reserves encapsulation space with MTUs of 1500 and 1450, but tests only a short application exchange. Validate the real path’s packet-size requirements separately before using it for larger workloads. In a production fabric, also review how remote MAC destinations are learned or distributed and how unknown traffic is handled. Those control-plane choices cannot be inferred from this two-endpoint success.

Sources