Troubleshooting NSX VXLAN jumbo frames in nested lab

Recently in troubleshooting nsx vxlan jumbo frames in nested lab environments, I turned up a rather interesting find on network adapter types.

In working recently in a nested VMware environment running three clusters and NSX, I ran into a jumbo frames issue with VXLAN connectivity across the clusters.  I was simulating different WAN/subnet environments using an Untangle appliance as a router in the “remote” cluster site.  It was interesting however, I was able to narrow down the problem was with the Untangle virtual router appliance configuration.  Let’s take a look at troubleshooting NSX VXLAN jumbo frames in nested lab and the issue I ran into.

Troubleshooting NSX VXLAN jumbo frames in nested lab

A little bit of troubleshooting background I ran before diving into the Untangle appliance.  Before looking into further testing, I made sure that at least at a vCenter level, everything was happy in the “installation” page which shows current status of the manager as well as controller VMs.


Moving onto actually pinging the vmkernel nics that are setup for VXLAN traffic.  The process to check via ping the response between VXLAN vmnics can be achieved using the following command:

ping ++netstack=vxlan -d -s 1572 -I vmk1 <vmkernel IP address>


As you can see above, pinging the VXLAN vmkernel IP address with a jumbo frame yields no ping response.  However, in using a regular sized packet, we receive a ping response as you can see below.


So, we know there is a problem with jumbo frames somewhere in the chain.  I have been using Untangle virtual routers in lab environments lately, as they do everything I need them to do and they are basically point and click to setup so it makes it easy for a lab environment.  However, after checking MTUs on vSwitches, etc, I started to suspect that something was going on with the router not passing jumbo frames through even though it was set to pass them as you can see below.


One thing I noticed with the Untangle virtual appliances that you can deploy with the OVA downloaded from Untangle is the network adapter type is set to Flexible.  I found a post of a similar nature concerning jumbo frames not being passed in certain NIC configurations.  What is ironic though the fix for me was to change from Flexible to E1000.  The E1000e wasn’t an option for me even after upgrading the machine hardware to version 11 most likely do to the operating system type.  The author in the link above noted that he had issues with the E1000 as opposed to the E1000e.  So far for me though it has worked.


After changing the adapter to E1000 on the Untangle virtual router appliance.  Pings to the vmkernel nic for VXLAN were successful.


As you can see below, pings to the vmkernel nic with the large packet can return successfully.



Your mileage may vary depending on how you are setting up your lab with NSX.  However, in troubleshooting NSX VXLAN jumbo frames in nested lab for me turned out to be the type of network adapter that is being used on the Untangle appliance.

Subscribe to VirtualizationHowto via Email 🔔

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Brandon Lee

Brandon Lee is the Senior Writer, Engineer and owner at Virtualizationhowto.com and has over two decades of experience in Information Technology. Having worked for numerous Fortune 500 companies as well as in various industries, Brandon has extensive experience in various IT segments and is a strong advocate for open source technologies. Brandon holds many industry certifications, loves the outdoors and spending time with family.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.