One of the best upgrades that I have undergone in my home lab is the move to 10 GbE networking. Without a doubt, that was one of the best performance upgrades I have made outside of NVMe storage. With 10 Gbps, VM migrations complete quickly, storage servers hand out files much more quickly, and backups seem to burn through the data almost instantly compared to 1 Gbps. However, despite all the good things that come with upgrading to 10 Gbps network connectivity, it does also come with a few surprises. Some are pleasant, but others are frustrating. Take a look at these 12 things worth knowing before making the leap to a 10 gig home lab upgrade.
1. DAC cables are cheaper but not always the right choice
When going up to 10 GbE connectivity and the SFP cages, one of the things that changes is how you connect up your devices. Most people begin their journey into 10 Gbps with passive DAC cables. The reason for this is they are relatively inexpensive. However, there are downsides to using DAC cables across different switches and NICs.
One of those downsides is they are not always universally compatible with all switches and vendors. Some cheaper DAC cables may not work with name brand switches or these may recognize these as “unofficial” cables that aren’t supported.
Also, the limit is 10 meters with passive DAC cables. So roughly you can go 33 feet or so with a DAC cable and that is it, so keep this in mind. There is another option if you need to go longer or run into issues with DAC cables and these are AOC cables and these stand for “Active Optical Fiber Cable”. These are basically fiber instead of the copper of the DAC cables and are active transceivers on both ends.
2. Mellanox ConnectX cards really are the best value
When it comes to your custom built servers you may be running, upgrading to 10 gig will require that you have a 10 gig NIC card to start with. Mellanox ConnectX cards are a brand and type that have a good reputation for working fairly consistently across different systems. These also work out of the box with Linux and are affordable.
You can definitely get cheaper cards, but cheaper cards tend to have much more challenging support when it comes to drivers. Also, you may run into flaky connectivity or instability altogether. So, do yourself a favor and get something that is more widely used as the few dollars extra you might pay up front will pay dividends down the road in terms of compatibility and problems.
3. Your switch will matter more than you expect
Most home lab upgrades may include switch models like the MikroTik CRS309, UniFi USW Pro Aggregation switch, QNAP QSW series, and Netgear XS708T. One thing to note is that 10 Gig switches behave very differently depending on their internal hardware.
Some use ASIC forwarding which is extremely fast. Others route through the CPU which caps performance when features like VLANs or ACLs are enabled. Be sure to research your switch you are considering to see how it is architected in terms of hardware and how packets are pushed.
I can share switches that I have personally had experience with in the home lab that worked really well and I saw good performance with.
- Netgear XS708T – this is the first 10 gig switch I had in my lab, worked great, but long in the tooth now. I have linked to the “S” model.
- Ubiquiti EdgeSwitch 16 10 gig switch – This is my current aggregation switch but another switch that is getting older.
- Cisco SG350X-28 – this one of my first core switches, for edge devices, but it has (4) 10 gig SFPs
These are just a few that I have used personally and there are many other good ones out there.
4. SFP plus and RJ45 10G-Base-T are not equal
So, this I think is a big one. There are essentially three ways you can link 10 gig equipment. The obvious is using SFPs with a DAC cable or AOC cable, or a pure fiber patch with LC connections. But, you can also use CAT6E with 10G-Base-T tranceivers like these: 10Gb SFP+ to RJ-45 Module Transceiver.
These do work great as they are able to use your existing copper cabling that you would use for any other endpoint as long as it is CAT6A (CAT6 will work technically but only up to 55 meters at which point attenuation and crosstalk kicks in).
However, the main downside to these are heat. These transceivers can get hot to the touch and not joking they are uncomfortably hot. This also means that consume a lot more power as you can imagine when running them. They may be a necessary evil in your lab if you have a server that is a BASE-T connection for the 10 gig uplink, but keep this in mind. I would always go DAC, AOC, or LC/LC fiber cable if you can when uplinking 10 gig connections.
5. Jumbo frames do not magically increase performance
I think many like myself who went through the virtualization era were indoctrinated with jumbo frames on everything that was a 10 gig connection, especially with storage networks. They definitely have their place and do make a difference in CPU usage when network traffic can be sent in larger chunks. However, jumbo frame configuration must match across all devices in the leg of communication for that particular traffic.
A common misconception is that jumbo frames makes all traffic “faster.” However, the truth is, for many workloads, just using the standard 1500 MTU is fine unless you are running specialized workloads like iSCSI over dedicated links in your virtualization environment.
6. Storage often becomes the real bottleneck
The funny thing about upgrades is that they just shift the bottleneck. If you bottleneck was the network, when you upgrade the network, it shifts that bottleneck somewhere else. It means that after upgrading your network throughput, you will likely expose storage as the new bottleneck, especially if you are still running spinning disks.
A spinning disk NAS with RAID 5 won’t be able to saturate a 10 gig link. Even SATA SSD arrays often fall short when handling mixed workloads. NVMe storage and ZFS caching like L2ARC and SLOG become much more important so as to keep up with the additional network bandwidth.
7. CPU offload and driver support matter
Another area that starts to matter is NICs that can offload checksum calculations and TCP segmentation to the card itself. When not offloaded, this work gets pushed to the CPU. On a hypervisor and pure storage servers, you can notice the difference in performance when you have offloading enabled vs not. Without offloading you can see high CPU usage during large transfers.
8. VM migrations and backups can show weak spots
Going back to this notion of revealing new bottlenecks, once you have things running at 10 gig speeds, things like live migrations, cluster sync operations, and backup jobs may finish very quickly. But, they may not still as the weak links in the chain simply move to other parts of the system that need to be upgraded. Now, you might have weak points like slow storage as we mentioned above, underpowered NAS devices, or even old cabling that isn’t really designed to run at new specifications.
9. Heat can become an issue with 10 Gig devices
You can tell a difference usually with network switches that run at 10 gig speeds vs 1 gig speeds when it comes to heat. 10 gig switches can run very warm. Also, as we have already mentioned, if you are using the 10G-Base-T transceivers, these get uncomfortably hot to the touch even.
If you are running compact mini PCs with dual 10 gig ports, these can run hot under sustained transfer loads. So, make sure you have proper ventilation even more so with 10 gig networking upgrades.
10. Cabling quality becomes much more important
We mentioned this as part of the spec for running 10 gig over copper cabling requires a certain specification to be certified to run at 10 gig speeds. CAT6 can do it, but at shorter distances. I have even had some experience with running 10 gig over CAT5e and it will work, again for short distances.
Keep in mind that CAT6A is the spec of structured copper cabling that is rated for 10 gig speeds and are your best best to be compliant with the specification. Fiber of course simplifies things and gives you ton of headroom in terms of what it can do, but it can create its own challenges as well.
11. Not all operating systems handle 10 Gig the same way
You might think the network is something that is outside of the operating system. However, not all operating systems handle 10 gig the same way as others. Linux distros usually work well with 10 gig interfaces. Windows Server may need to have specialized drivers or firmware updates to perform the best possible and be stable. VMware ESXi has historically not been friendly with Realtek network adapters, Intel and Broadcom adapters are usually the ones you can get to work without any issues.
FreeBSD based storage systems like TrueNAS may need to have specialized drivers or tuning to perform acceptably. Proxmox tends to work with most network adapters, but your mileage can vary here. If a card is compatible with Debian, this usually means it will work with Proxmox out of the box.
12. The need for new network design decisions
One thing calls for another. Usually, when you have your 10 gig links in place, it may call for new design decisions. Things like introducing VLANs for separating high bandwidth workloads, LACP bonds, spine and leaf topologies, or maybe even dual homing certain systems. A speed increase usually requires a network redesign and planning of topology changes as you usually introduce a new switch and move links over as you migrate.
Wrapping up
I think a 10 Gig home lab upgrade was absolutely a turning point for me in the home lab. It makes for having the backbone you need to play around with enterprise technologies like software-defined storage and other things. Just keep in mind, like any type of change, there are usually unexpected things that come up and these are just a few of the things that I didn’t really realize before moving to 10 gig networking in the home lab. How about you? What challenges are you running into? Is a 10 gig upgrade in store for your home lab network?
Google is updating how articles are shown. Don’t miss our leading home lab and tech content, written by humans, by setting Virtualization Howto as a preferred source.
About The Author
