10 gig vs 2.5 gig which should you buy?

For me the key was to make sure that my home wiring was able to accommodate the upgrade rather than to simply upgrade the communication speed between lab devices.

Rather than retrofitting my home wiring with cat-6 Ethernet, I chose to deploy MOCA adapters. MOCA (Multimedia over Cable Alliance) adapters allow you run 1 or 2.5Gb (depending on the adapter) Ethernet traffic over your home's existing coax cables. It's an inexplicably obscure technology for how well it works compared to something like powerline Ethernet. I'm using ScreenBeam ECB7250 adapters which have a 2.5Gb RJ45 port.

My topology is essentially point-to point, with an office and basement/lab "site". I have an additional adapter for the living room if I want to add a TV/gaming site in the future.

I purchased 2 KeepLink 2.5G 8-port unmanaged switches from AliExpress for $43 each on Black Friday and deployed one at each site.

A USB NIC in the office provides 2.5G service to my workstation. If I want to add mesh network satellites, they can use the wired network for backhaul instead of sharing a radio and halving the available bandwidth.

In the basement I added a 2.5G PCIe NIC to my Synology NAS. My homelab servers are currently a couple of miniPC devices that have integrated 2.5G NICs. These devices plus the 10G WAN port for my ISP router are connected to the basement switch.

The end result was a relatively inexpensive upgrade that provides a reliable low latency 2.5Gb connection to anything in my home that's plugged in to the wired network. I'm able to fully utilize my ISP's 1.5Gb fibre connection. I'm missing a managed switch for VLAN support and a security device that can provide outbound traffic control.

@brandon-lee I'm seeing about 10-11ms of latency to most sites on the external internet. 3.5-4.5ms of that is through my internal network to my router.

That is indeed with a direct fiber connection. They rolled out fiber to the home here a few years ago and it's possible to get up to a full 8Gbps symmetric connection if you have your house wired for 10GbE 🤯 

I am in the process of upgrading some of the infrastructure to 2.5gig but I'm mostly using it for the interconnects between homelab hosts for now. 

All of the devices on my network wouldn't benefit from anything faster than 1gig yet since they're largely connected via wireless. Inter-transfers within the homelab to/from the new NAS are finally saturating a 1gig link, so I'm looking to tackle that bottleneck first. 

One downside to using older 1L tiny-mini-micro nodes though is that I have to use USB3 to 2.5gig adapters. Not ideal, but it is working so far. I picked up an 8-port Mokerlink 2.5gig switch for interconnecting the "production" parts of the homelab. When I eventually consolidate everything into one rack, I'll be able to use the 10gig uplink on the Mokerlink switch to the UDM Pro. 

My network has undergone a number of transformations. At first, I was using a Netgear Nighthawk r7800 wifi router connected to my cable modem. From the start I didn't want to use Xfinity's equipment so I bought my own. Virtually nothing in the house was hard wired. Then I loaded OpenWRT onto the Netgear router so I could experiment with VLANs. It worked OK for a while, but once I decided to implement VLANs, I pretty quickly moved to a fanless J4125 pfSense device (which was my first device with 2.5gbe NICs), a wifi6 WAP and a 5 port gigabit managed switch to replace the Netgear box. When I ran out of ports on that original TPlink switch, I upgraded to a mokerlink 8 port 2.5gbe managed switch. More out of planning for the future than current need for 2.5gbe at the time. As I began to stand up more wired devices in my home lab, I quickly ran out of ports. I was using one link to connect to the pfsense router, one to connect to the WAP, three to connect to my Proxmox nodes, one to connect to my Ring alarm base station, one to connect to my Pi-star ham radio hotspot, and 2 to connect to my NAS devices. Over time, I upgraded most of my wired NICs to 2.5gbe or in the case of my HPz640 and Synology devices, I purchased 2.5gbe usb NICs. I actually ran out of ports and had to daisy chain in my original 5 port 1gbe switch to service some of the less important items like the ring alarm and the Pi-star hot spot. 

Just within the last month, I decided to upgrade once again. I purchased a Mokerlink managed switch that has 24 2.5gbe ports and 6 SFP+ ports. With the new abundance of ports, I kind of went a little crazy with link aggregation. I now have 2 2.5gbe LACP links from my pfSense box to the switch, and the same between my two NAS machines and the switch. My primary Proxmox node now connects via a 10gbe copper copper connection to a SFP+ transceiver. ALL of this is WAY overkill. My internet connections (I have two because I work at home, and I can't work without internet) max out at 600mbps. So honestly I could get by gigabit ethernet and not notice a difference. But I hope to upgrade to fiber someday, and a lot of this is just for the sake of learning and doing. But on the plus side, I have also set up pfSense with redundant WAN connections. They are set to fail over automatically, and my web servers go out through the T-mobile 5G WAN (via a firewall rule policy) since the upload speed of TMobile is 2X of the upload speed Xfinity gives me. 

Ethernet does not extend outside of my office. The rest of the home is served by my WAP. But it doesn't do too bad. I think it is limited by the gigabit connection from the WAP to the switch. Speed test from my dining room computer to my Proxmox node running speedtest in a docker container averages 850gbps upload and about 790gbps download.