10 gig vs 2.5 gig which should you buy?

For me the key was to make sure that my home wiring was able to accommodate the upgrade rather than to simply upgrade the communication speed between lab devices.

Rather than retrofitting my home wiring with cat-6 Ethernet, I chose to deploy MOCA adapters. MOCA (Multimedia over Cable Alliance) adapters allow you run 1 or 2.5Gb (depending on the adapter) Ethernet traffic over your home's existing coax cables. It's an inexplicably obscure technology for how well it works compared to something like powerline Ethernet. I'm using ScreenBeam ECB7250 adapters which have a 2.5Gb RJ45 port.

My topology is essentially point-to point, with an office and basement/lab "site". I have an additional adapter for the living room if I want to add a TV/gaming site in the future.

I purchased 2 KeepLink 2.5G 8-port unmanaged switches from AliExpress for $43 each on Black Friday and deployed one at each site.

A USB NIC in the office provides 2.5G service to my workstation. If I want to add mesh network satellites, they can use the wired network for backhaul instead of sharing a radio and halving the available bandwidth.

In the basement I added a 2.5G PCIe NIC to my Synology NAS. My homelab servers are currently a couple of miniPC devices that have integrated 2.5G NICs. These devices plus the 10G WAN port for my ISP router are connected to the basement switch.

The end result was a relatively inexpensive upgrade that provides a reliable low latency 2.5Gb connection to anything in my home that's plugged in to the wired network. I'm able to fully utilize my ISP's 1.5Gb fibre connection. I'm missing a managed switch for VLAN support and a security device that can provide outbound traffic control.

@termv that is a great setup! It sounds like you have made use of pretty inventive ways to connect at 2.5 gig. What kind of latency are you seeing between nodes with the MOCA adapters etc? I am going to have to check into this technology as I haven't heard of it before. Sounds very interesting.

Do you have direct fiber to your home with the 1.5 Gbps connection? I am currently limited to 1 gig here ☹️ 

@brandon-lee I'm seeing about 10-11ms of latency to most sites on the external internet. 3.5-4.5ms of that is through my internal network to my router.

That is indeed with a direct fiber connection. They rolled out fiber to the home here a few years ago and it's possible to get up to a full 8Gbps symmetric connection if you have your house wired for 10GbE 🤯 

@termv that is incredible. Unfortunately, I live in a fairly rural area and don't have access to direct fiber connections, only Coax. I am waiting to see what the local cable company does to keep up with the increasing speeds of competitors etc. That is crazy on the 8 Gbps full symmetrical connection. Would definitely require an entire retrofit of network wiring and gear for most 🤪 but I would be willing to go for it 🤣

I am in the process of upgrading some of the infrastructure to 2.5gig but I'm mostly using it for the interconnects between homelab hosts for now. 

All of the devices on my network wouldn't benefit from anything faster than 1gig yet since they're largely connected via wireless. Inter-transfers within the homelab to/from the new NAS are finally saturating a 1gig link, so I'm looking to tackle that bottleneck first. 

One downside to using older 1L tiny-mini-micro nodes though is that I have to use USB3 to 2.5gig adapters. Not ideal, but it is working so far. I picked up an 8-port Mokerlink 2.5gig switch for interconnecting the "production" parts of the homelab. When I eventually consolidate everything into one rack, I'll be able to use the 10gig uplink on the Mokerlink switch to the UDM Pro. 

My network has undergone a number of transformations. At first, I was using a Netgear Nighthawk r7800 wifi router connected to my cable modem. From the start I didn't want to use Xfinity's equipment so I bought my own. Virtually nothing in the house was hard wired. Then I loaded OpenWRT onto the Netgear router so I could experiment with VLANs. It worked OK for a while, but once I decided to implement VLANs, I pretty quickly moved to a fanless J4125 pfSense device (which was my first device with 2.5gbe NICs), a wifi6 WAP and a 5 port gigabit managed switch to replace the Netgear box. When I ran out of ports on that original TPlink switch, I upgraded to a mokerlink 8 port 2.5gbe managed switch. More out of planning for the future than current need for 2.5gbe at the time. As I began to stand up more wired devices in my home lab, I quickly ran out of ports. I was using one link to connect to the pfsense router, one to connect to the WAP, three to connect to my Proxmox nodes, one to connect to my Ring alarm base station, one to connect to my Pi-star ham radio hotspot, and 2 to connect to my NAS devices. Over time, I upgraded most of my wired NICs to 2.5gbe or in the case of my HPz640 and Synology devices, I purchased 2.5gbe usb NICs. I actually ran out of ports and had to daisy chain in my original 5 port 1gbe switch to service some of the less important items like the ring alarm and the Pi-star hot spot. 

Just within the last month, I decided to upgrade once again. I purchased a Mokerlink managed switch that has 24 2.5gbe ports and 6 SFP+ ports. With the new abundance of ports, I kind of went a little crazy with link aggregation. I now have 2 2.5gbe LACP links from my pfSense box to the switch, and the same between my two NAS machines and the switch. My primary Proxmox node now connects via a 10gbe copper copper connection to a SFP+ transceiver. ALL of this is WAY overkill. My internet connections (I have two because I work at home, and I can't work without internet) max out at 600mbps. So honestly I could get by gigabit ethernet and not notice a difference. But I hope to upgrade to fiber someday, and a lot of this is just for the sake of learning and doing. But on the plus side, I have also set up pfSense with redundant WAN connections. They are set to fail over automatically, and my web servers go out through the T-mobile 5G WAN (via a firewall rule policy) since the upload speed of TMobile is 2X of the upload speed Xfinity gives me. 

Ethernet does not extend outside of my office. The rest of the home is served by my WAP. But it doesn't do too bad. I think it is limited by the gigabit connection from the WAP to the switch. Speed test from my dining room computer to my Proxmox node running speedtest in a docker container averages 850gbps upload and about 790gbps download. 

One downside to using older 1L tiny-mini-micro nodes though is that I have to use USB3 to 2.5gig adapters. 

I have a HP Elitedesk mini 800 G9, and I was able to get a Marvel 10gbe NIC for it, that goes into the option port in the back of the box. 

I am not sure what kind of tiny mini micro nodes you are using. I know HP also makes a 2.5gbe NIC for that option port.

@louie1961 that is great insights. 10 GbE connectivity is often a downside to these mini machines, but great to know there is an option for that here.

@louie1961 Most of mine are Dell Optiplex 7060, one 7070 and the sandbox cluster is all 7050s. 

I believe Dell did make a 10Gig adapter card but I have been hard-pressed to source any of the proprietary add-on modules for these micro PCs sold separately. I've seen a few mentions of adapters on "3rd page search results" that rise to nearly 3 times the cost I paid for the micro PC itself. I wish they were more readily available but it seems Dell only offers them during a new purchases. 

I have two HP EliteDesk G3 nodes but they've not been too stable in the cluster so far. Onboard NIC was unusable with a v5.15 kernel, lasted about 10 minutes with a v6.2 kernel and doesn't register at all in the later v6.5 kernel in ProxMox 8.1x. I have a pair of USB NICs on them now but that isn't ideal. I'm probably going to retire those two PCs and use them for other projects as they seem perfectly fine running Windows 10/11.

@t3hbeowulf Great insights there. I have seen the same unfortunately....10 gig cards and adapters are ridiculously expensive on the third-party market. Hopefully this will change in the future.