My home lab is a bit scattered so I had to turn some parts into a collage.
Top: "Main Cluster" - 5-node ProxMox setup running nearly all house services. Has capacity for experiments but each node can run everything by itself if I need to go into low-power mode. The Synology NAS next to these nodes provides NFS shared storage for the nodes and is isolated from the house data.
Middle: "Sandbox" - 5-node ProxMox cluster running a number of VMs to support Kubernetes experiments. It's a work in progress, 5 litres of tiny-mini-micro power. Each node has a short link to the dedicated switch. Video goes through an HDMI switch for easy node selection. Power goes through a Kasa 6-in-1 power switch with energy monitoring so that I can cycle nodes remotely and watch/log power usage.Â
Bottom: "OG NAS" - First built in 2014, this system is an absolute trooper. 4x 6TB HDDs and dual NICs, this is the primary copy of house data.Â
"New NAS" - Also a work in progress, this is aiming to replace the aging OG-NAS. 5x 6TB HDDs, 64GB RAM, i7-4590K and a pair of 16TB drives for snapshots and additional copies. This is a ProxMox server running TrueNAS in a VM (HBA for drives passed through). It hosts NFS and SMB shares for the house data and other various projects that haven't been migrated off the OG-NAS.Â
Total Idle power draw of everything pictured is about 300W.Â
@t3hbeowulf that is great! You may quite possibly win the aware for the most nodes in the home lab! Thats great that you have a main cluster and a sandbox cluster for testing. I need to carve out more of a dedicated testing environment myself. Hey just curious, what are your specs for the Proxmox nodes? Are those Dell mini's?
Heh, I blame the Raspberry Pi shortage, really digging into ProxMox and eBay for starting that trend this year.Â
The main cluster is a mix of Dell, Lenovo and HP tiny/mini/micro nodes. (One Dell is on my bench for service so only 4 are pictured today) The Dells and Lenovo are Core i7-8700 (6c 12t) processors, the HP nodes are AMD Ryzen 5 3400G (4c 8t) and all have 32GB of RAM, 512GB NVMe storage and a 1TB SATA SSD for shared storage. (Sync'd ZFS pool in ProxMox)
The sandbox cluster was my "oops" cluster. I sent a low offer to a seller with a Lot of 5 thinking there was no way they'd accept, and they did. I then asked my wife for forgiveness. 🙃 They are a homogenous collection of Dell Optiplex 7050s with a Core i5-6500T, 256GB NVMe storage and 32GB RAM each. In almost every case, I took advantage of the default build config where Dell likes to stuff a single stick of memory. I was able to pick up matching pairs to get 32GB without having many sticks left over.
@t3hbeowulf that's great, I love how you took advantage of these and they look like they are working great for you. I am loving the mini PC market more and more as I move more into containerized environments for services. I think this is the future for the most part. Will be great to see what we see come out in 2024.
@malcolm-r @ghaleon @t3hbeowulf check out this potential home lab server....Aoostar NAS with Ryzen 5800U, 6 NVMe, 6 HDD, 10 gig, coming in January, looks to also be barebones if I am reading correctly: https://www.virtualizationhowto.com/community/home-lab-forum/aoostar-nas-with-ryzen-5800u-6-nvme-6-hdd-and-10-gig-network-home-server
That certainly ticks a lot of boxes in one small package. One of the things I struggle with is "Do I get a purpose-built solution and dedicate it to my desired task or do I try to build something from spare parts and fill in gaps as needed?"
If I was just starting out, that's appears to be a nice all-in-one solution with plenty of horsepower and features.
With everything already in the lab, I'd really have to commit to replacing and selling some gear because it's already unwieldy.Â
@t3hbeowulf I am right there with you! I have been holding off on my next round of upgrades for the home lab on the server side as I just haven't quite found the "it solution" that I want to pull the trigger on. I am hoping we see more of these tiny mini PC/NAS type boxes this next year. I think it could help consolidate a lot of things down for me.
Are you looking to stick with your current hardware another year or are you looking at refreshing this next year?
The short answer: I'm going to stick with my current hardware for a while longer because it is MORE than capable of running everything I throw at it.
The longer answer: With the exception of storage, almost everything I've built was purchased used. I have a soft guideline that I don't upgrade unless the equipment is experiencing some sort of imminent or current failure, or I have backed myself into a corner of storage/capacity and need to build out a replacement.Â
It all started with a single "NAS+Desktop" that was pictured at the bottom of the collage in my photos post. A Core i3-3220 dual core processor, 16GB of RAM.Â
It "works", but QuickSync on that processor is slow and 2 cores was constrained for running Portainer and a number of containers in addition to Samba for the file share.Â
I started pricing out replacements when I stumbled upon the micro PCs. I picked up one, then two for "redundancy" and started to get familiar with ProxMox. I picked up a pair of micros from HP because they had Ryzen processors in them and I wanted to toy with passing through the integrated graphics to a VM. Since I had 4 nodes and "even numbers are bad", I picked up a 5th and settled there. Any single node in the cluster is enough to run every service I have, but I spread them out over several nodes to ensure migration/backups keep working. (I test migration frequently) Of course, this didn't solve the original "NAS" problem but it did solve the workload/capacity problem.Â
The NAS in my picture was created because I had just enough other spare parts laying around that I thought: "with a used motherboard and some drives, I can build a new NAS". So I did... and have been slowly migrating storage volumes over to it, while keeping the trusty old Core i3 system online for now.Â
I'm finally in a place where I don't have any spare parts itching to find a use and a homelab complex enough to make hardware purchases an extremely considered action. I'm considering consolidation and using a more disciplined approach to spreading services out over nodes, and also staying off eBay.Â
@t3hbeowulf I really like your setup and you seem to not fall into the typical state that many of us do without thinking through purchases 😆 I know I fall into that category. It sounds like you methodically have built your lab and I really like how you have a "prod" and testing cluster. Hey I wanted to see if you would mind when you have time to create a new home lab forum topic on your prod vs testing cluster and how you test, promote, remediate, etc. I think that would be an interesting read and I would like to see the discussions we could have on that.
@brandon-lee I would definitely like to see @t3hbeowulf write up in more detail on his cluster configurations and test/prod.
I have this task pinned. It's one of my higher priority goals for 2024... better documentation.Â