My home lab upgrade...
 
Notifications
Clear all

My home lab upgrades this 2023

30 Posts
7 Users
24 Reactions
2,605 Views
Posts: 24
(@t3hbeowulf)
Eminent Member
Joined: 11 months ago

My home lab is a bit scattered so I had to turn some parts into a collage.

Top: "Main Cluster" - 5-node ProxMox setup running nearly all house services. Has capacity for experiments but each node can run everything by itself if I need to go into low-power mode. The Synology NAS next to these nodes provides NFS shared storage for the nodes and is isolated from the house data.

Middle: "Sandbox" - 5-node ProxMox cluster running a number of VMs to support Kubernetes experiments. It's a work in progress, 5 litres of tiny-mini-micro power. Each node has a short link to the dedicated switch. Video goes through an HDMI switch for easy node selection. Power goes through a Kasa 6-in-1 power switch with energy monitoring so that I can cycle nodes remotely and watch/log power usage. 

Bottom: "OG NAS" - First built in 2014, this system is an absolute trooper. 4x 6TB HDDs and dual NICs, this is the primary copy of house data. 

20231219 220611 COLLAGE

"New NAS" - Also a work in progress, this is aiming to replace the aging OG-NAS. 5x 6TB HDDs, 64GB RAM, i7-4590K and a pair of 16TB drives for snapshots and additional copies. This is a ProxMox server running TrueNAS in a VM (HBA for drives passed through). It hosts NFS and SMB shares for the house data and other various projects that haven't been migrated off the OG-NAS. 

PXL 20231220 024843391

Total Idle power draw of everything pictured is about 300W. 

Reply
1 Reply
Brandon Lee
Admin
(@brandon-lee)
Joined: 14 years ago

Member
Posts: 395

@t3hbeowulf that is great! You may quite possibly win the aware for the most nodes in the home lab! Thats great that you have a main cluster and a sandbox cluster for testing. I need to carve out more of a dedicated testing environment myself. Hey just curious, what are your specs for the Proxmox nodes? Are those Dell mini's?

Reply
Posts: 24
(@t3hbeowulf)
Eminent Member
Joined: 11 months ago

Heh, I blame the Raspberry Pi shortage, really digging into ProxMox and eBay for starting that trend this year. 

The main cluster is a mix of Dell, Lenovo and HP tiny/mini/micro nodes. (One Dell is on my bench for service so only 4 are pictured today) The Dells and Lenovo are Core i7-8700 (6c 12t) processors, the HP nodes are AMD Ryzen 5 3400G (4c 8t) and all have 32GB of RAM, 512GB NVMe storage and a 1TB SATA SSD for shared storage. (Sync'd ZFS pool in ProxMox)

The sandbox cluster was my "oops" cluster. I sent a low offer to a seller with a Lot of 5 thinking there was no way they'd accept, and they did. I then asked my wife for forgiveness. 🙃 They are a homogenous collection of Dell Optiplex 7050s with a Core i5-6500T, 256GB NVMe storage and 32GB RAM each. In almost every case, I took advantage of the default build config where Dell likes to stuff a single stick of memory. I was able to pick up matching pairs to get 32GB without having many sticks left over.

Reply
1 Reply
Brandon Lee
Admin
(@brandon-lee)
Joined: 14 years ago

Member
Posts: 395

@t3hbeowulf that's great, I love how you took advantage of these and they look like they are working great for you. I am loving the mini PC market more and more as I move more into containerized environments for services. I think this is the future for the most part. Will be great to see what we see come out in 2024.

Reply
Brandon Lee
Posts: 395
Admin
Topic starter
(@brandon-lee)
Member
Joined: 14 years ago

@malcolm-r @ghaleon @t3hbeowulf check out this potential home lab server....Aoostar NAS with Ryzen 5800U, 6 NVMe, 6 HDD, 10 gig, coming in January, looks to also be barebones if I am reading correctly: https://www.virtualizationhowto.com/community/home-lab-forum/aoostar-nas-with-ryzen-5800u-6-nvme-6-hdd-and-10-gig-network-home-server

Reply
2 Replies
(@t3hbeowulf)
Joined: 11 months ago

Eminent Member
Posts: 24

That certainly ticks a lot of boxes in one small package. One of the things I struggle with is "Do I get a purpose-built solution and dedicate it to my desired task or do I try to build something from spare parts and fill in gaps as needed?"

If I was just starting out, that's appears to be a nice all-in-one solution with plenty of horsepower and features.
With everything already in the lab, I'd really have to commit to replacing and selling some gear because it's already unwieldy. 

Reply
Brandon Lee
Admin
(@brandon-lee)
Joined: 14 years ago

Member
Posts: 395

@t3hbeowulf I am right there with you! I have been holding off on my next round of upgrades for the home lab on the server side as I just haven't quite found the "it solution" that I want to pull the trigger on. I am hoping we see more of these tiny mini PC/NAS type boxes this next year. I think it could help consolidate a lot of things down for me.

Are you looking to stick with your current hardware another year or are you looking at refreshing this next year?

Reply
Posts: 24
(@t3hbeowulf)
Eminent Member
Joined: 11 months ago

The short answer: I'm going to stick with my current hardware for a while longer because it is MORE than capable of running everything I throw at it.

The longer answer: With the exception of storage, almost everything I've built was purchased used. I have a soft guideline that I don't upgrade unless the equipment is experiencing some sort of imminent or current failure, or I have backed myself into a corner of storage/capacity and need to build out a replacement. 
It all started with a single "NAS+Desktop" that was pictured at the bottom of the collage in my photos post. A Core i3-3220 dual core processor, 16GB of RAM. 
It "works", but QuickSync on that processor is slow and 2 cores was constrained for running Portainer and a number of containers in addition to Samba for the file share. 

I started pricing out replacements when I stumbled upon the micro PCs. I picked up one, then two for "redundancy" and started to get familiar with ProxMox. I picked up a pair of micros from HP because they had Ryzen processors in them and I wanted to toy with passing through the integrated graphics to a VM. Since I had 4 nodes and "even numbers are bad", I picked up a 5th and settled there. Any single node in the cluster is enough to run every service I have, but I spread them out over several nodes to ensure migration/backups keep working. (I test migration frequently) Of course, this didn't solve the original "NAS" problem but it did solve the workload/capacity problem. 

The NAS in my picture was created because I had just enough other spare parts laying around that I thought: "with a used motherboard and some drives, I can build a new NAS". So I did... and have been slowly migrating storage volumes over to it, while keeping the trusty old Core i3 system online for now. 
I'm finally in a place where I don't have any spare parts itching to find a use and a homelab complex enough to make hardware purchases an extremely considered action. I'm considering consolidation and using a more disciplined approach to spreading services out over nodes, and also staying off eBay. 

Reply
2 Replies
Brandon Lee
Admin
(@brandon-lee)
Joined: 14 years ago

Member
Posts: 395

@t3hbeowulf I really like your setup and you seem to not fall into the typical state that many of us do without thinking through purchases 😆 I know I fall into that category. It sounds like you methodically have built your lab and I really like how you have a "prod" and testing cluster. Hey I wanted to see if you would mind when you have time to create a new home lab forum topic on your prod vs testing cluster and how you test, promote, remediate, etc. I think that would be an interesting read and I would like to see the discussions we could have on that.

Reply
(@techguy2023)
Joined: 11 months ago

New Member
Posts: 1

@brandon-lee I would definitely like to see @t3hbeowulf write up in more detail on his cluster configurations and test/prod.

Reply
Posts: 24
(@t3hbeowulf)
Eminent Member
Joined: 11 months ago

I have this task pinned.  It's one of my higher priority goals for 2024... better documentation. 

Reply
Page 2 / 3