Let’s face it, our time in the home lab among other things we have going on in our lives and families may be hard to come by and there are definitely more important things. But, if our time is limited, and we are trying to find low time, high impact changes that can be made in the home lab, there are certain things that don’t take much time, but bring a lot of value to the mix. So, if you have even 2 hours in the home lab, what could you do? Actually quite a bit. We can make sure our workloads are reasonably current, backups are usable, storage is healthy, critical services survived a reboot, and if there are things that need my attention, I can give it focused time. Let’s check out 7 things in order that I would do in 2 hours time in the lab this weekend.
1. Selectively pick what needs updates (15 minutes)
So, my first recommendation here would not be to just start clicking every “update button” that is available. I would take a look at what actually needs updating. This matters more than we might realize. Updating a disposable container is different than upgrading hypervisor hosts or your firewall.
On something like Proxmox it is a good idea to check and see what updates are available:
apt update
apt list --upgradableThis check will immediately give you a feel for what is waiting on being applied. I usually look for updates that involve kernel, QEMU, ZFS, networking components, Ceph (which I am running) and the Proxmox packages themselves. Also, look and see if the host has already received package updates and is just waiting on a reboot.
grep " upgrade " /var/log/dpkg.log | tail -50
So this is a good way to know if you have already packages applied and waiting on a reboot. For Docker images you can take a look at your running containers and the images associated as well with:
docker ps
docker images
With Docker Compose, a lot of the time, we use something like <image>:latest as the tag the image points to. In general to update the image to the latest version of the container, you can run the commands:
docker compose pull
docker compose up -dBut you may not want to do this in the environment against every container you run. If you have some containers that you would rather wait until you have more time to do some testing, then only reserve updating the lowest-hanging fruit.
You can actually update quite a few stacks this way. Or if you use something like Komodo, you can setup an “auto update” procedure that performs a global update across all your stacks if you want to have things on an automatic push button process.
2. Before making any changes, spend some time verifying backups (15 minutes)
Before you click the button to “update or upgrade”, it is wise to make sure that you have the backups that you need across your environment, especially for the workloads you are about to touch. It is good to take a few minutes to verify that your backups ran last night and that you have recent backups and copies of your data in the environment.
I want to make sure that the expected VMs are present in the backups and that again the copies are recent for critical data. Make sure you aren’t running out of capacity in your backup storage. But, just having “green checkboxes” across the board is only the beginning of what you want to see.
I also want to make sure I have copies of the configuration that doesn’t live inside a virtual machine backup. Make sure you have things like the following:
- Docker Compose files
- .env files and application configuration
- reverse proxy configuration
- firewall configuration
- switch configuration
- DNS zones
- scripts
- certificates
- SSH configuration
- critical documentation
Backups of your configs can sometimes be more valuable than an actual backup of your virtual machine itself. For example, I can redeploy a Docker host pretty quickly with a template, backups, etc. But if you need to remember how you had 15 Docker compose stacks configured, that is a different story altogether.
So, a couple of hours is not enough time to redesign your backups, but it does give you time to see if you have something going on with backups that is obviously wrong or that could end up in a data loss situation.
3. Do some storage cleanup (20 minutes)
Also, before you update or upgrade, it is a good idea to make sure your storage is in a good state and you have plenty of free space. There are lots of things that aren’t that difficult to get rid of and they will amount to tons of space oftentimes. These are things like stale Docker images, build caches, persistent volumes that you can get rid of. Also, ISO files and LXC container templates can add up over time. Don’t forget about your Proxmox Server itself or NAS device as well.
In Proxmox, you can look at the filesystem like any other Linux machine using the traditional tool:
df -h
Also, with ZFS which is popular with Proxmox, you can look at your disk space with:
zpool status
zfs listWith Docker, this is often a VERY surprising amount of space involved over time with a system that is in the middle of Docker builds and spinning up and down lots of containers over time. A good place to start is with the command:
docker system df
This will tell you how much disk space is used and how much you can reclaim. If you just want to look at images, you can use something like:
docker image pruneFor build cache:
docker builder pruneJust a note here, that I would be very careful about running the command below, as it could lead to data loss, especially if you have a stopped container that you didn’t know was stopped or in a bad state.
docker system prune -a -fIf you are running Ceph HCI storage in your Proxmox environment, before updating your hosts, it is also a great idea to make sure that your storage is healthy:
ceph status
4. Update the things that you deliberately selected in step 1 (25 minutes)
So, by the time you reach point 4, you know what needs attention. Also, you have verified your backups, and your storage looks healthy from a space perspective, then you can start making changes to the environment.
I think also the order of operations here matters quite a bit. I like to start with low risk systems before I start making any changes with the infrastructure that everything else is dependent on. This might mean I update a few app containers that are low-hanging fruit. I might patch a Linux VM that I need to patch that is out of the infrastructure operations side of my lab.
For low-risk docker compose stacks, you can quickly update with the following commands:
docker compose pull
docker compose up -dThen you can make sure everything starts up as expected with:
docker compose ps
I like to check the logs for the container as well a couple of minutes after updating and starting it back up:
docker compose logs --tail=1005. Reboot something you have been waiting to reboot (15 Minutes)
Maybe you have a Windows Server that has installed updates and is pending a reboot. Maybe it is a Linux server that installed a new kernel. If you have a system that you have been holding off on rebooting, this might be a good option that you have to reboot in a 2 hour Window of time in the home lab this weekend.
After you reboot a Linux server, you can check to see if there are any services that are failing to restart with the command:
systemctl --failed
On a Docker host, make sure the containers that you expect to be running are indeed running:
docker psYou can check Proxmox cluster resources with:
pvesh get /cluster/resources --type vm6. After a reboot, test your critical services (20 minutes)
One you have applied updates and rebooted, or just needed to reboot a server that was pending for a while, test the services from the point of view of an end user. Make sure that you can resolve things like DNS. Make sure apps can resolve hostnames. Are SSL certificates working? Can reverse proxies get to their backends and can your hosts see their provisioned storage? These are important questions.
Also, check and make sure that VMs can communicate across any VLANs configured and that nothing is filtered that is unexpected. For DNS, I might use something like this:
dig myservice.example.comor:
nslookup myservice.example.comFor HTTPS:
curl -I https://myservice.example.comI also like to pull up my Uptime Kuma dashboard as I have all of my critical services and checks in there and firing if something is not quite working right. The reason I like to manually pull up the dashboard is to eyeball things as it may not have reached the alerting threshold quite yet. I can proactively just navigate there and make sure I don’t see anything that is down.
7. Log what you did and what changed (10 minutes)
We are always looking for slick solutions that tell us what changed, etc. But I have found that just a manual note of some sort, maybe stored in Git that I add a log of changes to, works really well. For the last two hours or so, I document what was updated, rebooted, reconfigured, etc.
This doesn’t have to be some kind of formal enterprise change management document. This is a home lab. But you would be surprised at just how powerful this kind of simple logging can be. It could be a Markdown file, a Git commit, a note in your documentation system, or even just a simple text file.
I might record something like:
Home lab maintenance - September 11
Updated:
- pve01 packages
- Docker host OS
- Traefik container
- monitoring stack
Rebooted:
- pve01
- docker02
Checked the following:
- Proxmox backups
- ZFS health
- DNS
- reverse proxy
- application access
Follow up on:
- pve02 kernel update
- old VM snapshot
- backup repository at 78%
- certificate expiring next monthWrapping up
Running a home lab is about making your time count and doing things when you can do them, in an organized and methodical way. As shown here, you can actually accomplish a LOT in just a relatively short period of time. In just a couple of hours, there are definitely tasks that are worthwhile to accomplish that can keep your lab running smooth and updated. How about you? Do you have any of these types of “sprint” type tasks that allow you to accomplish a lot in a short amount of time? Please let us know in the comments.
More in this topic
Google is updating how articles are shown. Don’t miss our leading home lab and tech content, written by humans, by setting Virtualization Howto as a preferred source.
About The Author
