Have you ever sunk a weekend into fixing a Linux system that broke after you applied an update or lost track of configuration tweaks that you made? Maybe you forgot to take a snapshot and you are left trying to figure out exactly what broke and why. With this, you have experienced a “mutable” server. That is one of the characteristics of a traditional server. They allow you to change anything in the OS at any time you want. That can be powerful but it can also lead to issues like broken OS’s after applying updates. Modern OS’s especially for containers are following the trend of becoming “immutable”. If you want to have the most stability, automation tools, and spend a lot less time babysitting servers, immutable operating systems are awesome.
What is an immutable operating system?
First of all, let’s talk about what exactly an immutable operating system is. Immutable OS’s are ones where the system files are read-only and updated as a single image rather than through individual package updates like we are used to with most Linux and Windows operating systems.
So, overall, the system never drifts. It is updated by swapping out the entire versioned imaged during the upgrade. By design, it means you can’t change system files or accidentally overwrite critical configurations. So, everything that is important to the operating system is locked down.
If an update fails, you can roll back instantly to the last version of the operating system. This may not seem like it would be beneficial, but it makes your servers much more predictable and reliable and overall it makes your servers rock solid.
Examples of immutable OSes
Examples of immutable OS platforms include:
- Fedora CoreOS
- Flatcar Container Linux
- openSUSE MicroOS
- Ubuntu Core
- Vanilla OS
Each takes a slightly different approach, but the idea for all of these is roughly the same. Make your OS unbreakable and secure and do this at scale.
Why immutability fits in your home lab
Home labs are almost like their own living breathing things that grow with time. It’s funny how you tend to start with one machine, add a few VMs, Docker and before long, you have built a small data center. One of the downsides to experimentation is that you have configuration drift.
Things get tweaked, packages installed, beta software installed, and you end up with servers that act differently than they did in the beginning.
Immutable servers help to solve this problem. Using a server operating system that is fully immutable helps you to do the following:
- Have identical configurations across your nodes
- Roll out atomic upgrades where the entire OS is replaced instead of patched
- Eliminate most of the “it worked yesterday” troubleshooting
- You can restore a node instantly
When your base system can’t be modified by accident, it becomes easier to automate everything else. For example, you can focus your configuration management on deploying containers, applications, or VMs. This keeps you from having to worry about keeping the underlying OS patched, etc.
Security advantages
There are many advantages from a security standpoint with locking down the OS root filesystem. Attackers can modify core binaries. Rootkits can’t be installed that are persistent. SELinux and systemd hardening are enabled by default and aren’t disabled as these are persistent. Also, the system is a “signed image” meaning it becomes very difficult to tamper with the OS layer.
So, for home labs that are exposed to the internet using a reverse proxy or some type of remote access tool, immutability is an extra layer of protection. So, if you think about it, even if an attacker gains limited access, a reboot restores a clean state and any malicious changes will disappear.
This is the problem with “mutable” server operating systems
When you look at traditional server operating systems like Debian, Ubuntu Server, or CentOS Stream is that these allow full write access to every file. This kind of configurability is something that most admins expect. But, when you think about it, over time, it introduces risk.
Risk of things breaking, security risks, manual edits are all byproducts of having servers and operating system disks that can be written to or changed. It makes it almost impossible to have a known-good state or setup of that particular server.
Usually in the home lab, downtime doesn’t necessarily cost you money as it would a business, but it does cost you your time. If you are like me, you spend hours every week in the home lab configuring and changing things. The mutable server operating system model complicates things like backup and recovery as well.
Servers that are mutable become “snowflakes” that are almost impossible to reproduce that exact known-good state. Immutable servers solve this by versioning the entire system image and not just the user data.
Containers and immutable server operating systems are perfect together
When it comes to running containers and container hosts, immutable server operating systems are a great choice for a container host. When you think about it, Docker, Podman, and Kubernetes already have an immutable model where the container image doesn’t change after deployment.
This makes it easy to tear down and rebuild containers. But it seems counterproductive if your container host is constantly drifting and changing underneath your containers.
When you run Docker on an immutable OS like Fedora CoreOS or something like Flatcar Linux, it provides a clean separation of responsibilities. The OS manages the container runtime and system updates atomically.
Then, your containers handle the actual app workloads themselves. When you pair the two together, you get predictable deployments and your upgrades will go much more smoothly, in fact, will be painless.
Example upgrade scenario
Imagine running a three-node Docker Swarm cluster on Fedora CoreOS. Each of your nodes will update automatically using rpm-ostree in the background. They reboot into a new image, and come back online configured identically. Your containers don’t care about the update because their environment never changed. This is the kind of scenario that makes immutable server operating systems great for the home lab.
Setting up an immutable server in your home lab
If you want to try this approach, Fedora CoreOS is one of the easiest starting points, with my guide. In Proxmox, one of the easiest ways I have found to get Fedora CoreOS working is a project that is located here: GECO-IT/fedora-coreos-proxmox. When you clone down the repo you can run a script that will pull the Fedora CoreOS installation down and you just need to tweak a few things to your liking. One thing to mention the project is super old, but still works.
Configuration with this project is handled through an ignition file, but you have an easy way to control this with the Proxmox cloud-init settings. I did a walkthrough of this project and spinning this up some time ago: Proxmox Containers with Fedora CoreOS Install.
Here is a quick run through of how to use this. You can install git on your Proxmox host or clone down the repo and manually copy the files over. First download the scripts:
cd /opt
git clone https://github.com/GECO-IT/fedora-coreos-proxmox.git
cd fedora-coreos-proxmoxNext, configure the setup:
nano vmsetup-config.shEdit these settings in that file:
TEMPLATE_VMID="<VM ID goes here>"TEMPLATE_VMSTORAGE="<VM storage goes here>"SNIPPET_STORAGE="local"- (Optional) Update
VERSIONto latest CoreOS version
After the template is created, edit the cloud-init settings for the template VM. Configure these using Proxmox’s normal Cloud-Init interface (SSH keys, hostname, network settings)
Once you have the template VM like you want it and cloud-init settings configured, you can just clone this one (fcos-tmplt) in the screenshot above to another VM.
Viewing and triggering updates in CoreOS
Fedora CoreOS and Flatcar use rpm-ostree to manage OS versions. When a new update becomes available, the system downloads the new image alongside the existing one. It then prepares it, and switches the boot reference only after it is validated. If something goes wrong, the OS simply reboots back into the last known working image. The best I can relate it to is like a network switch firmware upgrade. You install into the secondary and then when it completes you boot from that image. If something happens, then you just rollback to the other image.
It also supports automated reboots and version pinning. You can schedule updates to happen overnight even if you want and wake up to a fully patched environment without fear of things breaking.
Below are a few of the commands you can use with rpm-ostree.
# Manually trigger an update
sudo rpm-ostree upgrade
# Reboot into new version after staging
sudo systemctl reboot
# Rollback to previous version
sudo rpm-ostree rollback
# See update history
rpm-ostree status -v
# Pin current deployment (prevent automatic removal)
sudo ostree admin pin 0
# See what packages are layered on top
rpm-ostree status --verbose
This process works very well with automation platforms like Terraform and others where you can control the image cycling process and easily deploy new immutable server operating systems.
When you may not want to use an immutable server operating system
There are some situations that you probably wouldn’t want to use immutable servers for. If you want to deploy a server for developing or tinkering with an application and its configuration files directly on the server, or use it for experimenting with new packages, immutable server read-only filesystems wouldn’t allow you to do that kind of work.
You could get around this by working with automation or container builds to deploy changes to your immutable server. Also something to consider with immutable server operating systems is that not every driver or peripheral will work with this kind of setup. Fedora CoreOS and Flatcar as examples are tweaked for use with headless operation. Desktop hardware or special GPU setups may be complicated to get to work. However, for most headless servers and lab nodes, immutable server operating systems work well.
Best immutable OS choices in 2025
- Fedora CoreOS: Best overall for Docker and general container workloads. Automatic updates and Ignition make it easy to deploy and maintain.
- Flatcar Container Linux: Minimalist and stable. Great for clusters or edge deployments.
- openSUSE MicroOS: Excellent for those who want a traditional package manager and desktop option alongside immutability.
- Ubuntu Core: Focused on IoT and appliance setups but ideal for small always-on lab services.
- Vanilla OS: A user-friendly desktop variant based on immutability concepts, great for hybrid environments.
Below are the options for Fedora CoreOS install, which are pretty broad:
Wrapping up
Immutability is a great capability that we have especially become familiar with in the world of backups. This is one of the best capabilities in general against things like ransomware. However, immutability doesn’t stop there. We can actually use this to our advantage as we have seen with the server operating system itself. This way, the operating system functions more like an immutable container. When you upgrade, you are essentially bringing down an entire atomic image change so that it is upgraded as one validated image. What about you, are you playing around with any immutable server operating systems in the home lab or even production? Let me know in the comments.
About The Author
