I wanted to post an update after using Flatcar for some months now and what has changed in terms of workflows and actual benefits or negatives from moving to Immutable Linux in the home lab for running my containerized workloads. This has been a really great but significant pivot for me. At first, I was interested in the security and reliability benefits that people often talk about when discussing immutable operating systems. What I did not fully appreciate was how much it would change the way I operated my home lab. Looking back now, the biggest changes were not the ones I expected. Some things became dramatically easier. Other things became more challenging.
Quick overview of Immutable Linux
First, just briefly, what is an immutable Linux distro? Immutable means that you can’t change the underlying operating system configuration once it is deployed. The core system files are read-only. So, instead of installing packages directly on the running system, updates are delivered as “images” that are swapped in place and activated on boot. I like to think of this much like we update network switches with the primary and secondary image locations.
This gives you much better reliability and security as well as consistency of your known good state. Examples of immutable OS’s include Flatcar Linux, Fedora CoreOS, Talos Linux, and Bottlerocket.
I stopped treating servers as pets
This probably the biggest change from deploying and managing servers like I used to manage them when deploying Ubuntu Server 24.04. When a Linux server is immutable, it means that you can’t change things like you used to change them with traditional Linux installations like Ubuntu.
With a traditional server as you know, you can develop the habit of SSH’ing into your host, installing a package, modding a config file, tweaking services, etc. Then, a few months later, you might not remember what you changed or why you changed it. This is the difficulty with doing things this way.
With Flatcar and immutable Linux in general, that workflow is not one that is feasible any longer. Once you have deployed the new operating system, you can’t just install packages and things of that nature the same as you would have done with Ubuntu Server, etc. This sounds like a subtle shift, but it actually completely changes the way you can do things.
Today, if I need to rebuild one of my Flatcar hosts, I can deploy a fresh instance and have it configured automatically from code, which is amazing. This gives you a level of repeatability and standardization that most of us don’t have until we go this direction.
Git becomes a LOT more important
This one is definitely expected since the whole premise around immutable Linux operating systems is that they are configured using config files that you can record in your infrastructure as code environments. I was already used to storing my Docker Compose files and other things like Kubernetes manifests in Git, so immutable operating systems fell right into place with this same workflow.
All of the changes that I would have previously made interactively on another Linux distro, are now made through configuration files that live in a Git repo. These include things like the following:
- Network config in the ignition file
- My systemd services are defined in Ignition
- My Docker Compose stacks are stored in Git repositories
- Deployment processes revolve around version control
The great thing about this is that my configs are basically creating documentation automatically. Every modification has a git history. Then also, your configuration can be rolled back using native Git processes. Every deployment is reproducible. When something breaks, I can usually trace exactly when a configuration changed and why.
Rebuilding is easier than troubleshooting
This one definitely plays into the whole “pets vs cattle” ideology. When you can just as easily rebuild as troubleshoot, who wouldn’t want to rebuild? It is just easier, less stressful, and you get a fresh new instance. It is a win/win situation all the way around.
Back in the day of legacy ways of doing things, if a host had an issue, I would start digging through logs, inspecting configurations, and trying to restore the system to a healthy state. Immutable Linux shifts that mindset in a great way. When the operating system itself is mostly static and configuration is defined as code, rebuilding becomes much less painful.
I would just pull down a fresh flatcar linux installation, deploy in Proxmox and then ignite it with my ignition file in git.
It sounds extreme and doesn’t feel right at first, until you experience the benefits firsthand and actually see that a full rebuild is just as painless. My servers can now be recreated from an Ignition file and container workloads can be redeployed automatically, rebuilding becomes a practical operational strategy rather than a last resort.
This approach mirrors what many organizations do in cloud-native environments, and I have found it works surprisingly well in the home lab too.
SSH became less important
I still use SSH regularly across my home lab for various purposes. However, I have found that I use it a whole lot less than I did before with traditional Linux operating systems. On traditional Linux distros, SSH is usually the primary management interface. Every change starts with opening a shell session to your home lab servers
Below, you can see that I can still SSH to my Flatcar hosts, but I don’t need to nearly as much as I did with Ubuntu Server, etc.
After moving to immutable Linux, most of the routine tasks I do have shifted away from SSH entirely. Now, when I make a change, I do that in Git and let my pipeline run and push changes to my new boxes. Infrastructure changes happen through configuration management. Host provisioning happens through Ignition.
Instead of logging into a server and making changes, I am modifying source files and redeploying configurations. The result is more straightforward operations and fewer undocumented changes to the environment. I still SSH into systems when troubleshooting or looking at certain things, but it is no longer my default approach for managing my infrastructure.
Updates are now less stressful, set it and forget it
One of the great benefits too of an immutable Linux operating system is that I now have basically “no stress” updates. Immutable Linux operating systems update differently than your traditional OS installations like Ubuntu Server.
Operating systems like Flatcar Linux that I use has an “image-based” update model. So instead of updating hundreds of individual packages, the system gets its update as one complete unit. This results in much less eventful update operations. It doesn’t eliminate 100% of the update issues, but it does give you even more peace of mind, especially when combined with Nebraska server with Flatcar.
Some software became harder to deploy
There is an aspect to this that you need to consider. While the deployment process in code is fantastic, there is the double edged sword to consider that once you deploy, you CAN’T go and install things super easily. You CAN technically go back and install things, but for the most part you have to do this using sysext something like Flatcar.
Also, you can’t install third-party “agents” as such for your endpoints running immutable Linux. I had to think my way around that since I did use a few agents here and there for monitoring like with Pulse and Netdata. But a great workaround is to run your agents inside containers.
Below, I have my Pulse monitoring agent running as a Systemd service which is persistent despite the immutable operating system underneath.
So, long story short, getting the agent running reliably with things like monitoring required more planning than it would have on a standard Ubuntu server. Once configured, it worked perfectly, but the initial setup needs to have a different mindset. This is one area where immutable Linux still has a learning curve I think.
Disaster recovery is now much better
Before, I didn’t really know what the config was on my Ubuntu servers that I had running in my “production” home lab. That is because these turn into “snowflakes” like we have described above due to tons of one-off SSH commands run on the system to customize and install dependencies and requirements. But there isn’t a good way to document this or capture this in state after things have been deployed manually.
Now that I have moved to my immutable Linux configurations though, I now have the ignition file that represents how to reproduce the entire server using that one configuration file. After quickly deploying a new Flatcar instance, I would just set the ignition file and deploy the VM allowing it to bootstrap from ignition.
If I do add something after the fact that I forgot to add in the ignition file from the start, I go back and add this in code in that file and commit it. That way, if I deploy it again, I have the changes captured.
This makes the operating system itself largely disposable as that is not the important component. Your configuration is. For me, that has been one of the most valuable long-term benefits of the setup.
Troubleshooting changes
Does running immutable Linux in your home lab totally eliminate all problems? No it doesn’t do that. But, I definitely have seen problems diminish in my environment with the move to immutable configurations from the operating system perspective.
In your traditional distros, problems are way more likely involving package conflicts, dependency mismatches, or accidental host modifications than they are in something like an immutable distro like Flatcar. So again, I spend less time worrying about things like the underlying operating system.
Wrapping up
Looking back at the past few months, would I recommend running an immutable operating system for your home lab workloads like Flatcar Linux? Absolutely. That does not mean immutable Linux is the right choice for every workload or every home lab. I think they do make a lot of sense for container hosts like your Docker hosts. My experience has been very positive with this workload. The biggest surprise is how it will change your management and administration behaviors. It just makes you have a complete mindset shift. Now, I rely more heavily on Git and I think more carefully about infrastructure design when I am spinning up new hosts, which is a good thing.
Traditional Linux distributions still make sense for many use cases. There are environments where package flexibility, direct host customization, or application requirements make immutable approaches less practical. Definitely find some time to experiment with immutable Linux. How about you? Is this on your list of things to try? Let me know in the comments.
Google is updating how articles are shown. Don’t miss our leading home lab and tech content, written by humans, by setting Virtualization Howto as a preferred source.
About The Author
Great one Brandon. I am glad you posted an update as I wanted to see what your thoughts on immutable Linux was over the long term. Thinking I may go this route too.