Proxmox Docker Containers Monster – 13000 containers on a single host
One of the cool little experiments in the home lab recently was seeing just how many Docker containers could run on a single physical Proxmox host, hosting a number of Docker container hosts. I could easily scale containers up to 1000 in each virtual machine using virtual machines and Docker Swarm. Let’s look at Proxmox Docker container host monster :). First though, let’s go through some basics of running Docker containers on Proxmox. You can skip directly to the section where I run 13000 containers in the table of contents.
Table of contents
Running Docker in VMs vs LXC Containers: A Quick Comparison
Let’s compare running Docker in a virtual machine vs. LXC containers. It’s worth comparing the two approaches.
Resource Efficiency: LXC containers are generally more resource-efficient than VMs, as they share the host system’s kernel and use fewer resources. This makes LXC containers a more lightweight option for running Docker containers, especially in environments where resources are limited.
Isolation: VMs provide a higher isolation level between the host and guest operating systems, as each VM runs its own kernel. This can be beneficial from a security perspective. However, LXC containers also provide a decent level of isolation, making them suitable for most use cases.
Compatibility: Docker containers should run consistently across different environments, whether in VMs or LXC containers. However, certain advanced features may be more challenging to implement in LXC containers compared to VMs. It is essential to test your specific use case to determine the best approach for your environment.
Ultimately, the choice between VMs and LXC containers for running your Proxmox Docker Container Monster setup depends on your needs and resource availability.
How to Set Up Docker Containers in Proxmox Using an LXC Container
Setting up Docker containers in Proxmox using an LXC container involves a few key steps. In this tutorial, we will walk you through creating an LXC container, installing Docker Engine, and setting up Docker Compose.
Use the Turnkey Core template
Another option for a base operating system is a Docker container host running on Turnkey Core Linux as a template you can download from your Proxmox host. Download the Turnkey Core container template from the Proxmox template repository. This will be used to create an LXC container for running Docker containers.
Downloading templates
Navigate to the storage location where you want to store the template, then click “Templates” and search for “Turnkey Core.” Download the template and wait for the process to complete. You can create a new LXC unprivileged container to run Docker as a Docker host on Proxmox VE.
Create a Ubuntu Server 22.04 VM
In preparing to run 13,000 containers. I found the LXC approach did not scale as well as virtual machines. I For me, going the route of virtual machine hosts for Docker provided the results I was after. How do you create a Docker container host using a virtual machine.
First, you must create a new VM running Ubuntu Server 22.04 as the base operating system. In the Proxmox web interface, click on “Create VM” and configure the necessary settings such as hostname, disk size, and storage location. Once the VM is created, proceed with installing Ubuntu Server 22.04.
Below, I have created an Ubuntu Server 22.04 virtual machine template for cloning additional VMs for container hosts.
Clone from your template virtual machine
Now that you have created the Ubuntu virtual machine template, we want to install Docker so we can use this installation to create a virtual machine template, which will save time. Once you have the VM template, you can clone from this virtual machine for your Docker container hosts. We can configure the container with the necessary resources, such as CPU, memory, and storage.
Installing Docker
Once the VM is created and running, access it via the Proxmox host console and run the following commands. Since I am using the Ubuntu Server platform, I am following the official guidance from Docker here on installing Docker in Ubuntu: Install Docker Engine on Ubuntu | Docker Documentation.
Proceed with the installation of a few prerequisites needed.
sudo apt-get update
sudo apt-get install
ca-certificates
curl
gnupgInstall the official GPG key:
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpgUse the following commands to setup the repository:
echo
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |
sudo tee /etc/apt/sources.list.d/docker.list > /dev/nullRun an apt update to update the package index:
sudo apt-get updateFinally, install Docker Engine on the Linux container.
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-pluginUse Portainer to manage your containers
To manage your Docker containers more easily, it’s recommended to install Portainer, a web-based interface for managing Docker environments. To install Portainer, run the following command:
docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ceOnce Portainer is installed, access its web interface by navigating to http://<your_proxmox_host_ip>:9000. Log in with your chosen username and password be able to manage your Docker containers.
Create Docker Containers in Portainer
Now that you have Portainer installed and configured, you can create Docker containers directly from the web interface. Click “Containers” in the left-hand menu, then click “Add container.” Provide the necessary details, such as the container name, image, and any required environment variables.
Docker compose and application stacks
To create containers using Docker Compose, you can either upload a Docker Compose YAML file or create one directly within the Portainer interface. Click “Stacks” in the left-hand menu, then click “Add stack.” Provide the necessary details, and Portainer will create the Docker containers based on the Docker Compose configuration.
You’ve successfully created a Proxmox Docker Container Monster environment using an LXC container with this setup. This allows you to run multiple Docker containers on your Proxmox host with minimal overhead and manage them easily using Portainer.
Install Docker Engine on the Linux Container host system
Before diving into the Docker installation process, ensure your LXC container is up to date by running the following commands:
sudo apt update sudo apt upgradeAfter updating the container, proceed with the installation of Docker by executing these commands:
curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo usermod -aG docker $USERThese commands download and install Docker Engine on your Linux container, allowing you to run Docker containers seamlessly.
Scaling up to 13000 Containers!
I wanted to see what my Supermicro E300-9D would do from a container perspective. It has 16 cores and 128 GB of memory. So I cloned the virtual machine template for VM Docker container hosts and was able to scale the containers in each one to 1000 containers each using 12 GB of memory configured.
I used Docker Swarm as an easy way to spin up (scaling) the containers to the desired number on each host.
docker swarm init
Below is an example of me testing a couple of containers on a single VM. As you can see below, I have spun up 800 nginx containers and 200 redis containers. However, I settled on just a single application container in the final test. So, I spun up 1000 containers on one host (nginx) and 1000 on another (redis), until I reached 13 VMs running 13000 containers.
To create a service, use this command:
docker service create --name nginx nginx:latestThen, to scale the service, you can use the command:
docker service scale nginx=1000
At 13,000 containers, the host’s resources were basically exhausted. I was running right at 100% CPU and memory was sitting at 95% memory. But it was up and running!
To recap:
- I cloned 13 virtual machines in Proxmox (Ubuntu 22.04)
- I initialized the virtual machine as a single Docker Swarm host
- I then create a service (alternated between nginx and redis with each VM)
- Then, I scaled the service in each virtual machine to 1000 containers
It was great to see that running so many containers was possible on a single server. It is a testament to just how efficient containers are, and when it comes to running applications at scale, they are superior to VMs. It is just a matter of pointing your DNS server to the applications configured in containers for end users to access.
Take a look at a video of the process here:
Related posts
Wrapping up
Proxmox offers an excellent platform for running Docker containers, whether you use LXC containers or VMs as container hosts. Following the steps outlined in this tutorial, you can set up a Proxmox Docker Container Monster environment and easily manage your Docker containers using the Portainer web interface.
With the versatility of Proxmox and the power of Docker at your fingertips, you can easily create, deploy, and manage containerized applications. This powerful combination enables you to harness the full potential of containerization in your environment, whether it’s a home lab or a production environment.
Remember to keep your system updated and monitor resource usage, ensuring your Proxmox Docker Container Monster setup runs smoothly and securely. Embrace the world of containerization and enjoy its benefits to your infrastructure management experience.
this is so insane, people don’t realize how much of a paradigm shift orchestration is
you running it on ZFS?
Interesting.
I’d like to know why you used 1000 containers and each using 12 GB of memory. Would 2000 containers and 24 GB of memory have worked also?
The screen shots indicate you have exceeded your ram with swap by about 800MB and KSM saved about 35 GB also. (about 3 VMs). Are each VM using swap also?
Also, the containers in the example have zero work load and hence would exceed the system resources even more once they are put to work.
You wouldn’t have needed ProxMox for that – the article basically just shows the power of Docker 😉
The article actually shown power of lxc
I’m not entirely convinced this article is not a joke. Right below the table of contents it starts off with “Now that you have learned how to…” Learned where? When?
You then proceed to say you will run Docker in LXC containers, download a Turnkey Code container template, apparently create one container that is apparently never booted (based on screenshots) and then proceed with creating a bunch of VMs instead of containers as if containers were never mentioned in the article at all.
Andy, my bad. I updated the article so it was less confusing. I put everything together quickly for this idea so thoughts got a bit jumbled in the article on the first draft. I am presenting the idea of running a Docker container host in either an LXC container or a full virtual machine. I added the note that the LXC container route did not scale as well for me as the virtual machine approach. With that, I pivoted from running a Docker host in LXC over to a full Ubuntu 22.04 VM. Hopefully this helps! Article updated and notes added.
In my home lab setup, I am running 3 Proxmox servers and working on the fourth. I have started doing a lot of container development for work and thought it would be great to run the workload on Proxmox. Unfortunately, Docker containers are required, not LXC. Since Proxmox doesn’t support Docker containers, I thought this article was going to solve my problems.
No so, instead, it is an iteration of what I’m already doing with a worse OS for the VM. Why would you use a desktop OS with a heavy dose of built-in adware for the base OS? Wouldn’t a server OS be better such as Debian, FreeBSD or RHEL?
Your original premise of the article was very interesting. However; now you need to change the article title to reflect the pivot to Docker on LXC isn’t great. Was a very disappointing article.
I don’t get what desktop OS are you talking about as neither Proxmox nor Ubuntu Server are one.
Well I for one fail to see the use beyond theoretical exercises. At least the way the exercise is shown.
If I have a physical server where I can create a virtualised one and running nginx on it, why would I spin multiple instances of nginx i.e. ” docker service scale nginx=1000″ if not to serve more requests. Those requests would have been served by the nginx process running on the virtual server with the same compute resources anyhow. So OK, we want two instances for High Availability, sure, but then you spin two VMs instead of one. What is the problem that spinning 1000 instances of nginx is solving? Then multipled by 13.
CM thank you for your comment. However, as stated in the opening, “One of the cool little experiments in the home lab recently”….this was an experiment, nothing more, nothing less.
Ok, fair enough.
Seems you wanted to create a clickbait title with some kind of Guinness Word Record and there you have it. Literally no conclusion can be made from this content where pointless containers are spun up in droves in a bunch of VMs until CPU load gets 100%.
If one genuine conclusion came out of this, it probably would have been why did you find that “In preparing to run 13,000 containers. I found the LXC approach did not scale as well as virtual machines” while on the other hand “LXC containers are generally more resource-efficient than VMs”.
Diakritikus, thank you for your critical comment 🙂 This was an experiment as expressly stated in the blog…”One of the cool little experiments in the home lab recently…” If you don’t want to read it, don’t. I really appreciate those who are positive and embrace those of us who are trying cool things in our home labs and sharing with the community. I encourage you to create a blog and keep fresh content and push the envelope. It is humbling and I think it will change your perspective when others work hard to produce content that others appreciate.
Or you could have found constructive feedback in there telling what was missing for me. But if you don’t want to read it like that, please don’t.
I’m happy I learned KVMs scale better than LXCs when it comes to maxing out the number of idle docker containers.
Diakritikus, I think we are on the same page here. Being constructive. I’m glad you found value in the article. Thank you for sharing your thoughts.
You talk us through setting up a VM host for Docker:
But then you conclude this setup with:
Confusing. Thanks for the article though.