Proxmox

Nested ESXi install in Proxmox: Step-by-Step

Learn how to use Proxmox VE for nested ESXi virtualization in your home lab with this step-by-step guide. Learn ESXi clustering and other configurations with nested virtualization.

If you have a Proxmox VE server in your home lab or production environment and want to play around with VMware ESXi, you can easily do that with Proxmox nested virtualization. Let’s look at the steps required for a nested ESXi server install in Proxmox.

Nested Virtualization in Proxmox

Nested virtualization in Proxmox VE is easy to set up and has real benefits in learning and setting up rather complex architectures without the physical hardware that would otherwise be needed to set them up physically.

Now, you can use something like VMware Workstation to easily nest ESXi. However, if you already have a dedicated Proxmox host, it is a better platform for a dedicated lab experience. There is always running it on VMware ESXi if you have a physical VMware host.

Proxmox nested virtualization allows exposing the CPU’s hardware virtualization characteristics to a nested hypervisor. This process to expose hardware assisted virtualization to the guest ESXi VM is required so the nested hypervisor can run virtual machines.

Preparing your Proxmox VE host to enable nested virtualization for ESXi

If you don’t know how to configure Proxmox Nested Virtualization or enabling hardware assisted virtualization, you can see my recent guide to do that here: How to Enable Proxmox Nested Virtualization.

An overview of the few steps exist to enable nested virtualization for Proxmox and run a nested VM hypervisor are as follows:

  • Make sure your CPU supports hardware-assisted virtualization
  • Enable hardware-assisted virtualization if it isn’t enabled already
  • Enable nested virtualization on the nested ESXI installation VM

Creating the ESXi VM in Proxmox

VMware hypervisors are extremely popular in the enterprise. Let’s look at the process to create the VMware ESXi VM in Proxmox. This is a normal creation process for the most part. I will show you guys one option I chose that didn’t work, surprisingly when creating the VM running ESXi.

Beginning the create virtual machine wizard
Beginning the create virtual machine wizard

Upload your VMware ESXi 8.0 U2 or other ESXi ISO to your Proxmox server and select this in the wizard. On the type, choose Other for the guest operating system.

Select your esxi iso image under the os tab
Select your esxi iso image under the os tab

Here I left VirtIO SCSI single selected for SCSI controller.

Leaving the defaults under system for esxi nested
Leaving the defaults under system for esxi nested

On the Disks screen, configure the disk size you want and also the Storage location for your VM files and hit Next.

Setting up the storage for the esxi vm
Setting up the storage for the esxi vm

Choose your CPU options.

Cpu settings for the esxi nested vm
Cpu settings for the esxi nested vm

Configure your memory.

Memory configuration
Memory configuration

Ok, so this is the step that surprised me a bit. I here selected Intel E1000 which is a standard Intel driver. But I will show you what happens during the install.

Setting the network adapter to e1000
Setting the network adapter to e1000

Confirm your configuration and click Finish.

Confirm the installation options
Confirm the installation options

Step-by-Step Installation of Nested ESXi

Let’s look at how to install ESXi in Proxmox after we have created the Proxmox virtual machine to house the nested virtual machine install.

Below is booting the VMware VM guest OS in Proxmox.

Beginning the esxi 8.0 u2 installation
Beginning the esxi 8.0 u2 installation
The esxi nested vm boots into the installation
The esxi nested vm boots into the installation

OK, so I told you there was something unexpected happen with the Intel E1000 driver. It didn’t detect the network adapter in ESXi.

No network card detected in esxi
No network card detected in esxi

I powered the ESXi VM down and went back and selected VMware vmxnet3 adapter for the model.

Changing the network adapter model to vmware vmxnet3
Changing the network adapter model to vmware vmxnet3

Now, the network adapter was recognized and the installation proceeded.

The installation of nested esxi continues
The installation of nested esxi continues

Now for the standard screens, but we will show them anyway. Accept the EULA.

Accept the eula 1
Accept the eula 1

Select the target storage for the installation.

Select the installation target storage for nested esxi
Select the installation target storage for nested esxi

Select the location for the keyboard layout.

Select the esxi keyboard layout
Select the esxi keyboard layout

Enter and confirm your password.

Configure the root password for esxi
Configure the root password for esxi

I am running on an older Xeon D processor so we see the alert about an outdated processor that may not be supported in future releases. You will see the same error on bare metal.

Warning about older cpu support in esxi 8.0 update 2
Warning about older cpu support in esxi 8.0 update 2
Confirm the installation of esxi and repartitioning
Confirm the installation of esxi and repartitioning

The installation begins.

Esxi installation progress begins
Esxi installation progress begins

Finally, we are prompted to remove the installation media and reboot.

Installation finished remove installation media
Installation finished remove installation media

Hopping back over to Proxmox, I remove the ESXi ISO before rebooting.

Removing the iso from the esxi vm in proxmox
Removing the iso from the esxi vm in proxmox

After initiating a reboot.

Rebooting the esxi installation
Rebooting the esxi installation

After the nested ESXi installation boots, we see it has correctly pulled an IP address from DHCP so the network adapter is working as expected.

Vmware esxi vm in proxmox boots and it correctly pulls a dhcp address
Vmware esxi vm in proxmox boots and it correctly pulls a dhcp address

Below, I logged into the VMware host client to manage the ESXi host running in Proxmox.

Logged into the esxi host client
Logged into the esxi host client

Managing Virtual Machines in a Nested Setup

The cool thing about working with ESXi that is nested in a Proxmox VM is that, for the most part, you won’t notice much difference if you are used to accessing the ESXi host client or adding the ESXi host to the vCenter Server and managing it with vCenter.

Using advanced features in nested VMs

The great thing about running ESXi as a nested hypervisor, is you won’t see any difference in the advanced features for nested VMs. You will still be able to do things like installing VMware Tools in Linux and your Windows Server operating system instances.

If you are configuring a cluster of ESXi hosts with vCenter, you can utilize features like vMotion and DRS within a nested VMware vSphere cluster.

Troubleshooting Common Issues in Nested Environments

Running nested ESXi in Proxmox can be a bit of a mind-bender on the networking side. However, this is not unique to Proxmox, as running nested ESXi on a physical ESXi host can be the same challenge.

First, though, you need to understand Proxmox VLANs. I just covered this recently as well. So, check out my post on Proxmox VLANs to first understand how to configure VLANs in Proxmox.

Just remember, on the nested VMware ESXi side, you can’t tag VLANs on your port groups as this will lead to “double tagging”. They will instead assume the tag from the Proxmox side.

What I like to do is set up the Proxmox Linux Bridge as a trunk bridge, which is the default configuration when you make it VLAN aware. Then, you can change the tag on your network adapter configured for your VMware ESXi VM to tag the traffic from the ESXi VM.

Frequently Asked Questions About Nested ESXi in Proxmox

How Does Nested ESXi Differ from Regular Virtualization in Proxmox?

Nested ESXi in Proxmox takes virtualization a step further by running a virtual machine (VM) within another VM. In nested setups, ESXi acts as a guest hypervisor within the VM to create and manage additional VMs in this second layer of virtualization.

Can I Run VMware Tools in a Nested ESXi VM?

Yes, VMware Tools can be installed and run within a VM running on nested ESXi in a Proxmox environment. This installation enhances the functionality and performance of the nested VMs. It provides better hardware compatibility and improved management capabilities.

What Are the Key Considerations for VM Hardware Settings in Nested Virtualization?

When configuring VM hardware in a nested virtualization setup, it’s important to allocate sufficient resources, such as CPU and memory, to ensure smooth operation. Additionally, you should enable promiscuous mode in the virtual switch settings to allow communication between nested VMs.

Is Nested ESXi Suitable for Production Environments?

Not really in most scenarios. You definitely won’t be supported by VMware in a nested environment and likely not Proxmox either. It is best to keep nested environments in their proper place, for learning and labbing and testing out configurations without the physical hardware to install on bare metal.

How Can I Optimize the Performance of Nested VMs in Proxmox?

Give attention to resource allocation, enabling hardware-assisted virtualization, and configuring network settings properly. Monitor your Proxmox VE host and nested ESXi VMs to make sure there are no performance issues.

Can Windows Server Be Used Effectively in a Nested ESXi Setup?

Windows Server can be run as a guest operating system in a nested ESXi VM. This setup allows for testing and development of Windows-based applications in a controlled, virtualized environment, leveraging the capabilities of both Proxmox and ESXi.

Are There Specific Network Configurations Required for Nested ESXi in Proxmox?

Nested ESXi in Proxmox requires specific network configurations, including setting up virtual switches and enabling promiscuous mode to allow proper network traffic flow between nested VMs. Proper configuration ensures seamless connectivity and communication within the nested environment.

What Are the Benefits of Using Intel VT-x in Nested Virtualization?

Using Intel VT-x in nested virtualization enhances the performance of nested virtualization. This technology enables more efficient emulation of hardware features. Really, you don’t want to use nested virtualization without it.

Wrapping up

Hopefully, this blog post has been a help to any who are running Proxmox as your hypervisor running your home lab environments. It is easy to get a virtual machine running with VMware ESXi in a Proxmox nested environment. Keep in mind the need to use the VMware vmxnet3 adapter and the note on Proxmox VLAN tagging. If you are running guest VMs in your ESXi VM, you will also need to keep in mind the need to enable promiscuous mode for your Proxmox bridge.

Subscribe to VirtualizationHowto via Email 🔔

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Brandon Lee

Brandon Lee is the Senior Writer, Engineer and owner at Virtualizationhowto.com and has over two decades of experience in Information Technology. Having worked for numerous Fortune 500 companies as well as in various industries, Brandon has extensive experience in various IT segments and is a strong advocate for open source technologies. Brandon holds many industry certifications, loves the outdoors and spending time with family.

Related Articles

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.