Proxmox GPU PassThrough: Step-by-step Guide

One of the great things about running Proxmox in your home lab is the ability to use GPU passthrough. Home lab enthusiasts often use a repurposed gaming computer or workstation with a fairly powerful GPU installed. GPU passthrough provides many advantages when running a virtual machine with a GPU exposed directly to the operating system from the Proxmox host. Let’s look at GPU passthrough and how this is configured with your internal PCI GPU.

What is Proxmox GPU Passthrough?

Passing through a GPU allows you to create a Proxmox virtual machine that can directly use the graphics card installed in your Proxmox VE host. If you need a virtual machine that has a lot of graphics processing power for gaming, 3D rendering, or running GPU-accelerated applications like AI, GPU passthrough allows you to have this in a VM.

AMD GPU or an NVIDIA GPU with NVIDIA drivers can be used for GPU passthrough, and it also doesn’t matter if it is an Intel processor or AMD processor as long as you have the options configured correctly in the BIOS and the required Proxmox configuration.

However, remember that some have better success passing through their primary GPU with NVIDIA corporation cards or AMD cards, depending on the motherboard and the specific graphics adapter.

GPU passthrough is an example of the general capability of Proxmox and other hypervisors to pass through sys BUS PCI devices to virtual environments. Since the VGA controller is likely a PCI device, you can pass it through to a virtual machine.

Why GPU Passthrough?

Instead of the hypervisor abstracting the physical graphics processing unit, with GPU passthrough VMs can the host’s full GPU potential with almost native performance. It also allows you to use your Proxmox host as a Type 1 hypervisor, have a full-blown graphics workstation running as a VM, and still host several other virtual machines and containers.

In this way, you don’t have to have a separate workstation for your other day-to-day tasks or running graphics-intensive software. So, you really get more bang for your buck and more out of your investment into more powerful Proxmox hardware.

***Disclaimer***

GPU passthrough with Proxmox is not for the faint of heart, and it often leads to frustration when it doesn’t work. There are many variables to this process, and some steps work for some people and the same steps don’t work for others.

It may require some experimentation on your side to find the right combination of settings and configuration that works for your hardware and individual setup.

There are also noted issues with certain video cards, such as the AMD GPU reset bug, that we will discuss below in the troubleshooting section.

Also, let me and others know in the comments below on what you may have had to do to get GPU passthrough working correctly, if different than the steps listed below, to add to the knowledge base for the community.

Setting Up Proxmox for GPU Passthrough

Several steps need to be accomplished before you can configure GPU passthrough in Proxmox. At a high level, let’s look at the list of tasks we need to accomplish:

1. Enable IOMMU settings in the BIOS

2. Install Proxmox

3. Edit modprobe configuration

4. Update the GRUB bootloader

5. Edit kernel modules

6. Identify and isolate the GPU

1. Enable IOMMU settings in the BIOS

Before initiating the Proxmox GPU passthrough, ensure your host system is adequately prepared. This preparation involves configuring the BIOS settings to support IOMMU interrupt remapping and selecting the appropriate CPU and GPU hardware.

Intel CPUs and IOMMU

For Intel CPUs, enable VT-d in the BIOS to facilitate IOMMU support. IOMMU (Input-Output Memory Management Unit) is a technology that allows the hardware to access and map the memory, essential for passthrough functionalities.

AMD CPUs and IOMMU

If you are using AMD CPUs, enable AMD-Vi in the BIOS, which is AMD’s IOMMU technology. Similar to Intel’s VT-d, AMD-Vi provides the necessary support for directing and isolating I/O accesses.

Below is a look at enabling IOMMU in the BIOS:

2. Install Proxmox

If you haven’t installed Proxmox already, you can do that easily using a Rufus or my personal favorite now, Ventoy, to install Proxmox on your host hardware. The latest Proxmox release is Proxmox 8.x.

Check out my Top 5 tips to get started with Proxmox here: Proxmox Homelab: First 5 Basic Configuration Steps.

Configuring the Proxmox Host

Once the host system is ready, proceed with configuring the Proxmox host to support GPU passthrough. This configuration involves editing specific files and adjusting settings to ensure compatibility and performance.

3. Edit modprobe configuration

In the /etc/modprobe.d/blacklist.conf file, add the following blacklist the nouveau, nvidia, and radeon drivers. Blacklisting these drivers prevents the host from loading them, allowing the passthrough to function correctly.

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf

4. Update the GRUB bootloader

After configuring the necessary files, update the GRUB bootloader to apply the changes. Edit the /etc/default/grub file to add the intel_iommu=on or amd_iommu=on (depending on your CPU) and vfio_pci.ids parameters.

#Intel
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
#AMD
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

Then, update the GRUB configuration and reboot the system. ***Note*** some have had to add many more parameters to the GRUB_CMDLINE_LINUX line to get passthrough to work for their particular card. See the Troubleshooting section below.

Make sure to reboot after you update grub.

update-grub reboot

5. Edit the kernel modules

Add the vfio, vfio_iommu_type1, vfio_pci, and vfio_virqfd modules to the /etc/modules file to load them at boot.

echo "vfio" >> /etc/modules  
echo "vfio_iommu_type1" >> /etc/modules 
echo "vfio_pci" >> /etc/modules 
echo "vfio_virqfd" >> /etc/modules

6. Identify and isolate the GPU

Locating the PCI Device

Identify the PCI device IDs associated with the GPU you intend to pass through. You can locate these IDs using the lspci command, which lists all PCI devices connected to the host system. This command will return devices like audio devices, USB controller, etc. However, look for your GPU.

#To see all devices use:
lspci -v
#To search for a specific device name:
lspci -nn | grep -i nvidia

Viewing IOMMU Groups

There is also a command to view IOMMU groups to see if the GPU is in its own group or with devices you intend to pass through together, which is helpful. The forums and other posts mention that generally speaking, users are more successful with GPU passthrough when the GPU is isolated to its own group. Use the following command to list the IOMMU groups and their associated devices:

find /sys/kernel/iommu_groups/ -type l

Editing /etc/modprobe.d/vfio.conf

Once you have identified the GPU’s PCI device IDs, isolate the device to prevent the host system from using it. This isolation ensures the GPU is available exclusively for passthrough to the VM. You can do that with the following commands:

#First
echo "options vfio-pci ids=1002:1638,1002:1637 disable vga=1" /etc/modprobe.d/vfio.conf

#Second
update-initramfs -u

#Third
reboot or reset

Next, identify and list the PCI device IDs of the GPU and its audio function in the /etc/modprobe.d/vfio.conf file. This step ensures that the VFIO driver binds to these devices during boot.

7. Create and configure your virtual machine

The next step is installing your guest operating system in a virtual machine. There isn’t anything really special about this step as this is installing the OS in the virtual machine that you are passing through your GPU to.

This configuration involves adding the PCI devices (GPU and audio function) to the VM and ensuring that the VM uses the OVMF (UEFI) firmware instead of the traditional BIOS.

Set the boot order for your virtual machine to boot from an uploaded ISO image file.

Below, we are installing Windows 11. After the installation, you should see your GPU passed through to your virtual machine, which you can verify in Device Manager (in Windows).

Testing and Troubleshooting

AMD reset bug

Many newer AMD GPUs have a problem known as the AMD Reset Bug. This bug means the card can’t reset correctly, so it can only be used once after the host computer is turned on. If you try to use the card a second time, Linux will try and fail to reset it. This can cause the VM to not start or make the guest, host, or both systems freeze.

Verifying GPU Passthrough Functionality

Once the VM is configured, start it and install the necessary GPU drivers. After installation, verify that the GPU passthrough functions correctly by checking the device manager (Windows) or using the lspci command (Linux).

Troubleshooting Common Issues

If you encounter issues during the setup or operation of GPU passthrough, consult the Proxmox documentation and community forums for guidance. Common problems may include IOMMU group conflicts, driver installation failures, or performance inconsistencies.

Extended GRUB configuration

Some have seen they need to enter the configuration parameters all in their GRUB bootloader config like so, with extra parameters for stability and to keep the host from crashing.

An example of an extended configuration is below:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction video=efifb:off video=vesa:off vfio-pci.ids=1002:1638,1002:1637 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 modprobe.blacklist=radeon,nouveau,nvidia,nvidiafb,nvidia-gpu"

Frequently Asked Questions

What happens if I skip blacklisting GPU drivers in /etc/modprobe.d/blacklist.conf?

If you overlook the step of adding GPU drivers to /etc/modprobe.d/blacklist.conf, the host system might load these drivers, preventing the GPU passthrough from functioning correctly. This misstep could lead to the GPU not being available for passthrough to the virtual machine.

Can I use Proxmox GPU passthrough with both Nvidia and AMD cards?

Yes, Proxmox GPU passthrough supports both Nvidia and AMD cards. However, the setup process might slightly differ due to the unique characteristics and drivers of each card type. Ensure to follow the specific instructions applicable to your GPU model.

Why is my virtual machine not recognizing the passed-through GPU?

Several reasons could lead to this issue. Firstly, ensure that you’ve correctly isolated the GPU and that the host isn’t using it. Secondly, check if the VM has the necessary drivers installed for the GPU. Lastly, verify that the IOMMU groups are appropriately configured, and the GPU is correctly assigned to the VM.

Is it necessary to update GRUB after modifying /etc/default/grub?

Yes, updating GRUB is crucial after making changes to /etc/default/grub. This step ensures that the modifications are applied during the boot process, facilitating the proper functioning of GPU passthrough.

How do I deal with the AMD Reset Bug?

The AMD Reset Bug is a known issue where the AMD GPU can’t reset properly, limiting its use to once per host power-on. To mitigate this, you might need to power cycle the host system after using the GPU in a VM. Be aware of this limitation when planning your GPU passthrough setup with AMD cards.

Can I pass through multiple GPUs to a single VM?

Passing through multiple GPUs to a single VM is possible, but it requires careful configuration. You need to ensure that each GPU is isolated and assigned to the VM correctly. Additionally, the VM’s operating system must support and correctly manage multiple GPUs.

What precautions should I take when editing /etc/modprobe.d/vfio.conf?

When editing /etc/modprobe.d/vfio.conf, you need to identify and list the PCI device IDs of the GPU and the audio device associated with the GPU. If you enter these values incorrect, it can lead to GPU passthrough just not working, host instability, and other issues.

Why is interrupt remapping important for Proxmox GPU passthrough?

Interrupt remapping is vital as it provides enhanced security and performance during the GPU passthrough process. It allows for the efficient handling and isolation of hardware interrupts, ensuring that the passed-through GPU can communicate effectively with the VM without interference.

Can I use GPU passthrough with integrated graphics?

Using GPU passthrough with integrated graphics can be challenging. Integrated GPUs often share resources with the system, making isolation and passthrough more complex. For a smoother experience, using dedicated GPUs is recommended.

How do I verify if my Proxmox host supports IOMMU?

To check if your Proxmox host supports IOMMU, you can run the dmesg | grep -E “DMAR|IOMMU” command in the terminal. If the host supports IOMMU, the command will return relevant entries, indicating that the technology is enabled and functional on your system.

Wrapping up

The Proxmox GPU passthrough process is not for the faint of heart, with many different steps to accomplish. The steps provided in this post provide the basics of GPU passthrough in Proxmox. However, results may vary from user to user and hardware to hardware.

It would be great to see a much more seamless approach to setting up GPU passthrough in Proxmox than the process involved currently. It would be great if Proxmox would work on making this process easier and more seamless in future releases. Again, please do share in the comments any additional steps you have found required as well as the detailed description of your hardware configuration that may help others.