Self-hosting Local LLMs

Self-hosting a GPT-style AI model in your home lab gives you much more control over your data. It also gives you control over AI with your infrastructure, and how your AI workloads actually run. In this video, I walk through how I self-host local LLMs using Ollama and Open WebUI. In this walkthrough, we look at the hardware requirements, Docker deployment, GPU acceleration, and Proxmox GPU passthrough.

We start by looking at why we would want to run AI locally instead of relying entirely on cloud services. Keeping prompts and interactions inside my own environment is one of the biggest advantages. This is true especially for experimenting with private data, home lab automation, coding workflows, and Kubernetes. I also show that you do not necessarily need the newest GPU to get started. Even older NVIDIA hardware can be useful depending on the model you choose and the amount of VRAM available.

From there, we look at deploying Ollama and Open WebUI with Docker. We look at how the two containers work together, how Open WebUI connects to the Ollama API, how to expose the web interface, and how to download and manage local models. We will also cover GPU acceleration in Docker Desktop, Linux, virtual machines, and Proxmox environments.

For my Proxmox setup, I show the major steps involved in passing an NVIDIA GPU through to a virtual machine. These steps include configuring IOMMU, identifying the PCI device, VFIO configuration, and attaching the GPU to the VM. We then see how to look at the performance difference once GPU acceleration is working. We look at several practical lessons I have learned from running local AI in my own home lab, including matching models to available VRAM, using NVMe storage, taking Proxmox snapshots during configuration, and using a reverse proxy for SSL.

Finally, we will see the real-world uses for a self-hosted LLM. These include private coding assistants, offline chatbots, Home Assistant integrations, Obsidian, and connecting tools such as kubectl-ai directly to an Ollama instance. If you want to build a private AI server with Proxmox, Docker, Ollama, Open WebUI, and GPU acceleration, we take a complete walkthrough in the video.