Run AI Locally on Fedora with Ollama: Live Ops 005

I wanted to run AI locally on Fedora using an Nvidia GPU I had sitting in my Dell PowerEdge R730 — and an hour and a half later, I actually got it working. Here’s what the road looked like.

The hardware is a Tesla K80, which is basically two 12GB VRAM cards in one chassis and cost me next to nothing off Amazon. The plan was straightforward: get the Nvidia drivers working on Fedora Server 41, pull down Ollama in a Podman container, add Open WebUI on top for a ChatGPT-like interface, and wire it all up behind NGINX Proxy Manager. I had notes. I had a cheat sheet. I was ready.

What I wasn’t ready for was Fedora’s relationship with Nvidia drivers. The Tesla K80 needs the 470.xx legacy driver, not the current one, and figuring that out took most of the first hour. Between installing the wrong driver version, conflicts with packages left over from previous attempts, and the driver service not loading cleanly, it took a force rebuild of akmods and dracut before nvidia-smi finally came back with output. AMD cards just work. Nvidia is a project.

Once the GPU was cooperating, getting Ollama running locally turned out to be easier than the container approach I’d planned, since the GCR image path for the Podman version kept returning errors. Ollama installed directly on the host, Llama 3.2 pulled down without complaint, and I had a working local model to talk to inside of a few minutes. DeepSeek R1 8B came along for the ride as a second model to compare. Open WebUI came up in a Podman container, and after a solid round of “why won’t this connect,” the fix turned out to be a single :Z flag on the volume mount — SELinux doing what SELinux does.

By the end of the stream, Open WebUI was running, both models were loaded, and I had a local AI setup I could actually use. The NGINX reverse proxy was still showing a bad gateway when I had to call it for the day, but that’s a problem for next session. On the to-do list: get RAG working so I can feed it my own blog posts as a local knowledge base, and eventually stop paying OpenAI for what I can run at home.

Watch the full stream above, and if you want to follow along as the homelab build continues, subscribe to the channel or grab the audio version of the main show at podcast.itguyeric.com.

Building a Golden Image | Live Ops 003

The goal was simple: build a reusable Fedora golden image homelab template I can clone whenever I need a new server. Nothing about it was simple.

The plan was to use Image Builder inside Cockpit to create a base Fedora server image, then store the resulting QCOW2 file on my unRAID box so I can spin up new VMs on demand. I had packages already lined up in the blueprint: Crony, Cockpit, NetworkManager, bash completion, VIM enhanced, TuneD, and a few others. The Ansible service account was configured, IPv6 disabled at the kernel level, and automatic partitioning set so I’m not hand-holding every deployment. It was supposed to be a quick stream before the Fedora podcast.

What actually happened was a solid hour of fighting permissions issues with the welder group, a blueprint package that didn’t exist in Fedora the way it does in RHEL, and Image Builder refusing to build ISO images while happily producing QCOW2 files without complaint. Once I got the QCOW2 built and pulled down, I had to convert it to an IMG file using qemu-img convert just to get unRAID to accept it, then resize the image to 32 gigs because the VM came up with a 4.2 gig root partition and nowhere to grow. ext4 on an unRAID VM turned out to be less cooperative than I’d hoped.

The good news: by the end of the stream I had a booting Fedora VM with bash completion, VIM enhanced, and Cockpit already running, SSH keys in place, and the performance profile defaulted to virtual-guest automatically. The less good news: I’m probably moving VM deployment over to my Fedora-based ITG02 hypervisor from Live Ops 003, where QCOW2 management is a lot less painful. The Ansible playbook to automate the whole build-convert-deploy pipeline is the next piece, and that’s where things should start getting cleaner.

Watch the full stream above, and if you want to follow along as the homelab build continues, subscribe to the channel or grab the audio version of the main show at podcast.itguyeric.com.