标签： Ollama

Recently deployed Proxmox server to my N100 machine again. Tried running a small LLM inside the VM but got a warning with running with CPU only, therefore tried to figure out how to enable GPU for Ollama under this condition.

N100 Background

N100 is a low power CPU with 6W TDP from Intel. It’s mainly used for a small energy efficient server running in the garage. It also comes with an Intel UHD Graphics built inside the CPU. This type of GPU is not supported by Ollama by default, as using it might not have more benefits than using CPU directly.

Proxmox Settings

In order to get access to iGPU inside the VM, iGPU needs to be passed into the VM through Proxmox.

To achieve this, login in to the Proxmox web management and go to Datacenter > Node > VM > Hardware > Add PCI Device.

Select Raw Device and add UHD Graphics with All Functions selected.

Reboot the VM after adding the device.

Notice the iGPU can only be passed to a single VM in this case.

Ollama Settings

Log in to the VM. Firstly install Vulkan support.

sudo apt install -y mesa-vulkan-drivers vulkan-tools

Then add Ollama user to the GPU groups.

sudo usermod -aG render,video ollama
sudo systemctl restart ollama

Change Ollama’s settings so it does use iGPU

sudo systemctl edit ollama

Make the following changes:

[Service]
Environment="OLLAMA_VULKAN=1"
Environment="OLLAMA_IGPU_ENABLE=1"
Environment="GGML_VK_VISIBLE_DEVICES=0"

(Given device 0 is your GPU)

Finally apply the changes, all done.

sudo systemctl daemon-reload
sudo systemctl restart ollama

Run any model, and then use ollama ps to observe if it’s using GPU now.