Notes on setting up a Windows 10 VM with GPU passthrough using QEMU/KVM and libvirt.

Prerequisite Changes

1 Enable IOMMU and AMD SVM

Enable both IOMMU and AMD SVM in your BIOS settings.

2 Collect Device IDs

Get device IDs for the passthrough GPU and anything else in the same IOMMU group. PCI bus devices do not count.

3 /etc/default/grub

Make sure the kernel recognizes IOMMU is enabled. Add device IDs to the vfio-pci module.

Add to GRUB_CMDLINE_LINUX_DEFAULT:


# /etc/default/grub

amd_iommu=on iommu=pt vfio-pci.ids=[[gpu device ids]]
            

4 Include vfio Modules in initramfs

Make sure the vfio, vfio_iommu_type1, and vfio_pci modules are loaded.


# /etc/initramfs-tools/modules

vfio
vfio_iommu_type1
vfio_pci
            

5 Load vfio Modules at Boot

Make sure the vfio, vfio_iommu_type1, and vfio_pci modules are loaded. This may be redundant given the previous section, but it doesn't hurt to include. The important thing is that the vfio-pci driver attaches to the GPU devices before other drivers.


# /etc/modules-load.d/vfio.conf

vfio
vfio_iommu_type1
vfio_pci
            

6 Change QEMU Config

Uncomment the relevant lines and populate with relevant info.


# /etc/libvirt/qemu.conf

user = "jack"
group = "kvm"
cgroup_device_acl = [
    [[evdev input devices]]
    ...
]
            

7 Scream

Download and build scream, then move the binary to /usr/bin/scream.

Create a systemd scream user service:


# ~/.config/systemd/user/scream.service

[Unit]
Description=Scream Audio Receiver for win10 VM
After=pulseaudio.service pipewire.service

[Service]
Type=simple
ExecStart=/usr/bin/scream -i virbr0
Restart=on-failure

[Install]
WantedBy=default.target
            

Enable the service:

systemctl --user enable scream.service

Create a libvirt hook to start and stop the service:


# /etc/libvirt/hooks/qemu

#!/bin/bash

TARGET_VM="win10"
USER_NAME="jack"
USER_UID="1000"

OBJECT="$1"
OPERATION="$2"

if [ "$OBJECT" == "$TARGET_VM" ]; then
    if [ "$OPERATION" == "prepare" ] || [ "$OPERATION" == "start" ]; then
        sudo -u "$USER_NAME" XDG_RUNTIME_DIR=/run/user/"$USER_UID" systemctl --user start scream.service
    elif [ "$OPERATION" == "stopped" ] || [ "$OPERATION" == "release" ]; then
        sudo -u "$USER_NAME" XDG_RUNTIME_DIR=/run/user/"$USER_UID" systemctl --user stop scream.service
    fi
fi
            

VM Setup Steps

1 UEFI OVMF Firmware

Select UEFI OVMF firmware for the VM.

2 Enable SATA CDROM ISO Boot Device

Add a SATA CDROM device with the Windows ISO as a boot device.

3 Change Disk to virtio

Change the disk bus type to virtio for better performance.

4 Disable Network Interface and Change to virtio

Disable the network interface for now and change its type to virtio.

5 Add Hardware: virtio-win SATA CDROM

Add a second SATA CDROM device with the virtio-win drivers ISO.

6 Add Hardware: GPU and Audio PCI Devices

Add the GPU and its associated audio PCI devices as host devices.

7 Hide Virtualization

Prevent the guest from detecting it is running in a VM.

Under hyperv features:


<vendor_id state="on" value="1234567890ab"/>
            

Under features:


<kvm>
    <hidden state="on"/>
</kvm>
            

8 Add vBIOS ROM

This goes under the hostdev tag (not source) where the source refers to the GPU device specifically.


<rom bar="on" file="/usr/share/vgabios/gpu.rom"/>
            

9 Set Up evdev Inputs

Add the domain namespace header:


<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
            

Add QEMU commandline arguments:


<qemu:commandline>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=mouse1,evdev=/dev/input/by-id/usb-Logitech_USB_Receiver-if02-event-mouse"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=kbd1,evdev=/dev/input/by-id/usb-Keychron_Keychron_K3-event-kbd,grab_all=on,repeat=on"/>
</qemu:commandline>
            

Add evdev input devices:


<input type="evdev">
    <source dev="/dev/input/by-id/usb-Logitech_USB_Receiver-if02-event-mouse"/>
</input>
<input type="evdev">
    <source dev="/dev/input/by-id/usb-Keychron_Keychron_K3-event-kbd" grab="all" grabToggle="ctrl-ctrl" repeat="on"/>
</input>
            

10 CPU Performance Improvements

Pin CPU cores and explicitly specify the topology.


<vcpu placement="static">12</vcpu>
<cputune>
    <vcpupin vcpu="0" cpuset="6"/>
    <vcpupin vcpu="1" cpuset="18"/>
    <vcpupin vcpu="2" cpuset="7"/>
    <vcpupin vcpu="3" cpuset="19"/>
    <vcpupin vcpu="4" cpuset="8"/>
    <vcpupin vcpu="5" cpuset="20"/>
    <vcpupin vcpu="6" cpuset="9"/>
    <vcpupin vcpu="7" cpuset="21"/>
    <vcpupin vcpu="8" cpuset="10"/>
    <vcpupin vcpu="9" cpuset="22"/>
    <vcpupin vcpu="10" cpuset="11"/>
    <vcpupin vcpu="11" cpuset="23"/>
</cputune>
            

<cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" clusters="1" cores="6" threads="2"/>
</cpu>
            

11 Remove Tablet Device

Remove the tablet device. It messes with evdev inputs.

12 Boot and Install

Boot the VM and install Windows.

13 Install virtio-win Drivers

Install the virtio-win drivers from the CDROM. Do NOT install guest tools. It breaks the pointer when attempting to install AMD drivers.

14 Update Windows

Make sure to enable the network device to download the update. This can be done on the fly while the VM is running.

15 Install AMD Drivers

Driver only, no CPU chipset.

16 Make Monitor the Main Display

Set the passthrough monitor as the primary display.

17 Remove Display Spice and Video QXL

Removing the Windows and virtio-win CDROM devices seems to break 1440p 144Hz monitors. It can cause an error saying the timing isn't supported by the display. This can be fixed by connecting to a second monitor, changing the display refresh rate to 144Hz, then reconnecting to the main monitor.

18 Scream

Download and install the scream driver in the Windows guest.