Building a game-streaming VM with Debian and LibVirt

I've gone desktop-less for a couple of years now, my laptop is good enough for my needs. I can dock it for productivity and it can even play some modern games at high settings, but it is not very convenient when I want to play something on the living room TV, for example. For that reason, I built myself a gaming VM on my server, like all the cool kids do.

I'm writing this to partially document what I put together, and how/why I did it. Also because most similar articles focus on Unraid, and I went with KVM on Debian instead.

Why a VM?

I've been thinking for a while about building a small mini-ITX PC to play games in the living room, but the lack of money and space kept it from happening. Meanwhile, my server is plenty powerful and always under-utilised, so why not slap a GPU in there and see what happens?

Basic requirements

• I want it to "just work", and somewhat elegantly too. No connecting to terminals or similar "computer touching" to fix stuff when I just want to play.
• Minimal latency. This is the point that put me off this idea for the longest, at least until I did some testing with Sunshine (more on it below), and was very impressed by its responsiveness.

Host specs :

• 8c/16t AMD Ryzen 7 4750G Pro
• 64GB RAM
• Nvidia RTX 3070 FE
• Debian 11 (bullseye)
• KVM Hypervisor + Libvirt for management

Guest specs:

• 8 CPUs
• 16GB of RAM
• The full GPU passed through via PCI-e VFIO
• Windows 11

Picking a game streaming solution that doesn't suck

Three main contenders exist in this arena: Steam Link, Parsec, and Nvidia GameStream.

Starting with Steam Link, it is "alright". Neither the latency or image quality felt great, and not everything I plan to run is on Steam, so it gets a "no".

Next is Parsec, which from reviews seemed promising, but is paid and proprietary, so "thanks but no, thanks".

That leaves only GameStream, which Nvidia has announced will be discontinued sometime around February next year. What to use then?

Thankfully, through the magic of open source, a 4th alternative exists: Sunshine is an open implemetation of the GameStream protocol, compatible with Moonlight, with lots of customisation options, and capable of outperforming the official GameStream server from Nvidia in some cases.

Sunshine runs as a service inside the VM and uses the GPU's video encoder to present the screen and audio to the client with minimal latency. It also simulates a virtual gamepad inside the VM, so your inputs on the client can be passed through. With the combo Sunshine/Moonlight and a few tweaks, I managed to bring the latency down to nearly imperceptible when playing over Ethernet!

Preparations

In order to enable PCI-e passthrough (connecting the GPU "directly" to the VM), you will need compatible hardware, and to pass a few arguments to the Linux kernel depending on your platform. I will not cover how to do that for every single Linux distro and bootloader. Google is your friend.

The parameters I used are:‌                  

modprobe.blacklist=nouveau modprobe.blacklist=nvidiafb video=vesafb:off video=efifb:off video=simplefb:off nofb vga=off

The first two prevent the host OS from loading drivers that may bind to the GPU and interfere with passthrough. The others might not all be necessary, but were added while trying to make Resizable BAR work (see Optimisations further down).

I also added this under /etc/modprobe.d/iommu_unsafe_interrupts.conf:

options vfio_iommu_type1 allow_unsafe_interrupts=1

The VM

I will also not cover the step by step here, and will assume you already know how to create a simple VM using Virt Manager. In my case, I created a Windows 11 VM with the specs listed previously, bridged it's network to my LAN, and removed all Spice-related components, including audio, which requires deleting the following line manually from the machine's XML (otherwise it throws an error when launching the VM):

<audio id="1" type="spice"/>

I also changed the VMs disks from SATA to VirtIO SCSI for better performance, the network controller to VirtIO, and set video type to "none".

Then I attached the PCI devices for the GPU and its "sound card" to the VM, as well as a mouse and keyboard passed through via USB, plugged the GPU into a monitor and installed the Nvidia drivers. At this point you have a fully functional VM with a physical screen, mouse and keyboard (which feels really weird).

Sunshine ☀️

Sunshine's setup is dead simple. Execute the installer, next, next, next, done.

Once installed, open the web UI, set a password, and go the PIN tab and register your Moonlight client.

In my install, the options I changed were:

• Name of the server
• Emulate a DS4 controller on the host (default is Xbox 360)
• Encoder: Nvidia NVENC
• FEC Percentage: 5

Wake-on-LAN

All versions of the Moonlight client have the very convenient option of waking up the source PC when it's offline, via Wake-on-LAN. In our case however, since we're dealing with a machine that does not exist, enabling WoL requires one extra piece to the puzzle.

This piece is virtwold (Virtual Wake-on-LAN daemon), a process that listens for WoL packets, matches the MAC address against your LibVirt VMs, and powers them on.

Installation is fairly simple: install Go on the host, clone the repo, run go build, then put the virtwold executable somewhere in your $PATH and copy and enable the example Systemd service unit in init-scripts/systemd/[email protected].

Performance optimisations

Resizable BAR

All my hardware supports a feature called "Resizable BAR", which in very simple terms allows the CPU and GPU to communicate more efficiently.

Getting this to work inside the VM took a lot of trial and error, but I eventually succeeded. Apart from the kernel boot options I mentioned in the "Preparations" section, I also had to go into the BIOS of the host and set the primary GPU to the integrated one, since leaving it on "auto" or "PCI-e" consistently prevented the BAR size from being changed. I also enabled "ROM BAR" for the PCI-e devices passed through in Virt Manager.

Thankfully after all the time I spent on this, it was all worth it. With Resizable BAR enabled, I get an extra ~20 FPS in Furmark, and even the remote sessions via Sunshine feel more snappy (although I don't have an empirical way of validating that).

CPU pinning

Mapping specific physical cores to each of the VM's cores can improve performance. There are many good guides online on how to do this depending on your host's CPU model.

Huge pages

Not worth the trouble. Stick to transparent huge pages (a.k.a. do nothing).

The end result

With all this in place, I can turn on my living room TV, pick up a controller, launch Moonlight, select "Wake device" to start the VM if not already running, and launch Steam in Big Picture mode. Pretty fucking great if you ask me!