This overlaps with my work on sbuild-qemu, which has its own way of provisionning virtual machines.]]

KVM bootstrap with libvirt

I got tired of dealing with VirtualBox and Vagrant: those tools work well, but they are too far from datacenter-level hosting primitives, which right now converge towards KVM (or maybe Xen, but that didn't seem to recover from the Meltdown attacks). VirtualBox was also not shipped in stretch because "upstream doesn't play in a really fair mode wrt CVEs" and simply ship updates in bulk.

So I started looking into KVM. It seems a common way to get started with this without setting up a whole cluster management system (e.g. Ganeti) is to use libvirt. The instructions here also include bridge setup information for Debian stretch since that makes it easier to host services inside the virtual machines than a clunky NAT setup.

Bridge configuration

Assuming the local Ethernet interface is called eno1, the following configuration, in /etc/network/interfaces.d/br0, enables a bridge on the host:

iface eno1 inet manual

auto br0
iface br0 inet static
    # really necessary?
    #hwaddress ether f4:4d:30:66:14:9a
    address 192.168.0.7
    netmask 255.255.255.0
    gateway 192.168.0.1
    dns-nameservers 8.8.8.8

    bridge_ports eno1

iface br0 inet6 auto

Then disable other networking interfaces and enable the bridge:

ifdown eno1
service NetworkManager restart
ifup br0

Finally, by default Linux bridges disable forwarding through the firewall. This works independently of the net.ipv[46].conf.all.forwarding setting, which should stay turned off unless we actually want to route packets for the network (as opposed to the guests). This can be tweaked by talking with iptables directly:

iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT

Or, preferably, by disabling the firewall on the bridge completely. This can be done by adding this to /etc/sysctl.d/br0-nf-disable.conf:

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

This was discovered in the libvirt wiki.

NAT configuration

The default configuration in libvirtd is a "NAT" configuration. That, in turn, injects firewall rules in the kernel when the "network" is started, to rewrite packets going in and out of the VM. dnsmasq is used for DNS and DHCP as well.

I had quite a battle with this network on my laptop, angela. At first nothing was getting through: IPv6 SLAAC configuration was working, but not DHCP. This was hanging the VM which led me to switch to systemd-networkd (see boot time optimizations). That didn't fix networking but boot would at least not hang for a full minute while DHCP failed.

Then the fix was to add a subset of the Puppet module's NFT ruleset, through this commit:

class { 'nftables::rules::qemu':
  masquerade => false,
}

That created the following patch on the ruleset:

--- /etc/nftables/puppet/inet-filter-chain-default_in.nft   2023-11-28 15:47:58.143874297 -0500
+++ /tmp/puppet-file20231128-15717-utsqvt   2023-11-28 15:59:57.891321815 -0500
@@ -6,6 +6,12 @@
   ip6 nexthdr ipv6-icmp accept
 #   Start of fragment order:50 rulename:default_in-avahi_udp
   ip saddr { 0.0.0.0/0 } udp dport 5353 accept
+#   Start of fragment order:50 rulename:default_in-qemu_dhcpv4
+  iifname "virbr0" meta l4proto udp udp dport 67 accept
+#   Start of fragment order:50 rulename:default_in-qemu_tcp_dns
+  iifname "virbr0" tcp dport 53 accept
+#   Start of fragment order:50 rulename:default_in-qemu_udp_dns
+  iifname "virbr0" udp dport 53 accept
 #   Start of fragment order:50 rulename:default_in-ssh
   tcp dport {22} accept
 #   Start of fragment order:50 rulename:default_in-syncthing


--- /etc/nftables/puppet/inet-filter-chain-default_fwd.nft  2023-11-28 15:47:58.151874290 -0500
+++ /tmp/puppet-file20231128-15717-rv4jlv   2023-11-28 15:59:57.903321806 -0500
@@ -1,4 +1,10 @@
 # Start of fragment order:00 default_fwd header
 chain default_fwd {
+#   Start of fragment order:50 rulename:default_fwd-qemu_iip_v4
+  iifname "virbr0" ip saddr 192.168.122.0/24 accept
+#   Start of fragment order:50 rulename:default_fwd-qemu_io_internal
+  iifname "virbr0" oifname "virbr0" accept
+#   Start of fragment order:50 rulename:default_fwd-qemu_oip_v4
+  oifname "virbr0" ip daddr 192.168.122.0/24 ct state related,established accept
 # Start of fragment order:99 default_fwd footer
 }

Note that the network range matters here, it needs to match the one visible in the output of:

virsh net-dumpxml default

Also note that I previously included the nftables::rules::qemu class as is, but that broke virtd networking with this error:

error: internal error: Failed to apply firewall rules /usr/sbin/iptables -w --table nat --list-rules: # Warning: iptables-legacy tables present, use iptables-legacy to see them

The solution was to do the above masquerade => false. Or, in a diff:

--- /etc/nftables/puppet/ip-nat-chain-POSTROUTING.nft   2023-11-28 14:55:32.881506364 -0500
+++ /tmp/puppet-file20231128-9849-fc3war    2023-11-28 15:47:58.163874281 -0500
@@ -4,15 +4,5 @@
   type nat hook postrouting priority 100
 #   Start of fragment order:02 rulename:POSTROUTING-policy
   policy accept
-#   Start of fragment order:50 rulename:POSTROUTING-qemu_ignore_broadcast
-  ip saddr 192.168.122.0/24 ip daddr 255.255.255.255 return
-#   Start of fragment order:50 rulename:POSTROUTING-qemu_ignore_multicast
-  ip saddr 192.168.122.0/24 ip daddr 224.0.0.0/24 return
-#   Start of fragment order:50 rulename:POSTROUTING-qemu_masq_ip
-  ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 masquerade
-#   Start of fragment order:50 rulename:POSTROUTING-qemu_masq_tcp
-  meta l4proto tcp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 masquerade to :1024-65535
-#   Start of fragment order:50 rulename:POSTROUTING-qemu_masq_udp
-  meta l4proto udp ip saddr 192.168.122.0/24 ip daddr != 192.168.122.0/24 masquerade to :1024-65535
 # Start of fragment order:99 POSTROUTING footer
 }

The IP address distributed by dnsmasq also doesn't seem quite correct, as it's trying to reach 10.0.2.3 for some reason. I had to do this for DNS to work:

echo nameserver 192.168.122.1 > /etc/resolv.conf

This might be solved by hardcoding a DNS server in systemd-networkd or elsewhere.

IPv6 is configured by default, so if you're on a IPv4-only network, some problems are likely to occur. The fix is to edit the network and remove the <ip> block for IPv6:

service libvirtd stop
virsh net-destroy default
virsh net-edit default
virsh net-start default
service libvirtd start

Base image build

Then we can build an image using virt-builder:

virt-builder debian-9 --size=10G --format qcow2 \
  -o /var/lib/libvirt/images/stretch-amd64.qcow2 \
  --update \
  --firstboot-command "dpkg-reconfigure openssh-server" \
  --network --edit /etc/network/interfaces:s/ens2/ens3/ \
  --ssh-inject root:string:'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7CY6+aTLlk6epl1+TK6wIaHg1fageEfmKFgn+Yov+2lKFIhNRkcWznQVcyViVmC7iaZkEIei1gP9+0lrsdhewtTBjvkDNxR18aIORJsiH95FFjFIuJ0HQjrM1jOxiXhQZ0xLlnhFkxxa8j9l52HTutpYUU63e3lvY0CBuqh7QtkH3un7iT6EaqMR34yFa2ym35ag8ugMbczBwnTDJYn3qpL8gKuw3JnIp+qdSQb1sGdLcC4JN02E2/IY7iw8lzM9xVab1IgvemCJwS0C/Bt9LsmhCy9AMpaVFaAYjepgdBpSqIMa/8VcoVOrhdJWfIc7fLtt+njN1qojsPmuhsr1n' \
  --hostname stretch-amd64 --timezone UTC

This is not ideal, as it fetches the base image from libguestfs.org, in the clear (as opposed to debian.org infrastructure):

[   1.9] Downloading: http://libguestfs.org/download/builder/debian-9.xz

There is, fortunately, an OpenPGP signature on those images but it might be better to bootstrap using debootstrap (although bootstrapping using the above might be much faster).

Also notice how we edit the interfaces file to fix the interface name. For some reason, the interface detected by virt-builder isn't the same that shows up when running with virt-install, below. The symlink trick does not work: adding --link /dev/null:/etc/systemd/network/99-default.link to the virt-builder incantation does not disable those funky interface names. So we simply rewrite the file.

Finally, we inject our SSH key in the root account. The build process will show a root password but we won't need it thanks to that.

If the build fails with this error:

[ 156.9] Resizing (using virt-resize) to expand the disk to 10.0G
virt-resize: error: libguestfs error: /usr/bin/supermin exited with error 
status 1.

It might be that you ran out of space in /var/tmp. You can use TMPDIR to switch to a larger directory.

Autopkg builders

Images can also be built thanks to autopkgtest which itself delegates the job to vmdb2, with something like:

sudo autopkgtest-build-qemu stable /var/lib/libvirt/images/debian9-amd64-autopkgtest.qcow2

There are obviously many, many more options for building such images, that's just the ones I found the most practical.

Official images

An alternative way of getting a base image is to just download images from https://cloud.debian.org/. They have QCOW2 images that are minimal and can serve as a template for multiple VMs. For example, this downloads the latest bookworm build:

cd /var/lib/libvirt/images
curl -L -O https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-nocloud-arm64-daily.qcow2

The above is a really bare image. You might want cloud-init to make your life easier:

curl -L -O https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-genericcloud-arm64-daily.qcow2

Virtual machine creation

Then the virtual machine can be created and started with:

virt-install --import --noautoconsole \
  --memory 1024 \
  --name debian-12-amd64-test \
  --disk path=/var/lib/libvirt/images/debian-12-amd64-test.img 

The path argument can be simplified by using existing volume pools, which can be listed with:

# virsh pool-list
 Name                 State      Autostart 
-------------------------------------------
 boot-scratch         active     yes
 default              active     yes

[[!tip """Notice how the virsh command is called as root. That's not absolutely necessary, but by default when called as a user, it will connect to the user-specific session (qemu:///session) instead of the system-level one (qemu:///system). This can be worked around by using the --connect qemu:///system argument or by changing the default URI.