Although performance is not its strong point, QEMU can be a pretty useful tool for running OpenBSD instances on OpenBSD. Some years back, kqemu support allowed for much better OpenBSD performance but, unfortunately, kqemu support was removed quite some time ago. While we are waiting for bhyve to support OpenBSD guests and eventually maybe OpenBSD as a host, qemu will have to do.
A few days ago, I started working on setting up an OpenBSD instance in qemu to do some custom package building. I immediately ran into some issues with bridging the qemu network interface to the physical interface. The OpenBSD qemu port's README file gives some indication of what to do but I couldn't seem to get it functioning while running as a user other than root. I still plan to get this working right but, for the time being, I am using a simpler method running as root for now.
Before we get started, I should note that qemu provides its own built-in DHCP service and networking but setting up a bridged connection yourself allows for much greater flexibility and control over the process. There is also additional functionality not available with the built-in facilities.
I had talked to Andrew Fresh when we were in Portland in July about some scripts he wrote to launch flashrd instances under qemu. I encouraged him to make them available publicly and he did just that shortly thereafter. Those scripts were very helpful in this process, especially for the idea to use vether(4).
Settings up the interfaces and NAT
The basic concept is that we will have a virtual interface (vether(4)) that has an IP address, NAT that network through my main network interface, provide DHCP addresses to the virtual interface, and then bridge(4) the tun(4) interface to the virtual interface.
If you're brand new to OpenBSD, I would suggest you familiarize yourself with the system conventions and configurations before getting into all of this but this is the general idea of what is needed.
First, I need to create a vether(4) interface. I'm going to do this by creating a hostname.if(5) for this interface. I picked a private address space range for this purpose in the 192.168.0.0/16 block. The /etc/hostname.vether0 file contains this:
inet 192.168.54.1 255.255.255.0 NONE
All of my qemu instances will be running on the 192.168.54.0/24 network which I will then NAT through my main network. I'm using macros in my /etc/pf.conf file but I will leave those out for the purposes of clarity. I added the following lines to /etc/pf.conf on my 5.4-current laptop above the default block and pass rules:
ext_if="iwn0"
int_if="vether0"
match out from $int_if:network to any nat-to ($ext_if:0)
These additions to /etc/pf.conf modify the existing rules in order to match all packets from the internal virtual network that are heading to another network. The $inf_if:network statement refers to the network associated with $int_if which is replaces with vether0 in my case. The network ends up being 192.168.54.0/24. The ($ext_if:0) part refers to the first address (referenced by :0) of the $ext_if which happens to be the address of the iwn0 interface. The parentheses around $ext_if:0 tell pf to update the address if it changes because it is a dynamic address. Without the parentheses, pf would statically use the current IP address which will change in the future.
Running pfctl -sr gives the following output:
match out inet from 192.168.54.0/24 to any nat-to (iwn0)
block drop all
pass all flags S/SA
block drop in on ! lo0 proto tcp from any to any port 6000:6010
The first line is the only one I added. Lines 2-4 were there by default. The macro definitions are not shown by pfctl -sr but are there in /etc/pf.conf.
The final part of the network interfaces is configuring tun0 and bridge0. I added the following /etc/hostname.tun0 which enables layer 2 operation via the link0 flag:
link0 up
I added the following /etc/hostname.bridge0:
add vether0 add tun0 up
With this completed, restart your network interfaces with:
sh /etc/netstart
DHCP and DNS
After the vether0 and pf.conf configs, the next step is configuring /etc/dhcpd.conf. I simply took the default /etc/dhcpd.conf and modified it according to what I needed:
option domain-name "example.com";
option domain-name-servers 192.168.54.1;
subnet 192.168.54.0 netmask 255.255.255.0 {
option routers 192.168.54.1;
range 192.168.54.100 192.168.54.199;
}I then added the appropriate entry to /etc/rc.conf.local to launch dhcpd at startup:
dhcpd_flags=""
After getting dhpcd ready to go, the next step is providing a DNS resolver. Although it is possible to use some other resolver on the network, the easiest and most error-proof option is to install unbound from ports and configure it to allow requests on the localhost addresses and 192.168.54.1. I won't go into the details here but this should be relatively easy once you install the unbound package or build it from ports and then modify /var/unbound/etc/unbound.conf as needed. Finally add unbound to the pkg_scripts line of /etc/rc.conf.local.
With these steps done, we can now move on to working with qemu directly.
Launching a VM
The first step is creating a virtual machine image using qemu-img.
qemu-img -f qcow2 first-image.img 10G
In this case, I created a 10GB image in qcow2 format. This is also a good time to download the latest iso image for OpenBSD from any OpenBSD mirror. I used install54.iso from the latest snapshot.
I used the following script to launch the qemu instance of OpenBSD with the intention of installing from the iso image:
#!/bin/sh
export ETHER=vether0
export BRIDGE=bridge0
qemu-system-x86_64 -m 512 -monitor stdio -vnc :0 -no-fd-bootchk \
-net nic,model=e1000,macaddr=52:54:00:4e:62:8a -net tap \
-hda first-image.img -cdrom install54.iso -boot d
This script launches an amd64 instance of OpenBSD with 512MB of memory, a VNC server running on port 5900, no floppy, a network interface using the e1000 driver with MAC address of 52:54:00:4e:62:8a (which should be changed), the network interface is bridged using tap, the 10GB image configured as the hard drive, the iso image configured as the cdrom, and the VM will boot from d which is the cdrom.
At this point you should have ssvnc installed from packages or ports and be able to connect to the newly created vm using vncviewer 127.0.0.1:5900 run from the local command line.
After installation, shutdown the VM by typing quit at the qemu prompt after you have done halt -p from the VM's command line.
Next, modify the script by removing the -cdrom and -boot flags:
#!/bin/sh
export ETHER=vether0
export BRIDGE=bridge0
qemu-system-x86_64 -m 512 -monitor stdio -vnc :0 -no-fd-bootchk \
-net nic,model=e1000,macaddr=52:54:00:4e:62:8a -net tap \
-hda first-image.img
This will launch your OpenBSD instance. Access it via vncviewer as above and you are off and running. You can copy the script to make additional instances but create a new disk image and change the MAC address for each new instance.
Aftermath
I'm hoping that bhyve will support OpenBSD eventually. Although qemu works for now, it is very slow. I compiled dovecot and associated packages inside the qemu instance and it took hours. I was surprised at how long it actually took. Where qemu is especially useful is to test router configs and such like Andrew Fresh is doing as I mentioned above. Another idea I've had is to run a qemu instance of Debian and then rsync from it to OpenBSD in order to use Dropbox with OpenBSD. It would probably work fine but is a hack at best.
Until bhyve or other solutions are faster, I'm going to use SmartOS in the datacenter in order to facilitate OpenBSD VMs. I have been using it for about six months in brief stints and it seems to work quite well. OpenBSD also runs just fine under VMware ESXi with local datastores but I would rather focus on open source alternatives even though I am a VMware Certified Professional. If VMware completely moves away from its Windows requirement I would be more interested and it seems things are heading that direction.
In particular, thanks to Andrew Fresh for his scripts and Michael Dexter for keeping me up to date on bhyve and its progress.