Running Windows on FreeBSD using Bhyve

Yes, it’s possible since beginning of October if you run FreeBSD 11-CURRENT.

I have waited for quite a while for this. This means no more need for KVM or XEN or what have you.
Bhyve does it all.

It’s a bit rough around the edges. Biggest issue (at least for me) is the lack of video support but serial console is there and that is enough to get going. The documentation on windows installation is a bit lacking, but this will probably change quickly as it only just got pushed to CURRENT.

I have had it running for a day now and it seems stable to me.

Currently all the docs regarding Windows on Bhyve that are available (that I’m aware off) are here:

And really that is enough to get you going and if you have had some experience with bhyve before then you’re probably going to be ok.

These are my notes on getting Windows 2012 R2 running on Bhyve.

First, make sure bhyve is usable on your machine. Try booting some linux vm as a test.

Get an installation ISO.
I happen to have an MSDN subscription so this is where I got mine.

I used en_windows_server_2012_r2_with_update_x64_dvd_4065220.iso

You will have to modify the ISO, so make sure you have the tools for that:

pkg install cdrtools-devel p7zip  

Since there is no video output, the installation has to happen automatically. Luckily windows does support this. What you need is a Autounattend.xml file to be present in the root of your installation ISO file.

There are excellent examples available on github. So let’s clone that:

git clone  

I used win2012r2_AutoUnattend.xml file and the only modification I did was adding the product key:
Find the line

<ProductKey />  

And replace it with actual key:


Then rename the file to AutoUnattend.xml and keep it as you’ll need it in a moment. I kept mine in ~/

Next step, get the virtio drivers:


Extract the windows installation ISO. I extract mine into destination directory win2k12r2 like so:

7z -owin2k12r2 x en_windows_server_2012_r2_with_update_x64_dvd_4065220.iso  

change into this directory and copy over the prepared AutoUnattend.xml file so that it would now be in the root of the unpacked ISO:

cd win2k12r2  
cp ~/AutoUnattend.xml .  

now, you need to add the virtio drivers.
Again, while being in the root of the unpacked ISO, create a directory virtio and unpack the virtio drivers there:

mkdir virtio  
tar xfv ~/virtio-win-0.1.96.iso  

These are all the modifications needed, so you can go ahead and re-pack the ISO:

mkisofs \  
    -b boot/ -no-emul-boot -c BOOT.CAT \
    -iso-level 4 -J -l -D \
    -N -joliet-long \
    -relaxed-filenames -v \
    -V "Custom" -udf \
    -boot-info-table -eltorito-alt-boot -eltorito-platform 0xEF \
    -eltorito-boot efi/microsoft/boot/efisys_noprompt.bin \
    -no-emul-boot \
    -o ~/win_repack.iso .

Now, one last dependency before you boot it, you need the UEFI binary as modern Windows versions require UEFI:


Create a file that will serve as the disk for your Windows VM

truncate -s 30G win.img  

Make sure that nmdm module is loaded as we’ll use serial console:

kldload nmdm  

You can now start the installation:

bhyve \  
      -c 2 \
      -s 0,hostbridge \
      -s 3,ahci-hd,win.img \
      -s 4,ahci-cd,win_repack.iso \
      -s 10,virtio-net,tap0 \
      -s 31,lpc \
      -l com1,/dev/nmdm0A \
      -l com2,/dev/nmdm1A \
      -l bootrom,BHYVE_UEFI_20151002.fd \
      -m 2G -H -w \

Once the bhyve process is running, you can connect to the serial port from another console:

cu -l /dev/nmdm0B -s 9600  

Once connected, you’ll find yourself in Special Administration Console. Type ch to get a list of channels and once the installation has started you can follow the progress by connecting to channel #1 by typing:

ch -si 1  

The first stage of installation will copy over the files and eventually the machine will reboot. At this point the bhyve process will exit and you’ll have to restart it. This time, do it without attaching the ISO:

bhyve \  
      -c 2 \
      -s 0,hostbridge \
      -s 3,ahci-hd,win.img \
      -s 10,virtio-net,tap0 \
      -s 31,lpc \
      -l com1,/dev/nmdm0A \
      -l com2,/dev/nmdm1A \
      -l bootrom,BHYVE_UEFI_20151002.fd \
      -m 2G -H -w \

This stage again takes a bit of time but eventually you are done.

I ran vmstat to see when it settles down.

Once it has, you can type i in the SAC console to find out the IP address that your new Windows VM got from the DHCP and then you can connect over RDP.

The default username is Administrator and the default password is Test123 but you can change that in the AutoUnattend.xml

That’s it. Can’t say that it’s straightforward, but it’s certainly doable and if you’ve done it once then it all makes sense.

NFS exports with ZFS

If you have a ZFS based storage server and other machines in the network, it would be a shame not to share the awesome storage with them.

I have 2 servers at home.
frisbie is the ZFS storage machine and yotta is just a powerful machine but with only one disk, so I’d like to run VMs on it but I’d also like to avoid crying whenever the disk on it dies 🙂

First, enable and start up all the required daemons for NFS server to function.

sysrc -f /etc/rc.conf nfs_server_enable="YES"  
sysrc -f /etc/rc.conf rpc_lockd_enable="YES"  
sysrc -f /etc/rc.conf rpc_statd_enable="YES"  
for service in {nfsd,statd,lockd}; do service $service start; done  

So I create a new zfs filesystem on Frisbie

zfs create data/yotta_bhyve  

And then set it to be shared via NFS:

zfs set sharenfs="-maproot=root yotta,rw" data/yotta_bhyve  

The maproot option on FreeBSD is the same as nosquashroot on linux. Since I will be using it as root on my client I need it to work. Might not be elegant, but hey…

On the client:

sysrc -f /etc/rc.conf nfs_client_enable="YES"  
service nfsclient start  
mkdir /bhyve  
mount frisbie:/data/yotta_bhyve /bhyve  

And afterwards it makes sense to add something like this to /etc/fstab

frisbie:/data/yotta_bhyve /bhyve        nfs     rw      0       0  

Playing with PF on FreeBSD

I have been meaning to try out both IPFW and PF.

After reading bsdnow pf tutorial and skimming over nice FreeBSD handbook’s pf section I was a bit hesitant to try PF on FreeBSD as it was said to have slightly older version from OpenBSD and that would cause syntax to differ. So I tried out IPFW first and it was really simple to set up however after adding some rules I realized that I’m not loving the syntax.

So I thought that it’s time to give PF a go (especially since I’m not too comitted to IPFW yet) and it turns out it’s not so bad with those syntax differences, though I did run into one.

Once I wrote up a minimal ruleset with PF and tried to load load it I was getting syntax error:

[fx@badger ~]$ sudo pfctl -f /etc/pf.conf
   ALTQ support in kernel
ALTQ related functions disabled  
/etc/pf.conf:21: syntax error
pfctl: Syntax error in config file: pf rules not loaded  

However there really was nothing eye catching on that line.
So after some experimenting and more careful re-reading of handbook it turns out that FreeBSD’s PF version wants all the lists to be enclosed in curly braces (just as it shows on the nice FreeBSD handbook). Doooh.

After changing that one syntax problem PF complained a bit more about rules ordering

/etc/pf.conf:13: Rules must be in order: options, normalization, queueing, translation, filtering

But that was an easy fix of changing the order (which totally makes sense).

Then I tested rebooting the machine and realized that even though I have /etc/rc.conf set up correctly, the PF is not loading the rules on boot.
Enlightment came when I realized that this is because I am using my OpenVPN tun0 interface in the rules and it is not available until after OpenVPN has come up. So I changed the rules to use vpn IP instead of the nic and all was good.

I really like the fact that PF does some syntax checking before it loads the rules. Much better than realizing that your SSH session has hung because you made a noobish mistake in the ruleset 🙂
And also loving the nice stats and counters it can print out.

Overall a positive first experience with PF.