NFS exports with ZFS

If you have a ZFS based storage server and other machines in the network, it would be a shame not to share the awesome storage with them.

I have 2 servers at home.
frisbie is the ZFS storage machine and yotta is just a powerful machine but with only one disk, so I’d like to run VMs on it but I’d also like to avoid crying whenever the disk on it dies 🙂

First, enable and start up all the required daemons for NFS server to function.

sysrc -f /etc/rc.conf nfs_server_enable="YES"  
sysrc -f /etc/rc.conf rpc_lockd_enable="YES"  
sysrc -f /etc/rc.conf rpc_statd_enable="YES"  
for service in {nfsd,statd,lockd}; do service $service start; done  

So I create a new zfs filesystem on Frisbie

zfs create data/yotta_bhyve  

And then set it to be shared via NFS:

zfs set sharenfs="-maproot=root yotta,rw" data/yotta_bhyve  

The maproot option on FreeBSD is the same as nosquashroot on linux. Since I will be using it as root on my client I need it to work. Might not be elegant, but hey…

On the client:

sysrc -f /etc/rc.conf nfs_client_enable="YES"  
service nfsclient start  
mkdir /bhyve  
mount frisbie:/data/yotta_bhyve /bhyve  

And afterwards it makes sense to add something like this to /etc/fstab

frisbie:/data/yotta_bhyve /bhyve        nfs     rw      0       0  

Docker on FreeBSD

Docker is now available of FreeBSD.
How cool is that?
And setup is so simple that it would be a shame not to try it out:

sudo pkg install docker-freebsd ca_root_nss  

As seems to be the case quite often with FreeBSD packages, after installing the package you get quite nice instructions on what to do next:

[fx@yotta /usr/home/fx]$ sudo pkg install docker-freebsd
Updating yotta.tln repository catalogue...  
Fetching meta.txz: 100%    264 B   0.3kB/s    00:01  
Fetching packagesite.txz: 100%   56 KiB  57.3kB/s    00:01  
Processing entries: 100%  
yotta.tln repository update completed. 209 packages processed.  
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:  
        docker-freebsd: 06252015
        go: 1.4.2,1

The process will require 142 MiB more space.  
23 MiB to be downloaded.

Proceed with this action? [y/N]: y  
Fetching docker-freebsd-06252015.txz: 100%    3 MiB   2.8MB/s    00:01  
Fetching go-1.4.2,1.txz: 100%   21 MiB  21.6MB/s    00:01  
Checking integrity... done (0 conflicting)  
[1/2] Installing go-1.4.2,1...
[1/2] Extracting go-1.4.2,1: 100%
[2/2] Installing docker-freebsd-06252015...
[2/2] Extracting docker-freebsd-06252015: 100%
Message for docker-freebsd-06252015:  
Docker requires a bit of setup before usage.

You will need to create a ZFS dataset on /usr/docker

# zfs create -o mountpoint=/usr/docker <zroot>/docker

And lastly enable the docker daemon  
# sysrc -f /etc/rc.conf docker_enable="YES"
# service docker start


Starting the docker service will also add the following PF rule:

nat on ${iface} from to any -> (${iface})

Where $iface is the default NIC on the system, or the value  
of $docker_nat_iface. This is for network connectivity to docker  
containers in this early port. This should not be needed in future  
versions of docker.  

So I did what the nice instruction said to do:

sudo zfs create -o mountpoint=/usr/docker zroot/docker  
sudo sysrc -f /etc/rc.conf docker_enable="YES"  
sudo service docker start  

It really is that simple.

So now, to run a Debian container, simply download and run it by doing:

sudo docker pull debian  

and then run a bash shell inside of Debian container

sudo docker run -t -i debian /bin/bash  


[fx@yotta /usr/home/fx]$ sudo docker run -t -i debian /bin/bash
root@:/# uname -a  
Linux  2.6.32 FreeBSD 11.0-CURRENT #0 r285684: Sun Jul 19 10:19:35 UTC 2015 x86_64 GNU/Linux  

One slight issue is that it is still experimental and some stuff does not seem to work.
For example in a Debian Wheezy container mostly everything segfaults:

[fx@yotta ~]$ docker run -t -t debian:wheezy /bin/bash
jail: /bin/bash: exited on signal 11  

stracing does not help because it does not work:

strace: test_ptrace_setoptions_for_all: PTRACE_TRACEME doesn't work: Function not implemented  

But other than that stuff works and it does not get much more simpler than that.
You get Docker + ZFS.
One reason less to run linux as your host os.

Deploying Ghost blog

Yesterday I started using Ghost and I simply ran
npm start in a Tmux session.

This is obviously not the best way to do it so I have made some improvements now.

After reading a bit of deployment docs, which by the way are very nice, I decided to give pm2 a try.

So if you already have Ghost blog configured then you only need to install pm2 and maybe create some simple start script.

sudo npm install -g pm2  

Then, to launch and daemonize ghost, simply do:

pm2 start index.js  

To check on its status

pm2 show index  

Pretty neat.
The (super complicated) start script for my ghost is now:

NODE_ENV=production pm2 start index.js  
pm2 show index  

Ghost on FreeBSD

I have been thinking for couple of days now that I should give Ghost a try.
Since I am re-starting blogging and all, might as well try different blog engines while I’m at it.

It would be no fun installing it on linux, so I decided to use FreeBSD VM on Digitalocean + a Jail inside of it.

Downloaded and unpacked ghost and then hit a bug.
Nodejs sqlite module would not install.

2695 error FreeBSD 10.1-RELEASE-p10  
2696 error argv "node" "/usr/local/bin/npm" "install" "sqlite3"  
2697 error node v0.12.6  
2698 error npm  v2.12.1  
2699 error code ELIFECYCLE  
2700 error sqlite3@3.0.8 install: `node-pre-gyp install --fallback-to-build`  
2700 error Exit status 1  
2701 error Failed at the sqlite3@3.0.8 install script 'node-pre-gyp install --fallback-to-build'.  
2701 error This is most likely a problem with the sqlite3 package,  
2701 error not with npm itself.  
2701 error Tell the author that this fails on your system:  
2701 error     node-pre-gyp install --fallback-to-build  
2701 error You can get their info via:  
2701 error     npm owner ls sqlite3  
2701 error There is likely additional logging output above.  
2702 verbose exit [ 1, true ]  

Googled a bit and found Ghost forum thread and a page on ghost wiki

The problems there were related but also a bit different and those solutions did not apply fully for me. I found though by reading the debug log that actually I was missing gmake and python in my jail.
I managed to resolve those errors by installing gmake and python and by updating npm modules. But later on discovered that much simpler solution is to install sqlite3 from FreeBSD package.

I have now tested this on a clean jail and at least from my perspective to successfully install ghost on FreeBSD 10.1 you only need to:

sudo pkg install node npm sqlite3  
mkdir superduperblog  
cd superduperblog  
unzip ../  
npm install --production  
npm start  

Super simple.

Playing with PF on FreeBSD

I have been meaning to try out both IPFW and PF.

After reading bsdnow pf tutorial and skimming over nice FreeBSD handbook’s pf section I was a bit hesitant to try PF on FreeBSD as it was said to have slightly older version from OpenBSD and that would cause syntax to differ. So I tried out IPFW first and it was really simple to set up however after adding some rules I realized that I’m not loving the syntax.

So I thought that it’s time to give PF a go (especially since I’m not too comitted to IPFW yet) and it turns out it’s not so bad with those syntax differences, though I did run into one.

Once I wrote up a minimal ruleset with PF and tried to load load it I was getting syntax error:

[fx@badger ~]$ sudo pfctl -f /etc/pf.conf
   ALTQ support in kernel
ALTQ related functions disabled  
/etc/pf.conf:21: syntax error
pfctl: Syntax error in config file: pf rules not loaded  

However there really was nothing eye catching on that line.
So after some experimenting and more careful re-reading of handbook it turns out that FreeBSD’s PF version wants all the lists to be enclosed in curly braces (just as it shows on the nice FreeBSD handbook). Doooh.

After changing that one syntax problem PF complained a bit more about rules ordering

/etc/pf.conf:13: Rules must be in order: options, normalization, queueing, translation, filtering

But that was an easy fix of changing the order (which totally makes sense).

Then I tested rebooting the machine and realized that even though I have /etc/rc.conf set up correctly, the PF is not loading the rules on boot.
Enlightment came when I realized that this is because I am using my OpenVPN tun0 interface in the rules and it is not available until after OpenVPN has come up. So I changed the rules to use vpn IP instead of the nic and all was good.

I really like the fact that PF does some syntax checking before it loads the rules. Much better than realizing that your SSH session has hung because you made a noobish mistake in the ruleset 🙂
And also loving the nice stats and counters it can print out.

Overall a positive first experience with PF.

Openvpn on FreeBSD

All of my linux servers are connected over OpenVPN and now came the turn for my new FreeBSD droplet to join them.

Ran into one small issue. Upon installing and configuring OpenVPN I was getting this error:

Sun Jul 12 17:59:49 2015 us=222242 Cannot allocate TUN/TAP dev dynamically  
Sun Jul 12 17:59:49 2015 us=222330 Exiting due to fatal error  

Since I am using tun and not tap, tried manually loading tun module:

[fx@badger ~]$ sudo kldload if_tun
kldload: can't load if_tun: module already loaded or in kernel  

But that just showed that this is not the problem.
Tried creating tun interface manually and no probs:

[fx@badger ~]$ sudo ifconfig tun0 create
[fx@badger ~]$ sudo ifconfig tun0
tun0: flags=8010<POINTOPOINT,MULTICAST> metric 0 mtu 1500  

That pointed me to my openvpn.conf config and sure enough I had specified my VPN device as “VPN”:

dev VPN  

Ths works great in Linux but apparently not in FreeBSD. Changed that to tun0 and problem resolved. Yay.

So just for the record, working OpenVPN installation steps:

sudo pkg install openvpn  

Enable OpenVPN starting at boot by adding this to /etc/rc.conf:


Configure it, the default config file is /usr/local/etc/openvpn/openvpn.conf and looks like you have to create /usr/local/etc/openvpn directory manually.
Working config:

dev-type tun  
dev tun0  
proto tcp  
remote 1194  
resolv-retry infinite  
# certs that you got from vpn server
ca /usr/local/etc/openvpn/keys/ca.crt  
cert /usr/local/etc/openvpn/keys/vpn-client.crt  
key /usr/local/etc/openvpn/keys/vpn-client.key  
# don't forget to make sure this is rotated
log-append /var/log/openpvn.log  
ns-cert-type server  
verb 4  
mute 20  

Testing jekyll

It has been couple of years at least (more like three) and I am thinking about blogging again.
Why? Well, why not.

So let’s see if this becomes yet another failed attempt or not.

This time I’m using Jekyll as the blog engine.

Easy to set up and I relly like the idea of static content. Otherwise with the various dynamic options it would be yet another thing that needs maintaining and security patching.