Tonight's fun was spent learning bpf's internals (the pseudo-machine code it uses). The point was to find out exactly how much effort it would take to add secure bpf support to jails. Ideally, we'd want to expose the bpf(4) device to any jail but only make available the traffic that is actually destined for the jail (or broadcast traffic).

It seems like you could get away with this, if you prefixed all jailed bpf filters with: (ip and (host [jail_ip] or multicast or broadcast)). I've got userland-code that does exactly this. Once I knew how to inject my own bpf code into an existing bpf_program struct, I was basically ready to go. The only other thing left was to figure out how, in the FreeBSD kernel, to figure out if you were in a jail and what that jail's IP was - turns out this is a trivial operation :)

Userland example code: pcapinject.c

Working Patch: bpf-jail.patch

The code in the patch is crappyish and has a pile of debug statements, but it does appear to work as intended.

I wrote a script a while ago to build a very tiny freebsd world. It's extremely fast and only builds a freebsd image in approximately 10 megs of space. It lets you quickly create new jail enviroments or system images for small embedded platforms.

If you look at the script itself, you'll get an idea of what it installs. I used a variant of this script to build the system I run on my Soekris net4501 which runs FreeBSD and is under 20 megs.

There are lots of "make a small freebsd system" scripts, but most of the ones I've found rely heavily on 'buildworld' and what not. This takes a live system and copies the binaries you need, then uses ldd(1) to track down required libraries.

view minibsd.sh

Example usage:

kenya(~/t) % rm -rf ./soekris/
kenya(~/t) % time sudo ./minibsd.sh
sudo ./minibsd.sh   0.16s user 0.65s system 61% cpu 1.326 total
kenya(~/t) % sudo chroot ./soekris /bin/sh
# pwd
/
# exit

Simple jail config (rc.conf):

jail_enable="YES"
jail_list="test"
jail_test_rootdir="/home/jls/t/soekris"
jail_test_hostname="test"
jail_test_ip="10.1.1.1"
jail_test_interface="tl1"

Put something simple in this jail's rc.conf (/home/jls/t/soekris/etc/rc.conf):

sshd_enable="YES"
sendmail_enable="NONE"

Let's test the jail now:

kenya(~/t) % sudo /etc/rc.d/jail start
Configuring jails:.
Starting jails: 

At this point, it's probably hung (assuming you enabled sshd). If you hit CTRL+T you'll see what command has the foreground and what it's doing.* This is because it's prompting you (output is directed to JAILROOT/var/log/console.log) for entropy for the ssh-keygen. Smash a few keys then hit enter. It'll finish eventually.

kenya(~/t) % sockstat -4 | grep 10.1.1.1:22 
root     sshd       2258  3  tcp4   10.1.1.1:22           *:*

Our sshd is running happily inside that jail we made. This whole process took about 5 minutes.

* FreeBSD's CTRL+T terminal handler feature has to be the best thing ever invented. I wish Linux had something like this. Here's what hitting CTRL+T when running cat looks like:

kenya(~) % cat
load: 0.45  cmd: cat 2324 [ttyin] 0.00u 0.00s 0% 600k
load: 0.42  cmd: cat 2324 [ttyin] 0.00u 0.00s 0% 600k

It clearly shows you the command name, the pid, and the syscall-type-thing it's doing. Clearly cat is waiting for input from the tty. <3 FreeBSD.