Tonight's fun was spent learning bpf's internals (the pseudo-machine code it uses). The point was to find out exactly how much effort it would take to add secure bpf support to jails. Ideally, we'd want to expose the bpf(4) device to any jail but only make available the traffic that is actually destined for the jail (or broadcast traffic).

It seems like you could get away with this, if you prefixed all jailed bpf filters with: (ip and (host [jail_ip] or multicast or broadcast)). I've got userland-code that does exactly this. Once I knew how to inject my own bpf code into an existing bpf_program struct, I was basically ready to go. The only other thing left was to figure out how, in the FreeBSD kernel, to figure out if you were in a jail and what that jail's IP was - turns out this is a trivial operation :)

Userland example code: pcapinject.c

Working Patch: bpf-jail.patch

The code in the patch is crappyish and has a pile of debug statements, but it does appear to work as intended.