Tonight's fun was spent learning bpf's internals (the pseudo-machine code it uses). The point was to find out exactly how much effort it would take to add secure bpf support to jails. Ideally, we'd want to expose the bpf(4) device to any jail but only make available the traffic that is actually destined for the jail (or broadcast traffic).

It seems like you could get away with this, if you prefixed all jailed bpf filters with: (ip and (host [jail_ip] or multicast or broadcast)). I've got userland-code that does exactly this. Once I knew how to inject my own bpf code into an existing bpf_program struct, I was basically ready to go. The only other thing left was to figure out how, in the FreeBSD kernel, to figure out if you were in a jail and what that jail's IP was - turns out this is a trivial operation :)

Userland example code: pcapinject.c

Working Patch: bpf-jail.patch

The code in the patch is crappyish and has a pile of debug statements, but it does appear to work as intended.

I was trying to install something from ports tonight, and package fetching kept hanging for a long time. Turns out, for whatever reason, it was trying to connect to some of the servers on ipv6, which would fail and it would then try ipv4. This is probably due to some change I made recently but forgot about.

At any rate, I needed a quick way to kill ipv6 activity on the machine. Trying '/etc/rc.d/network_ipv6 onestop' didn't seem to have the desired effect, so I added this rule to my pf.conf:

block return out inet6 all

Everything is happy now. Probably not the most optimal solution, but I'm not looking for the optimal solution right now.

I setup ccache again (trivial) to help me with building freebsd repeatedly. I noticed that much of the time spent in the kernel build process was in building dependency lists using awk.

Why couldn't we apply the ccache idea to everything else? If the same input always means the same output, then we could cache it if it is computationally expensive to compute that output.

Script: dcache.sh

Above is a hack that runs like ccache, but tracks all files created by the process (and its subprocesses). Here's a sample run, of counting the number of lines in a file with awk and outputing the result (within awk) to another file.

% /usr/bin/time ./dcache.sh awk '{x++} END { print "Total records: " x > "/tmp/hello"}' bigdata
Running original...
        1.60 real         0.05 user         0.74 sys
% cat /tmp/hello
Total records: 1000000

# Remove the old output file
% rm /tmp/hello

# Rerun it again, unmodified, and it will use the cached output.
% /usr/bin/time ./dcache.sh awk '{x++} END { print "Total records: " x > "/tmp/hello"}' bigdata
Using cache...
        0.06 real         0.00 user         0.06 sys
% cat /tmp/hello
Total records: 1000000

It doesn't work with everything just yet, but the problems seem to be with truss's behavior and not the script's fault, like sometimes truss hangs, or it doesn't follow a fork like it should. Beyond truss problems, the scripts doesn't track file renames. It also doesn't understand how to figure out what the input files for each command is. Ideally it would checksum any inputs and use that as the cache key; currently it only checksums the commandline arguments and not the external files being used (such as 'bigfile').

I started initially without using truss, but awk doesn't call open(2) via libc when it opens files, for some reason, and I can't figure out a clean way to capture specific function calls from a process (even a child process).

Dtrace would be sexy here, but it is unavailable in the main freebsd trunk.

The speedup is pretty obvious for cpu-intensive things, but the real test is to see how it performs when working properly and hooked into the freebsd kernel build.

I wrote a script that will go through every variable named MASTER_SITE_[something] in bsd.sites.mk and basically sort each variable's contents by ping time.

The implementation uses asyncore in python, and uses tcp connections instead of pinging (pinging is not guaranteed to work). The output is sorted by fastest response time and in a format suitable for Makefile (and thus /etc/make.conf).

% python fastest_site.py
 => Checking servers for MASTER_SITE_GENTOO (77 servers)
MASTER_SITE_GENTOO=\
        ftp://ftp.ecc.u-tokyo.ac.jp/GENTOO/%SUBDIR%/ \
        ftp://gentoo.kems.net/pub/mirrors/gentoo/%SUBDIR%/ \
        ftp://files.gentoo.gr/%SUBDIR%/ \
... <output cut> ...

I recommend that you send the output to a separate file, such as /usr/local/etc/ports_sites.conf, and add the following line to /etc/make.conf:

.include "/usr/local/etc/ports_sites.conf"

Now generate the file:

% python fastest_site.py > /usr/local/etc/ports_sites.conf
 => Checking servers for MASTER_SITE_GENTOO (77 servers)
 => Checking servers for MASTER_SITE_TCLTK (11 servers)
 => Checking servers for MASTER_SITE_GET_E (11 servers)
 => Checking servers for MASTER_SITE_BERLIOS (4 servers)
...

Download: fastest_sites.py

I've had a src commit bit in freebsd for a while, and I haven't done much with it. Yes, I suck. I'm working on getting my mouse code into the tree, finally, after almost 2 years of its life and almost 2 years of my slacking off on getting it ready for submission.

I think one of the main reasons I've directed energy elsewhere is because there's a (from my perception) thick metawork process to get real work done. Culture shock, mostly. Almost all of the tools and methods are different from my own. My experience at Google has given me good practice in dealing with systems foreign to me, so why do I hesitate to work on FreeBSD stuff?

Outside of the processes involved in getting code into the FreeBSD source tree, one of the main problems I've had working on specifically kernel changes in FreeBSD are that I haven't come up with a good solution for separating workspaces other than simply creating a new virtual machine for each logical workspace. In Perforce, you can create multiple clients and work on independent changes in each client. In userland code, you can simply just build a new binary in a different directory, and you can test both binaries independently.

With kernels, I have a hard time multitasking. Not specifically multitasking different kernels, but if I'm making kernel and userland changes which are unrelated to eachother, I can't safely test a new kernel on the same system as a userland change. Isolating these as easy as making a new virtual machine, but copying virtual machines is not as fast and easy as, say, making a new perforce client.

I haven't come up with a good solution yet, but I'm sure someone else has and perhaps I'll build on that. Maybe some kind of hack where I would use a pristine, read only system image and all changes would be written to a memory filesystem on top of that pristine image? But this basically means all systems have to have the same pristine image (copying the image is nontrivial in time)...

Hopefully some of this makes sense. I'm open to suggestions :)

I finally picked up some small parts (leds, breadboard, leads, multimeter) from Fry's to start work on my universal remote project. Yes, I know you can buy universal remotes. I want to learn more electronics, so why not use this as a starter project?

Since my Soekris box is now free of it's router duties, I can use it for this project. To do this, I'll need two pieces: an infrared receiver, and an infrared emitter. I bought an infrared emitter led today, and I just need to buy a receiver online (they're like $2).

Before getting there, I needed to learn how to drive the GPIO ports on the net4501.

It was pretty simple to do, nd after hooking up a few wires I had an LED that blinked after a few hours of reading and hacking. The code itself was trivial to write, I just had to learn how to talk to the GPIO ports.

After writing the blinking code, I decided the next step was clearly to add fading to the LED. This is commonly done with PWM (pulse-width modulation). Apple made this technique famous with its "breathing" LED lights on the monitors and laptops when the devices were in sleep mode.

At any rate, I have successfully written code that makes both the error led and the PIO5 (GPIO 0) pin "breathe".

Download glow.c

I've definitely spent 10+ hours in the past 2 days trying to get this system upgraded from 6.0 to 7.0...

Why? First, I was using the wrong power supply, so it would randomly reboot during startup (4 hours, after before I figured that one out, after compiling zillions of kernels, even trying GENERIC from 6.2, 6.3, and 7.0). Then, I couldn't get a stable system image that would boot successfully. It would halt trying to run /sbin/init. Not sure why.

I was initially making my own image with mdconfig, and rsyncing the entire system into a file-backed fs. Then I dd'd this image to my compactflash and put it in the net4501. I gave up on that after several hours, and did this instead:

1. Bring up a new vmware instance with 2 disks. 1 800mb disk and 1 5gb disk.
2. Install 7.0-RC1, only install kernel and base.
3. Mount the 5gb disk (newfs /dev/sd1, mount /dev/sd1 /usr/src) and unpack the kernel sources there. Build kernel (For the soekris box).
4. Make any necessary config changes (serial console, etc)
5. Shutdown machine
6. Use qemu-img to convert the vmware disk into a raw disk image
7. dd the new image to compact flash.
8. Rejoice. I have a working net4501 now.

In testing in qemu, I get random timeouts talking to ad0. On the soekris board, I get random timeouts to ad0. wtf :(

Boot into safe-mode, and everything seems find. (Safe mode turns off DMA, iirc)

Using slight variations on the techniques mentioned in my previous post, I've got a vmware instance running Fedora 8 that permits any and all logins. These login sessions are logged with script(1).

Fedora 8 comes with selinux enabled by default. This means sshd was being denied permission to execute my special logging shell. The logs in /var/log/audit/ explained why, and audit2allow even tried to help make a new policy entry for me. However, I couldn't figure out (read: be bothered to search for more than 10 minutes) how to install this new policy. In searching, I found out about chcon(1). A simple command fixed my problems:

chcon --reference=/bin/sh /bin/sugarshell

The symptoms prior to this fix were that I could authenticate, but upon login I would get a '/bin/sugarshell: Permission Denied' that wasn't logged by sshd.

There are plenty of honeypot software tools out there, but I really wasn't in the mood for reading piles of probably-out-of-date documentation about how to use them. This hack (getpwnam + pam_permit + logging shell) took only a few minutes.

As a bonus, I found a feature in Fedora's yum tool that I like about freebsd's packaging system: It's trivial to ask "Where did this file come from?" Doing so made me finally look into how to do it in Ubuntu.

FreeBSD: pkg_info -W /usr/local/bin/ssh

/usr/local/bin/ssh was installed by package openssh-portable-4.7.p1,1

Fedora: yum whatprovides /usr/bin/ssh

openssh-server.x86_64 : The OpenSSH server daemon

Ubuntu: dpkg -S /usr/bin/ssh

openssh-client: /usr/bin/ssh

Let's see what I catch.

Put the vmware-remotemks' program where vmware console wants it

Symlink vmware-remotemks to /lib/vmware-server-console/bin/vmware-remotemks

Mount linprocfs to /proc

mount -t linprocfs - /proc

Hack fix for vmware dep library

From vmware-server-console-distrib/lib/bin/:
for i in ../lib/lib*/*; do ln -s $i `basename $i`; done

Copy the pixmaps

sudo cp -R share/ /usr/lib/vmware-server-console/share

Remote console works for consoling into freebsd guests, but for some reason it doesn't display console for my solaris guest. Though, I can take screenshots and those look fine. Weird.

I've had to buildworld on a test machine atleast 4 times today. After the 2nd time, I got fed up and installed ccache to make the build go faster. I'm on the 4th buildworld now, and it is running much faster now.

ccache in FreeBSD, when installed from ports, comes with examples on how to get ccache to work with src and ports building.Copy and paste, and you're ready to go. This tool rocks.

dev# ccache -s
cache directory                     /root/.ccache
cache hit                           7868
cache miss                         20433

So far so good. Most of the cache misses were from the first buildworld with ccache. This dev box is running in vmware on a very, very cheap Dell workstation, and it's not the fastest thing to run vmware, so any speed boosts will help.

Also, day 5 of the week of unix is coming tomorrow. I'm slowly finishing it.