I setup ccache again (trivial) to help me with building freebsd repeatedly. I noticed that much of the time spent in the kernel build process was in building dependency lists using awk.

Why couldn't we apply the ccache idea to everything else? If the same input always means the same output, then we could cache it if it is computationally expensive to compute that output.

Script: dcache.sh

Above is a hack that runs like ccache, but tracks all files created by the process (and its subprocesses). Here's a sample run, of counting the number of lines in a file with awk and outputing the result (within awk) to another file.

% /usr/bin/time ./dcache.sh awk '{x++} END { print "Total records: " x > "/tmp/hello"}' bigdata
Running original...
        1.60 real         0.05 user         0.74 sys
% cat /tmp/hello
Total records: 1000000

# Remove the old output file
% rm /tmp/hello

# Rerun it again, unmodified, and it will use the cached output.
% /usr/bin/time ./dcache.sh awk '{x++} END { print "Total records: " x > "/tmp/hello"}' bigdata
Using cache...
        0.06 real         0.00 user         0.06 sys
% cat /tmp/hello
Total records: 1000000

It doesn't work with everything just yet, but the problems seem to be with truss's behavior and not the script's fault, like sometimes truss hangs, or it doesn't follow a fork like it should. Beyond truss problems, the scripts doesn't track file renames. It also doesn't understand how to figure out what the input files for each command is. Ideally it would checksum any inputs and use that as the cache key; currently it only checksums the commandline arguments and not the external files being used (such as 'bigfile').

I started initially without using truss, but awk doesn't call open(2) via libc when it opens files, for some reason, and I can't figure out a clean way to capture specific function calls from a process (even a child process).

Dtrace would be sexy here, but it is unavailable in the main freebsd trunk.

The speedup is pretty obvious for cpu-intensive things, but the real test is to see how it performs when working properly and hooked into the freebsd kernel build.

I wrote a script that will go through every variable named MASTER_SITE_[something] in bsd.sites.mk and basically sort each variable's contents by ping time.

The implementation uses asyncore in python, and uses tcp connections instead of pinging (pinging is not guaranteed to work). The output is sorted by fastest response time and in a format suitable for Makefile (and thus /etc/make.conf).

% python fastest_site.py
 => Checking servers for MASTER_SITE_GENTOO (77 servers)
MASTER_SITE_GENTOO=\
        ftp://ftp.ecc.u-tokyo.ac.jp/GENTOO/%SUBDIR%/ \
        ftp://gentoo.kems.net/pub/mirrors/gentoo/%SUBDIR%/ \
        ftp://files.gentoo.gr/%SUBDIR%/ \
... <output cut> ...

I recommend that you send the output to a separate file, such as /usr/local/etc/ports_sites.conf, and add the following line to /etc/make.conf:

.include "/usr/local/etc/ports_sites.conf"

Now generate the file:

% python fastest_site.py > /usr/local/etc/ports_sites.conf
 => Checking servers for MASTER_SITE_GENTOO (77 servers)
 => Checking servers for MASTER_SITE_TCLTK (11 servers)
 => Checking servers for MASTER_SITE_GET_E (11 servers)
 => Checking servers for MASTER_SITE_BERLIOS (4 servers)
...

Download: fastest_sites.py

I've had a src commit bit in freebsd for a while, and I haven't done much with it. Yes, I suck. I'm working on getting my mouse code into the tree, finally, after almost 2 years of its life and almost 2 years of my slacking off on getting it ready for submission.

I think one of the main reasons I've directed energy elsewhere is because there's a (from my perception) thick metawork process to get real work done. Culture shock, mostly. Almost all of the tools and methods are different from my own. My experience at Google has given me good practice in dealing with systems foreign to me, so why do I hesitate to work on FreeBSD stuff?

Outside of the processes involved in getting code into the FreeBSD source tree, one of the main problems I've had working on specifically kernel changes in FreeBSD are that I haven't come up with a good solution for separating workspaces other than simply creating a new virtual machine for each logical workspace. In Perforce, you can create multiple clients and work on independent changes in each client. In userland code, you can simply just build a new binary in a different directory, and you can test both binaries independently.

With kernels, I have a hard time multitasking. Not specifically multitasking different kernels, but if I'm making kernel and userland changes which are unrelated to eachother, I can't safely test a new kernel on the same system as a userland change. Isolating these as easy as making a new virtual machine, but copying virtual machines is not as fast and easy as, say, making a new perforce client.

I haven't come up with a good solution yet, but I'm sure someone else has and perhaps I'll build on that. Maybe some kind of hack where I would use a pristine, read only system image and all changes would be written to a memory filesystem on top of that pristine image? But this basically means all systems have to have the same pristine image (copying the image is nontrivial in time)...

Hopefully some of this makes sense. I'm open to suggestions :)

I finally picked up some small parts (leds, breadboard, leads, multimeter) from Fry's to start work on my universal remote project. Yes, I know you can buy universal remotes. I want to learn more electronics, so why not use this as a starter project?

Since my Soekris box is now free of it's router duties, I can use it for this project. To do this, I'll need two pieces: an infrared receiver, and an infrared emitter. I bought an infrared emitter led today, and I just need to buy a receiver online (they're like $2).

Before getting there, I needed to learn how to drive the GPIO ports on the net4501.

It was pretty simple to do, nd after hooking up a few wires I had an LED that blinked after a few hours of reading and hacking. The code itself was trivial to write, I just had to learn how to talk to the GPIO ports.

After writing the blinking code, I decided the next step was clearly to add fading to the LED. This is commonly done with PWM (pulse-width modulation). Apple made this technique famous with its "breathing" LED lights on the monitors and laptops when the devices were in sleep mode.

At any rate, I have successfully written code that makes both the error led and the PIO5 (GPIO 0) pin "breathe".

Download glow.c

I've definitely spent 10+ hours in the past 2 days trying to get this system upgraded from 6.0 to 7.0...

Why? First, I was using the wrong power supply, so it would randomly reboot during startup (4 hours, after before I figured that one out, after compiling zillions of kernels, even trying GENERIC from 6.2, 6.3, and 7.0). Then, I couldn't get a stable system image that would boot successfully. It would halt trying to run /sbin/init. Not sure why.

I was initially making my own image with mdconfig, and rsyncing the entire system into a file-backed fs. Then I dd'd this image to my compactflash and put it in the net4501. I gave up on that after several hours, and did this instead:

1. Bring up a new vmware instance with 2 disks. 1 800mb disk and 1 5gb disk.
2. Install 7.0-RC1, only install kernel and base.
3. Mount the 5gb disk (newfs /dev/sd1, mount /dev/sd1 /usr/src) and unpack the kernel sources there. Build kernel (For the soekris box).
4. Make any necessary config changes (serial console, etc)
5. Shutdown machine
6. Use qemu-img to convert the vmware disk into a raw disk image
7. dd the new image to compact flash.
8. Rejoice. I have a working net4501 now.

In testing in qemu, I get random timeouts talking to ad0. On the soekris board, I get random timeouts to ad0. wtf :(

Boot into safe-mode, and everything seems find. (Safe mode turns off DMA, iirc)

Using slight variations on the techniques mentioned in my previous post, I've got a vmware instance running Fedora 8 that permits any and all logins. These login sessions are logged with script(1).

Fedora 8 comes with selinux enabled by default. This means sshd was being denied permission to execute my special logging shell. The logs in /var/log/audit/ explained why, and audit2allow even tried to help make a new policy entry for me. However, I couldn't figure out (read: be bothered to search for more than 10 minutes) how to install this new policy. In searching, I found out about chcon(1). A simple command fixed my problems:

chcon --reference=/bin/sh /bin/sugarshell

The symptoms prior to this fix were that I could authenticate, but upon login I would get a '/bin/sugarshell: Permission Denied' that wasn't logged by sshd.

There are plenty of honeypot software tools out there, but I really wasn't in the mood for reading piles of probably-out-of-date documentation about how to use them. This hack (getpwnam + pam_permit + logging shell) took only a few minutes.

As a bonus, I found a feature in Fedora's yum tool that I like about freebsd's packaging system: It's trivial to ask "Where did this file come from?" Doing so made me finally look into how to do it in Ubuntu.

FreeBSD: pkg_info -W /usr/local/bin/ssh

/usr/local/bin/ssh was installed by package openssh-portable-4.7.p1,1

Fedora: yum whatprovides /usr/bin/ssh

openssh-server.x86_64 : The OpenSSH server daemon

Ubuntu: dpkg -S /usr/bin/ssh

openssh-client: /usr/bin/ssh

Let's see what I catch.

Put the vmware-remotemks' program where vmware console wants it

Symlink vmware-remotemks to /lib/vmware-server-console/bin/vmware-remotemks

Mount linprocfs to /proc

mount -t linprocfs - /proc

Hack fix for vmware dep library

From vmware-server-console-distrib/lib/bin/:
for i in ../lib/lib*/*; do ln -s $i `basename $i`; done

Copy the pixmaps

sudo cp -R share/ /usr/lib/vmware-server-console/share

Remote console works for consoling into freebsd guests, but for some reason it doesn't display console for my solaris guest. Though, I can take screenshots and those look fine. Weird.

I've had to buildworld on a test machine atleast 4 times today. After the 2nd time, I got fed up and installed ccache to make the build go faster. I'm on the 4th buildworld now, and it is running much faster now.

ccache in FreeBSD, when installed from ports, comes with examples on how to get ccache to work with src and ports building.Copy and paste, and you're ready to go. This tool rocks.

dev# ccache -s
cache directory                     /root/.ccache
cache hit                           7868
cache miss                         20433

So far so good. Most of the cache misses were from the first buildworld with ccache. This dev box is running in vmware on a very, very cheap Dell workstation, and it's not the fastest thing to run vmware, so any speed boosts will help.

Also, day 5 of the week of unix is coming tomorrow. I'm slowly finishing it.

In rc.conf, I can put:

ipv4_addrs_bge0="192.168.0.1-254/24 192.168.5.50-100/24"

And do /etc/rc.d/netif restart bge0

and we get:

% ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        inet6 fe80::20a:e4ff:fe3f:92ee%bge0 prefixlen 64 scopeid 0x1 
        inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2
        inet 192.168.0.3 netmask 0xffffffff broadcast 192.168.0.3
...
        inet 192.168.0.253 netmask 0xffffffff broadcast 192.168.0.253
        inet 192.168.0.254 netmask 0xffffffff broadcast 192.168.0.254
        inet 192.168.5.50 netmask 0xffffff00 broadcast 192.168.5.255
        inet 192.168.5.51 netmask 0xffffffff broadcast 192.168.5.51
...
        inet 192.168.5.99 netmask 0xffffffff broadcast 192.168.5.99
        inet 192.168.5.100 netmask 0xffffffff broadcast 192.168.5.100
        ether 00:0a:e4:3f:92:ee
        media: Ethernet autoselect (none)
        status: no carrier

Neat. That's one way to take an entire subnet.

I wrote a script a while ago to build a very tiny freebsd world. It's extremely fast and only builds a freebsd image in approximately 10 megs of space. It lets you quickly create new jail enviroments or system images for small embedded platforms.

If you look at the script itself, you'll get an idea of what it installs. I used a variant of this script to build the system I run on my Soekris net4501 which runs FreeBSD and is under 20 megs.

There are lots of "make a small freebsd system" scripts, but most of the ones I've found rely heavily on 'buildworld' and what not. This takes a live system and copies the binaries you need, then uses ldd(1) to track down required libraries.

view minibsd.sh

Example usage:

kenya(~/t) % rm -rf ./soekris/
kenya(~/t) % time sudo ./minibsd.sh
sudo ./minibsd.sh   0.16s user 0.65s system 61% cpu 1.326 total
kenya(~/t) % sudo chroot ./soekris /bin/sh
# pwd
/
# exit

Simple jail config (rc.conf):

jail_enable="YES"
jail_list="test"
jail_test_rootdir="/home/jls/t/soekris"
jail_test_hostname="test"
jail_test_ip="10.1.1.1"
jail_test_interface="tl1"

Put something simple in this jail's rc.conf (/home/jls/t/soekris/etc/rc.conf):

sshd_enable="YES"
sendmail_enable="NONE"

Let's test the jail now:

kenya(~/t) % sudo /etc/rc.d/jail start
Configuring jails:.
Starting jails: 

At this point, it's probably hung (assuming you enabled sshd). If you hit CTRL+T you'll see what command has the foreground and what it's doing.* This is because it's prompting you (output is directed to JAILROOT/var/log/console.log) for entropy for the ssh-keygen. Smash a few keys then hit enter. It'll finish eventually.

kenya(~/t) % sockstat -4 | grep 10.1.1.1:22 
root     sshd       2258  3  tcp4   10.1.1.1:22           *:*

Our sshd is running happily inside that jail we made. This whole process took about 5 minutes.

* FreeBSD's CTRL+T terminal handler feature has to be the best thing ever invented. I wish Linux had something like this. Here's what hitting CTRL+T when running cat looks like:

kenya(~) % cat
load: 0.45  cmd: cat 2324 [ttyin] 0.00u 0.00s 0% 600k
load: 0.42  cmd: cat 2324 [ttyin] 0.00u 0.00s 0% 600k

It clearly shows you the command name, the pid, and the syscall-type-thing it's doing. Clearly cat is waiting for input from the tty. <3 FreeBSD.