Continuing the flashback series. Here's 2004.

Still in school. Hacking on countless projects in perl.

While writing a command-line interpreter, I wanted you to be able to type 'st' or 'sta' instead of 'status' and have the code find a best match for your command. This is often called 'ambiguous commands' or somesuch. I came up with a clever way to do this using regular expressions (May 2004).

I also posted about using regular expression to match quoted and nonquoted strings (June 2004).

Finally, I experiment with using regular expressions to find word boundaries near the cursor for my command line interpreter (Oct 2004).

Are you drowning in perl and regular expressions, yet? ;)

I'm working on rolling out iptables rules to all of our servers at work. It's not a totally simple task, as many things can go wrong.

The first problem is the one where you can shoot yourself in the foot. Install a new set of rules for testing on a remote server, and suddenly your ssh session stops responding. I covered how to work around that in a previous post.

Another problem is ensuring you make your firewall changes atomically. All rules pushed in a single step. In linux, if you have a script with many lines of 'iptables' invocations, running it will make one rule change per iptables command. And what if you write your rules like this?

# Flush rules so we can install our new ones.
iptables -F

# First rule, drop input by default
iptables -P INPUT DROP

# Other rules here...
iptables -A INPUT ... -j ACCEPT
iptables -A INPUT ... -j ACCEPT

If your server is highly trafficked, then the delay between the 'DROP' default and accept rules can mean dropped traffic. That sucks. This is an example of a race condition. Additionally, there's a second race condition earlier in the script where, depending on the default rule for INPUT, we may drop or accept all traffic for a very short period. Bad.

One other problem I thought could occur was a state tracking problem with conntrack. If previously we weren't using conntrack, what would happen to existing connections when I set default deny and only allowed connections that were established? Something like this:

iptables -P INPUT DROP
iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 --syn -j ACCEPT

I did some testing with this, and I may be wrong here, but it does not drop my existing sessions as I had predicted. This is a good thing. Turns out, when this runs, the 'conntrack' table is populated with existing connections from the network stack. This further helps us not drop traffic when pushing new firewall rules. You can view the current conntrack table in the file /proc/net/ip_conntrack.

What options do we have for atomically applying a bunch of rules so we don't drop traffic? The iptables tool set comes with 'iptables-save' which lets you save your existing iptables rules to a file. I was unable to find any documentation on the exact format of this file, but it seems easy enough to read. The output includes rules and counters for each table and chain. Counters are optional.

All the documentation I've read indicates that using 'iptables-restore' will apply all of the rules atomically. This lets us set a pile of rules all at once without any race conditions.

So I generate an iptables-restore file and use iptables-restore to install it. No traffic dropped. I'm generating it with a shell script, so there was one gotcha - I basically take iptables commands and output them to a file. I do this with a shell function I wrote, called 'addrule'. However, I have some rules like this:

addrule -A INPUT -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level debug

I quoted the argument in the addrule invocation, but we need to also produce a quoted version in our iptables-restore rule file, otherwise --log-prefix will get set to 'Denied' and we'll also fail because 'TCP:' is not an option iptables expects. It appears to be safe to quote all arguments in the iptables-restore files except for lines declaring chain counters (like ':INPUT ACCEPT [12345:987235]'), defining tables (like '*filter'), or the 'COMMIT' command. Instead of quoting everything, I just quote everything with spaces in an argument.

The fix makes my 'addrule' function look like this:

rulefile="$(mktemp)"

addrule() {
  while [ $# -gt 0 ] ; do
    # If the current arg has a space in it, output "arg"
    if echo "$1" | grep -q ' '  ; then
      echo -n "\"$1\""
    else
      echo -n "$1"
    fi
    [ $# -gt 1 ] && echo -n " "
    shift
  done >> $rulefile
  echo >> $rulefile
}

# So this:
#   addrule -A INPUT -j LOG --log-prefix "Hello World"
# will output this to the $rulefile
#   -A INPUT -j LOG --log-prefix "Hello World"

So now the quoted arguments stay quoted. All of that madness is in the name of being able to simple replace 'iptables' with 'addrule' and you're good to go. No extra formatting changes necessary.

One last thing I did was to make sure iptables-restore didn't reject my file, and if it did, to tell me:

if iptables-restore -t $rulefile ; then
  echo "iptables restore test successful, applying rules..."
  iptables-restore -v $rulefile
  rm $rulefile
else
  echo "iptables test failed. Rule file:" >&2
  echo "---" >&2
  cat $rulefile >&2
  rm $rulefile
  exit 1
fi

Throw this script into puppet and we've got automated firewall rule management that won't accidentally drop traffic on rule changes.

It's time for a new keynav release. The main push for this release was to fix problems with building against the new libxdo (xdotool) versions.

Additionally, there is one new feature: "record"

Recording lets you record a series of actions in keynav and replay them later much like you would in vim. Only keynav actions are recorded.

0.20100302.*:
  - Started using glib for dynamic arrays (GPtrArray)
  - Uses new versioning scheme major.date.svnrev
  - Added 'version' (or -v or --version) to output the keynav version
  - Now requires libxdo.so.1 (via xdotool)
  - Add ability to record keynav actions with the 'record' command. Optional argument
    is a filename to save the recordings to for persistence across keynav runs.
    Example in keynavrc to bind 'q' to record to ~/.keynav_macros:
      q record ~/.keynav_macros
    Works similar to vim recording.
      1) Hit record once
      2) Type the key you want to record to
      3) Do things in keynav you want recorded.
      4) Any 'end' operation or hitting record again will terminate and save
         this recording.
    Recordings only persist if you specify a file to save to, otherwise they are lost
    across keynav restarts.

20091231.04:
  - Try repeatedly to grab the keyboard with XGrabKeyboard on 'start' commands.
    This loop is a necessary workaround for programs like xbindkeys that could
    launch keynav but at the time of launch still hold a keyboard grab
    themselves. Reported by Colin Shea.

20091231.02:
  - Nonfunctional bug fixes and other code cleanup from patches by Russell Harmon

Thanks to some early testing and feedback from the previous xdotool release, I've put together a new release that fixes a few minor problems.

Download: xdotool-1.20100302.2708.tar.gz

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

1.20100302.*:
  - Some test fixes suggested by Daniel Kahn Gillmor
  - Don't use isatty(0) anymore as it isn't totally reliable in determining if
    we should try to read from stdin.

1.20100301.*:
  - Minor libxdo patch to avoid exporting unnecessary symbols (Daniel Kahn Gillmor)
  - Minor Makefile fix to only link libxdo (and not xdotool) against libX11 and
    libXtst (also Daniel Kahn Gillmor)
  - Some test cleanup patches by Daniel Kahn Gillmor
  - Fixed the 'type hello' test that would fail sometimes when run in Xephyr.
  - Fix a minor bug in xdo_window_setclass that incorrectly aborts when
    XGetClassHint fails.

Matt Simmons suggested that this week be a flashback week. That's easy enough to do, so I'm doing one per year starting in 2003.

A common feature request of vim is to be able to split window and have a shell be there. Well, as it turns out, that's easy to do if you run vim inside screen (gvim not applicable here).

The original post details a simple vim keybinding that will split the screen and attach a shell and close the split when the shell exits.

August 2003: Neat vim/screen trick

However, I've got a better version now. You'll want this script, "screener.sh":

#!/bin/sh

screen -X split
screen -X focus down
screen -X screen sh -c "cd $PWD; $SHELL; screen -X remove"

Then in your .vimrc:

:map \s :silent !screener.sh<CR>

The new version mostly puts the screen invocations into a shell script. Another improvement is to change to $PWD so the new shell will be in the same working directory as vim started in.

It's time for another xdotool release.

Lots of big fixes and changes in the latest release of xdotool. Many of the improvements in this release incorporate feedback I've received on this list and in bugs filed. Please enjoy :)

Download: http://semicomplete.googlecode.com/files/xdotool-1.20100227.2679.tar.gz

Full changelist since previously-announced release:

1.20100227.*:
  - Bump major version since there were lots of libxdo changes (now libxdo.so.1)
  - Support '--pid NUM' for searching for windows by pid:
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=10
    Example: xdotool search --pid 12345
    Example: xdotool search --pid 12345 "hello world"
  - Add 'xdotool getwindowpid' command to get the pid that owns a window.
  - Add --window to xdotool mousemove, click, mousedown, and mouseup
  - Add --screen to xdotool mousemove
  - The --clearmodifiers flag now clears mouse buttons, too.
    Part of http://code.google.com/p/semicomplete/issues/detail?id=6
  - The --clearmodifiers flag now clears caps lock.
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=4
  - Have 'make install' create all required directories
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=17
  - Support multiple screens in searches. Default is to search all screens.
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=11
    Example: 'xdotool search --screen N' specifies a single screen to search.
  - Support reading a list of xdotool commands via stdin or a file. Usages:
    * read from stdin:
      xdotool -
    * read from stdin (use isatty(0) to detect redirection):
      xdotool < somefile
    * An executable script also works when it has this at the top:
      #!/path/to/xdotool

0.20100118.*:
  - Slight change in how we present versions. Now versions have major, date,
    and svn revision. The main addition was to add the major which maps to the
    libxdo library abi version (libxdo.so.0).
  - Add 'xdotool version' for querying version
  - libxdo now has xdo_version function for fetching the library version.
  - Allow typing keys that are not mapped to any keyboard button.
    For example, you can type the euro symbol like this, even if you don't have
    a key on your keyboard that can do it:
      xdotool key EuroSign
    http://code.google.com/p/semicomplete/issues/detail?id=16
  - Fixed some pod errors in the manpage

I'm working on a new monitoring system because I can't find one that solves enough of my problems. It's going to be free and have an unrestricted open source license.

I could use your help.

At this stage, the best way you can help is to make sure I get lots of data about various infrastructure architectures, monitoring needs, reporting needs, alerting needs.

If you can, please share with me the following:

• A description (or diagram) of your infrastructure including network, servers, services, storage, etc.
• What you are using now for monitoring (can be any number of tools)
  • Why you like them
  • Why you don't like them
  • What you'd rather have, if anything
• What tools are missing that you wish existed?
• Would more documentation on monitoring, in general, help?
• Do you carry a pager? If not, why not? If so, what does it support? (email, sms, html email, mobile web, normal web)
• Would more documentation help?
  • Better documentation on how to monitor the things you need to monitor, such as best practices, better tool docs, etc?
  • Best practices for monitoring various scenarios?

Any thoughts, please email me jls-calmon@semicomplete.com - I'll be collecting this data into my design document, which you can view in unfinished form, here: Current Design Doc

Ever configured a firewall remotely? Ever blocked yourself and had to get physical hands to fix it?

Kind of sucks.

So you're going to start playing with some new firewall rules, but you learned from the past and now you have a cron(8) or at(8) job that will reset the firewall rules to permissive every so often, just in case you lock yourself out.

I used to do that. Until I realized today that I'm frankly too lazy to wait the N minutes I'll have to wait for my at(8) job to kick in.

Now I sniff packets and have a script trigger from that.

On the remote server, I'll use ngrep to watch for a specific payload in an icmp echo packet. This works because bpf(4) gets packets before the firewall has a chance to filter them, meaning even if you deny all packets, bpf(4) (libpcap, tcpdump, ngrep, etc) will still see those packets. Here's the script I use on the remote server:

# Look for any icmp echo packets containing the string 'reset-iptables'
ngrep -l -Wnone -d any 'reset-iptables' 'icmp and icmp[icmptype] = icmp-echo' \
| grep --line-buffered '^I ' \
| while read line ; do 
  iptables -F
  iptables -P INPUT ACCEPT
  iptables -P OUTPUT ACCEPT
  iptables -P FORWARD ACCEPT
done

The ngrep line will output this whenever it sees a matching packet:

remotehost% ngrep -l -Wnone -d any 'reset-iptables' 'icmp and icmp[icmptype] = icmp-echo'
interface: any
filter: (ip) and ( icmp and icmp[icmptype] = icmp-echo )
match: reset-iptables
##
I XX.XX.XX.XX -> XX.XX.XX.XX 8:0
....reset-iptablesipta

We'll grep for just the 'I' line, then trigger a full firewall reset.

I couldn't figure out how to use ping(8) and set a specific payload, so I'll use scapy.

workstation% echo 'sr1(IP(dst="remotehost.example.com")/ICMP(type="echo-request")/"reset-iptables")' | sudo scapy

Now, if I accidentally lock myself out through firewall rule changes, I can trivially reset them using that 'echo | scapy' onliner.

Obviously, I don't keep the reset script running after the firewall rules are tested and known-good, but it's a great instant-gratification means to solving the locked-out problem you may face when testing new firewall rules.

I started the decade still in high school, as a junior, with after-school activities including track, marching band, and computer team. Already building my late-night hacking habits, I spent the late hours doing things like scripting in mircscript. Computer team was QBASIC or C++, but most of us used QBASIC, including me, despite having easily passed AP Computer Science the year prior (which taught C++).

I remember trying C++ outside of the AP class, but the C++ they taught us used this horrible bastard misimplementation what looked like STL - instead of "vector" you'd use "apvector" etc... which I couldn't find anywhere outside of class and thus ended my C++ adventures. Here's a tip for teachers and professors: Always use practical material when doing introductory material. Don't create some crap sandbox that isn't useful knowledge outside of the classroom.

I started my computer science degree at RIT in 2001 and joined the Computer Science House (CSH). CSH turned out to be the best part of college by a large measure. From many perspectives, CSH is the best part about RIT: social, learning, common ground, etc. Classes were sometimes interesting, but were often boring and unchallenging. This fact explains why I had a 2.8 GPA when graduating in 2006.

I socialed my way (by knowing professors) into some senior-level IT classes without any prerequisites. I remember them being fun, but mostly I remember the fleet if whiney IT students complaining of the work which I viewed as necessary and also exciting. I took two or three senior IT classes, and the whining was consistent throughout. Kind of depressing, because I was still trying to find an environment outside of CSH where folks actually cared about what they were doing. This student culture seemed to cause RIT to dumb-down it's IT program frequently while I was enrolled. I don't know if it's improved.

School also helped push me to like meritocracy. One summer co-op, I worked for a professor, who's PhD thesis we were implementing in code. When we had questions about implementation problems, the professor gave us the "I don't know" response. I'm implementing your idea, and you don't know? And you got a doctorate from this? I lost respect for master and PhD status after that; I treat them now like any certification and mostly ignore them.

Respect and status is earned, not given. Apparently PhDs and master awards are sometimes given, not earned.

From 2002 to today, there are more than 70 projects in my svn repo. Linux, Solaris, FreeBSD, and Windows projects are here. C, C++, Perl, Python, Ruby, JavaScript, XML, and more. It is totally overwhelming to consider all of the things I've worked on over the past few years, so I'll just pick a few.

I started learning PHP in early 2002, then Perl in late 2002, Python in 2004, Ruby in 2007. The first version of this site was written in PHP, then redone in Perl and HTML::Mason, then I migrated to pyblosxom.

In late 2002, I found out about RSS; a friend said I should write an rss aggregator, so I did. I used this project as an excuse to learn perl. I wrote the first version of the aggregator was written using perl CGI and DBI, even had a few folks besides me using it. I let the project die after I didn't really gain from having an rss reader.

Another project was an awesome jukebox software called Pimp. Pimp started as local mp3 player in perl, then became a telnet-controlled jukebox system written. Later versions were in python and sported a decent web interface, multiple simultaneous streams, and some othe cool features. For streaming, I had to reverse engineer the shoutcast protocol to add the in-stream metadata (showing you what song is playing).

In 2003, I wrote an aim client in perl using Net::OSCAR. I used it for a year or two, and don't remember why I stopped. That project taught me a lot about terminal interfaces. I also came up with some clever regular expression tricks for doing line editing.

I've learned a great deal of things outside of code, too. I've read Rands in Repose and Eric Raymond's Hacker Howto. Communication mediums like BarCamp, IRC, Twitter, and others are also important here. These have helped me find communities, large and small, like similar to what I had at CSH.

Then there's my career path. Ultimately, I'm a hacker at heart, so I'm looking for challenging problems to solve. Through college and my early jobs, I've learned more about what I need to help me focus on solving challenging problems. I want an environment that is supportive and productive. I want active communication, especially about blocking issues or direction changes.

I've also discovered that I enjoy learning from mistakes. It is difficult to own up to failures of any size and scope, but I find it is more educational, personally, to admit failures and move on to work on learning from those failures. I've had the C-level folks at my current job (Rocket Fuel) explain this exact thing during meetings - about a problem that was quickly acknowledged, responded to, repaired, and post-mortemed. That's responsibility and passion.

I joined Google immediately after graduating from RIT (seriously, I graduated on friday, and started work on the monday). I knew nothing of negotiations, only that I knew I wanted to work there. As a result, I got a crap salary with crap stock. Now I know. Always negotiate. I left Google for OnLive and it's technical challenge. As a side note, my career at Google was going no where; after almost two years, I hadn't really moved, despite some effort, in areas of greater responsibility or in technical challenge.

I left OnLive because the company was going nowhere. Terrible leadership, poor communication, nepotism, and irresponsibility permeated the employee stack up to (and especially) the C-level folks. If successful startups need great people to succeed, then I can only conclude that OnLive will fail. Leaving was a pity, for me, because the exciting technical challenge was still there. My favorite response from management when I talked to them about company-wide problems was "Wait 3 months" - waiting is hoping, and hope is not a strategy. Despite that, I did stay longer, and that was a mistake. Their other senior operations sysadmin guy left shortly after me for the same reasons. Now, I'm a stock holder and also have friends there, so I still hope OnLive does great things, but again, hope is not a strategy.

Both Google and OnLive gave me great perspective and input on what I should look for in future employment.

More career-wise, I continue to have high expectations of my coworkers and especially of leadership. I value responsibility and passion more than I do technical prowess, because responsibility and passion are more rare qualities and are at least as valuable as technical prowess. I expect good communication. I like to ask 'why' questions, and I expect that an answer of "I don't know" should immediately followed by "but I will find out". Living in the dark is counterproductive. I've learned to be patient and learned to translate. Translation is critical. Using a common syntax and terminology is critical when communicating with others. Patience is critical.

Email is the worst, so to assure you that I am in a cheery mood and willing to assist, I will drown your inbox with smiley-face emoticons - nothing else seems to work. It is extra difficult because of the BOFH persona folks generally attribute to support groups like corp IT and production staff. BOFH is hilarious, but it's an antipattern for treating your users, just like calling them 'lusers'. People have been trained to think they are at fault when some crappy piece of software they use misbehaves, and they have been trained to feel like they are massively inconveniencing you when they ask for support that you were hired to give. It's upsetting - I'm here to help.

I've also found the uncomfortable realization that my jobs are often not just hacking on problems. Politics (and the falsehoods required) are an unavoidable part of the job process.

On the life side, I got married to the greatest girl in the world and got two dogs. Wendy and I had been together for almost six years before getting married. Our first dog died suddenly of an autoimmune disease at age five, and our current doggy is quite goofy and going happy at age two.

While at RIT, I learned to rollerblade, skateboard, and play ice hockey. I still skateboard and would like to get back into hockey.

I also traveled a lot. I lived in Georgia, then New York, then California. I've been to Dublin, Seattle, Washington DC, the Caribbean, Vegas, and New Orleans. I started going with friends to Defcon and got involved with Hack or Halo at Shmoocon. I also went to BarCamps in NYC, Rochester, and the Bay Area.

Dublin, Ireland and Seattle were for Google work trips. Dublin was awesome. My trip coincided with Mashup Camp, where I met up with some Yahoo folks on a after-event bar hop. Dublin's Temple Bar district is good fun and reminded me a bit of Bourbon Street in New Orleans with all its shenanigans. Seattle was more business and less social, only stopping in at Burger Master for some good burgers while in town.

BarCamps were awesome everywhere. The first one I went to was NYC in 2006, where a fellow CSHer demoed the first version of jQuery; I remember staying up until almost 4AM at CollegeHumor's offices (the barcamp location, I think?) hacking with it.

There were also Yahoo! Hack Days; I went to both of the ones held here in the Bay Area. The first hack day had me hacking up del.icio.us and also writing keynav (a project still maintained and used!). My del.icio.us hack resulted in a Wall Street Journal "Marketing" section front-page column, which was one of the most awesome things ever (article also viewable here). I'd never been in any newspaper before.

Two years later, Yahoo! held another open hack day where I wrote SnackUpon, which got coverage on lifehacker and a few other sites. These events weren't my first 12-hour hackathons. At RIT, CSH ran yearly team coding competitions sponsored by Red Bull and Bawls; my team won two years.

These hackathons repeatedly highlight that my productivity spikes after midnight. I've been fortunate to have jobs that understand this and don't require me to be at the office every day at 9AM.

Wrapping up a decade that had lots of travel, learning, hacking, and networking (with people) on and offline, it doesn't seem likely the trend of travel, learning, hacking, and networking is showing any signs of downturn.

A new release of xdotool is available. This release has only minor changes based on changes needed to help Debian and other folks package this. Speaking of which: thanks to Daniel Kahn Gillmor and Wen-Yen Chuang for helping maintain the Debian packages for xdotool and keynav and for working with me on these changes.

Hop on over to the xdotool project page and download the new version.

The changelist follows:

20091231.02:
  No functional changes.
  - fix linking problems and just use $(CC) for build and linking
  - Make the tests headless (requires Xvfb and GNOME)
  - Make the t/run.sh test runner exit-code friendly

20091231.01:
  No xdotool changes.
  libxdo changes:
    * Rename keysymcharmap -> keysym_charmap
    * Expose keysym_charmap and symbol_map as xdo_keysym_charmap()
      and xdo_symbol_map()

xdotool-20091231.02