Search this site

[prev]  Page 5 of 54  [next]





New project: eventmachine-tail

Logstash uses EventMachine, which is an event-driven library for Ruby. Part of logstash's requirements is the ability to watch logfiles like 'tail -f' would. Previously, I was using File::Tail, but this was not EventMachine friendly.

Additionally, it's pretty common for applications to write to files with generated names that include timestamps, etc, so it was clear logstash would need a way to watch a pattern of files, like a glob such as /var/log/*.log

Thus was born eventmachine-tail.

You can install it with:

gem install eventmachine-tail
And try the 'rtail' tool that comes with it:
rtail -x "*.gz" "/var/log/**/*"
The project is hosted on github: jordansissel/eventmachine-tail.

This is the first project I've tried to use git with. I'm not really happy with git as it only seems to complicate my workflow, but I'll try to stick with it.

gzip and other file progress checking

While I wait for a mysql backup to replay against a test server, I was wondering how far along the replay was.

I am building the database using this command:

% gzip -dc admin.sql.20100511.060001.gz| mysql -uroot -proot

Pipes have a finite buffer. That is, the mysql command is busy reading stuff from stdin while gzip is busy writing to stdout. If gzip outputs faster than mysql can consume, gzip will end up filling the pipe's buffer and its next write will block, pausing it momentarily until mysql can catch up.

If we can inspect what position gzip is currently at, we can use that data along with the input file size to give us a progress indicator.

In linux 2.6.22 and beyond, there is /proc/<pid>/fdinfo which will tell you the current seek position of any open file in a process, so let's use that. First we'll need to find what file descriptor number is the input file, then ask what position it is at.

% pgrep -f 'gzip -dc admin'
% PID=6261
% cd /proc/$PID/
% ls -l fd
total 0
lrwx------ 1 jsissel jsissel 64 May 11 15:04 0 -> /dev/pts/12
l-wx------ 1 jsissel jsissel 64 May 11 15:04 1 -> pipe:[31912237]
lrwx------ 1 jsissel jsissel 64 May 11 15:04 2 -> /dev/pts/12
lr-x------ 1 jsissel jsissel 64 May 11 15:04 3 -> /home/jsissel/admin.sql.20100511.060001.gz 
% cat fdinfo/3
pos:    149028864
flags:  0104000

# Get the file size
% size=$(stat -L fd/3 -c "%s")

# Get the position
% pos=$(awk '/pos:/ { print $2 }' fdinfo/3)

% echo "$pos/$size" | bc -l
The above output says we are 25% complete.

You could go a step further and include the process uptime to show an estimate of time remaining.

cd /proc/$PID
size=$(stat -L fd/3 -c "%s")
pos=$(awk '/pos:/ { print $2 }' fdinfo/3)
start=$(date -d "$(ps -p $PID -o lstart=)" +%s)
now=$(date +%s)
echo "Minutes elapsed:"
echo "($now - $start) / 60" | bc -l
echo "Minutes remaining (estimated):"
echo "((($now - $start) / ($pos / $size)) - ($now - $start)) / 60" | bc -l

# Output
Minutes elapsed:
Minutes remaining (estimated):
Update: Some commenters pointed out pv as a solution here. It looks pretty good.

Headless wrapper for ephemeral X servers

For various projects I'm doing right now, I need an easy way to automatically run code in an X server that may not necessarily be the active display. This code may even run on servers in production that don't have video cards or monitors attached.

For some history on this, check out this post on xvfb and firefox. You can solve the problem in that post by simply launching firefox with the tool below, and your X server will exit when firefox exits.

To solve my need for automated headless Xserver shenanigans, I wrote a script that will wrap execution with a temporary X server of your choosing. For xdotool tests I generally use Xephyr and Xvfb.

What's so special about this script? It picks an unused display number reliably and runs your command with that set as DISPLAY. This is a major win because you don't have to pre-allocate X servers (headless or otherwise) before you start your programs. will pick the first unused display, just as binding a socket to port 0 will pick an unused port. Finally, when your process exits, the xserver and windowmanagers are killed. Example usage:

% ./ firefox
Trying :0

Fatal server error:
Server is already active for display 0
        If this server is no longer running, remove /tmp/.X0-lock
        and start again.

Trying :1
Xvfb looks healthy. Moving on.
Using display: :1
Running: firefox
It does health checks to make sure the X server is actually up and running before continuing.

You can also specify a window manager to run:

% ./ -w openbox-session firefox
Trying :1
Xvfb looks healthy. Moving on.
Using display: :1
Starting window manager: openbox-session
Waiting for window manager 'openbox-session' to be healthy.
... startup messages from openbox ...
openbox-session looks healthy. Moving on.
Running: firefox
The default X server is Xvfb. You can use Xvnc, Xephyr, or any X server. Here we run Xephyr with some options:
% sh -x "Xephyr -ac -screen 1280x720x24" -w openbox-session firefox
Xephyr looks healthy. Moving on.
Using display: :1
Starting window manager: openbox-session
Waiting for window manager 'openbox-session' to be healthy.
openbox-session looks healthy. Moving on.
Running: firefox
Screenshot of what this looks like is here: xephyr-ephemeral-x-example.png

I use this script to run my xdotool tests. Additionally, parallelizing test execution can often lead to faster tests. Wrapping each test run with ensure that each test suite runs in a clean environment, untainted by any previous test, allowing me to run all test suites in parallel. Prior to writing this script, I would run each test suite in serial on the same X server instance.

I use a similar technique at work to start ephemeral X servers for running WebDriver tests in hadoop. Each mapper starts its own X server, safely, and kills it when it is completed. This is implemented in java, instead of shell, since the mappers all launch from java and mapreduce launch differently than a standard command would in your shell.

Code lives here:
xdotool tests: xdotool/t

Project Updates

I spent this weekend working on implementing automatic pattern discovery in the C version of grok. I updated the C and Ruby API to be able to use this discovery:
% irb -rrubygems -rgrok
>> grok =
>> grok.add_patterns_from_file("/usr/local/share/grok/patterns/base")
>> pattern ="There's no place like")
=> "\\QThere's no place like \\E%{IP}\\Q\\E"
>> grok.compile(pattern)
>> grok.match("There's no place like").captures
=> {"IP"=>[""]}
>> grok.match("There's no place like 234.345.12.4")
=> false
This will most likely get used in logstash to provide a helping hand in building log patterns. It could also be used on events that have no pre-defined pattern so we can at least attempt to parse the log line.

On a more important note, this was one of the few remaining features the perl grok implementation had that the C version did not.

You can download the latest version of grok here. You can also install the ruby grok gem with gem install jls-grok.

On another project node, I pushed a new fex release that includes some small fixes and also an rpm spec.

Flashback 2004: Too much about perl and regexp

Continuing the flashback series. Here's 2004.

Still in school. Hacking on countless projects in perl.

While writing a command-line interpreter, I wanted you to be able to type 'st' or 'sta' instead of 'status' and have the code find a best match for your command. This is often called 'ambiguous commands' or somesuch. I came up with a clever way to do this using regular expressions (May 2004).

I also posted about using regular expression to match quoted and nonquoted strings (June 2004).

Finally, I experiment with using regular expressions to find word boundaries near the cursor for my command line interpreter (Oct 2004).

Are you drowning in perl and regular expressions, yet? ;)

Making iptables changes atomically and not dropping packets.

I'm working on rolling out iptables rules to all of our servers at work. It's not a totally simple task, as many things can go wrong.

The first problem is the one where you can shoot yourself in the foot. Install a new set of rules for testing on a remote server, and suddenly your ssh session stops responding. I covered how to work around that in a previous post.

Another problem is ensuring you make your firewall changes atomically. All rules pushed in a single step. In linux, if you have a script with many lines of 'iptables' invocations, running it will make one rule change per iptables command. And what if you write your rules like this?

# Flush rules so we can install our new ones.
iptables -F

# First rule, drop input by default
iptables -P INPUT DROP

# Other rules here...
iptables -A INPUT ... -j ACCEPT
iptables -A INPUT ... -j ACCEPT
If your server is highly trafficked, then the delay between the 'DROP' default and accept rules can mean dropped traffic. That sucks. This is an example of a race condition. Additionally, there's a second race condition earlier in the script where, depending on the default rule for INPUT, we may drop or accept all traffic for a very short period. Bad.

One other problem I thought could occur was a state tracking problem with conntrack. If previously we weren't using conntrack, what would happen to existing connections when I set default deny and only allowed connections that were established? Something like this:

iptables -P INPUT DROP
iptables -A INPUT -i eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 22 --syn -j ACCEPT
I did some testing with this, and I may be wrong here, but it does not drop my existing sessions as I had predicted. This is a good thing. Turns out, when this runs, the 'conntrack' table is populated with existing connections from the network stack. This further helps us not drop traffic when pushing new firewall rules. You can view the current conntrack table in the file /proc/net/ip_conntrack.

What options do we have for atomically applying a bunch of rules so we don't drop traffic? The iptables tool set comes with 'iptables-save' which lets you save your existing iptables rules to a file. I was unable to find any documentation on the exact format of this file, but it seems easy enough to read. The output includes rules and counters for each table and chain. Counters are optional.

All the documentation I've read indicates that using 'iptables-restore' will apply all of the rules atomically. This lets us set a pile of rules all at once without any race conditions.

So I generate an iptables-restore file and use iptables-restore to install it. No traffic dropped. I'm generating it with a shell script, so there was one gotcha - I basically take iptables commands and output them to a file. I do this with a shell function I wrote, called 'addrule'. However, I have some rules like this:

addrule -A INPUT -p tcp -m limit --limit 5/min -j LOG --log-prefix "Denied TCP: " --log-level debug
I quoted the argument in the addrule invocation, but we need to also produce a quoted version in our iptables-restore rule file, otherwise --log-prefix will get set to 'Denied' and we'll also fail because 'TCP:' is not an option iptables expects. It appears to be safe to quote all arguments in the iptables-restore files except for lines declaring chain counters (like ':INPUT ACCEPT [12345:987235]'), defining tables (like '*filter'), or the 'COMMIT' command. Instead of quoting everything, I just quote everything with spaces in an argument.

The fix makes my 'addrule' function look like this:


addrule() {
  while [ $# -gt 0 ] ; do
    # If the current arg has a space in it, output "arg"
    if echo "$1" | grep -q ' '  ; then
      echo -n "\"$1\""
      echo -n "$1"
    [ $# -gt 1 ] && echo -n " "
  done >> $rulefile
  echo >> $rulefile

# So this:
#   addrule -A INPUT -j LOG --log-prefix "Hello World"
# will output this to the $rulefile
#   -A INPUT -j LOG --log-prefix "Hello World"
So now the quoted arguments stay quoted. All of that madness is in the name of being able to simple replace 'iptables' with 'addrule' and you're good to go. No extra formatting changes necessary.

One last thing I did was to make sure iptables-restore didn't reject my file, and if it did, to tell me:

if iptables-restore -t $rulefile ; then
  echo "iptables restore test successful, applying rules..."
  iptables-restore -v $rulefile
  rm $rulefile
  echo "iptables test failed. Rule file:" >&2
  echo "---" >&2
  cat $rulefile >&2
  rm $rulefile
  exit 1
Throw this script into puppet and we've got automated firewall rule management that won't accidentally drop traffic on rule changes.

New keynav release available (0.20100302)

It's time for a new keynav release. The main push for this release was to fix problems with building against the new libxdo (xdotool) versions.

Additionally, there is one new feature: "record"

Recording lets you record a series of actions in keynav and replay them later much like you would in vim. Only keynav actions are recorded.

  - Started using glib for dynamic arrays (GPtrArray)
  - Uses new versioning scheme
  - Added 'version' (or -v or --version) to output the keynav version
  - Now requires (via xdotool)
  - Add ability to record keynav actions with the 'record' command. Optional argument
    is a filename to save the recordings to for persistence across keynav runs.
    Example in keynavrc to bind 'q' to record to ~/.keynav_macros:
      q record ~/.keynav_macros
    Works similar to vim recording.
      1) Hit record once
      2) Type the key you want to record to
      3) Do things in keynav you want recorded.
      4) Any 'end' operation or hitting record again will terminate and save
         this recording.
    Recordings only persist if you specify a file to save to, otherwise they are lost
    across keynav restarts.

  - Try repeatedly to grab the keyboard with XGrabKeyboard on 'start' commands.
    This loop is a necessary workaround for programs like xbindkeys that could
    launch keynav but at the time of launch still hold a keyboard grab
    themselves. Reported by Colin Shea.

  - Nonfunctional bug fixes and other code cleanup from patches by Russell Harmon

xdotool 1.20100302 release

Thanks to some early testing and feedback from the previous xdotool release, I've put together a new release that fixes a few minor problems.

Download: xdotool-1.20100302.2708.tar.gz

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

  - Some test fixes suggested by Daniel Kahn Gillmor
  - Don't use isatty(0) anymore as it isn't totally reliable in determining if
    we should try to read from stdin.

  - Minor libxdo patch to avoid exporting unnecessary symbols (Daniel Kahn Gillmor)
  - Minor Makefile fix to only link libxdo (and not xdotool) against libX11 and
    libXtst (also Daniel Kahn Gillmor)
  - Some test cleanup patches by Daniel Kahn Gillmor
  - Fixed the 'type hello' test that would fail sometimes when run in Xephyr.
  - Fix a minor bug in xdo_window_setclass that incorrectly aborts when
    XGetClassHint fails.

Flashback 2003: vim + screen trick

Matt Simmons suggested that this week be a flashback week. That's easy enough to do, so I'm doing one per year starting in 2003.

A common feature request of vim is to be able to split window and have a shell be there. Well, as it turns out, that's easy to do if you run vim inside screen (gvim not applicable here).

The original post details a simple vim keybinding that will split the screen and attach a shell and close the split when the shell exits.

August 2003: Neat vim/screen trick

However, I've got a better version now. You'll want this script, "":


screen -X split
screen -X focus down
screen -X screen sh -c "cd $PWD; $SHELL; screen -X remove"
Then in your .vimrc:
:map \s :silent !<CR>
The new version mostly puts the screen invocations into a shell script. Another improvement is to change to $PWD so the new shell will be in the same working directory as vim started in.

New xdotool (1.20100227.2679) released!

It's time for another xdotool release.

Lots of big fixes and changes in the latest release of xdotool. Many of the improvements in this release incorporate feedback I've received on this list and in bugs filed. Please enjoy :)


Full changelist since previously-announced release:

  - Bump major version since there were lots of libxdo changes (now
  - Support '--pid NUM' for searching for windows by pid:
    Example: xdotool search --pid 12345
    Example: xdotool search --pid 12345 "hello world"
  - Add 'xdotool getwindowpid' command to get the pid that owns a window.
  - Add --window to xdotool mousemove, click, mousedown, and mouseup
  - Add --screen to xdotool mousemove
  - The --clearmodifiers flag now clears mouse buttons, too.
    Part of
  - The --clearmodifiers flag now clears caps lock.
  - Have 'make install' create all required directories
  - Support multiple screens in searches. Default is to search all screens.
    Example: 'xdotool search --screen N' specifies a single screen to search.
  - Support reading a list of xdotool commands via stdin or a file. Usages:
    * read from stdin:
      xdotool -
    * read from stdin (use isatty(0) to detect redirection):
      xdotool < somefile
    * An executable script also works when it has this at the top:

  - Slight change in how we present versions. Now versions have major, date,
    and svn revision. The main addition was to add the major which maps to the
    libxdo library abi version (
  - Add 'xdotool version' for querying version
  - libxdo now has xdo_version function for fetching the library version.
  - Allow typing keys that are not mapped to any keyboard button.
    For example, you can type the euro symbol like this, even if you don't have
    a key on your keyboard that can do it:
      xdotool key EuroSign
  - Fixed some pod errors in the manpage