Search this site

Page 1 of 2  [next]





new grok version available (1.20101030)

Another grok release is available. Major changes include:
  • Pattern discovery as described here.
  • Doxygen (C) and RDoc (Ruby) docs now available.
  • Much improved ruby support (gem install jls-grok).

Hop on over to the grok project page and download the new version.

Changes since last announced release:

  - Add 'make package-debian' to produce a .deb build of grok.

  - API docs via doxygen
  - rdoc for the Ruby module
  - Add Grok::Pile to Ruby

  - Fix tests
  - Add a ruby example for pattern discovery
  - Add grok-web example (runs grok in ruby via sinatra to show pattern discovery)
  - Add more time formats (US, EU, ISO8601)
  - Fix bug that prevented multiple patterns with the same complexity from being
    used in discovery.

  - Add pattern discovery through grok_discover (C) and Grok#discover (Ruby)
    Idea for this feature documented here:
  - The ruby gem is now called 'jls-grok' since someone already had the 'grok'
    gem name on gemcutter.
  - Fix some pattern errors found in the test suite.
  - New version numbering to match my other tools.

Project Updates

I spent this weekend working on implementing automatic pattern discovery in the C version of grok. I updated the C and Ruby API to be able to use this discovery:
% irb -rrubygems -rgrok
>> grok =
>> grok.add_patterns_from_file("/usr/local/share/grok/patterns/base")
>> pattern ="There's no place like")
=> "\\QThere's no place like \\E%{IP}\\Q\\E"
>> grok.compile(pattern)
>> grok.match("There's no place like").captures
=> {"IP"=>[""]}
>> grok.match("There's no place like 234.345.12.4")
=> false
This will most likely get used in logstash to provide a helping hand in building log patterns. It could also be used on events that have no pre-defined pattern so we can at least attempt to parse the log line.

On a more important note, this was one of the few remaining features the perl grok implementation had that the C version did not.

You can download the latest version of grok here. You can also install the ruby grok gem with gem install jls-grok.

On another project node, I pushed a new fex release that includes some small fixes and also an rpm spec.

New xdotool (1.20100227.2679) released!

It's time for another xdotool release.

Lots of big fixes and changes in the latest release of xdotool. Many of the improvements in this release incorporate feedback I've received on this list and in bugs filed. Please enjoy :)


Full changelist since previously-announced release:

  - Bump major version since there were lots of libxdo changes (now
  - Support '--pid NUM' for searching for windows by pid:
    Example: xdotool search --pid 12345
    Example: xdotool search --pid 12345 "hello world"
  - Add 'xdotool getwindowpid' command to get the pid that owns a window.
  - Add --window to xdotool mousemove, click, mousedown, and mouseup
  - Add --screen to xdotool mousemove
  - The --clearmodifiers flag now clears mouse buttons, too.
    Part of
  - The --clearmodifiers flag now clears caps lock.
  - Have 'make install' create all required directories
  - Support multiple screens in searches. Default is to search all screens.
    Example: 'xdotool search --screen N' specifies a single screen to search.
  - Support reading a list of xdotool commands via stdin or a file. Usages:
    * read from stdin:
      xdotool -
    * read from stdin (use isatty(0) to detect redirection):
      xdotool < somefile
    * An executable script also works when it has this at the top:

  - Slight change in how we present versions. Now versions have major, date,
    and svn revision. The main addition was to add the major which maps to the
    libxdo library abi version (
  - Add 'xdotool version' for querying version
  - libxdo now has xdo_version function for fetching the library version.
  - Allow typing keys that are not mapped to any keyboard button.
    For example, you can type the euro symbol like this, even if you don't have
    a key on your keyboard that can do it:
      xdotool key EuroSign
  - Fixed some pod errors in the manpage

new grok version available (20091227.01)

The latest release is another important step in grok's life. Most major changes were outside of the code:
  • FreeBSD users can install grok via ports: sysutils/grok. Thanks to sahil and wxs for making this happen.
  • The project has online documentation and also ships with a manpage.

Hop on over to the grok project page and download the new version.

Changes since last announced release:

 - Add function to get the list of loaded patterns.
 - Ruby: new method Grok#patterns returns a Hash of known patterns.
 - Added flags to grok: -d and --daemon to daemonize on startup (after config
   parsing). Also added '-f configfile' for specifying the config file.
 - Added manpage (grok.1, generated from grok.pod)

 - match {} blocks can now have multiple 'pattern:' instances
 - Include samples/ directory of grok configs in release package.

grok 20091103 release

Lots of changes since the last announced release. Grok should get some more activity now that I'm actually using it in a few places. If you find bugs or have feature requests, please file them on googlecode issue tracker (see below)

The largest changes are:

  • we ship with Ruby and C API.
  • lots of new testing code.
  • we now use tokyocabinet internally instead of bdb.
Grok documentation:
File bugs/features:

This release has all tests passing in these configurations:

  • FreeBSD 7.1. tokyocabinet 1.4.30, pcre 8.00, libevent 1.4.12
  • Ubuntu 9.04. tokyocabinet 1.4.35, pcre 7.8-2, libevent 1.3e-3
  • CentOS 5.3. tokyocabinet 1.4.9-1, pcre 7.8-2, libevent 1.1a-3.2.1
Thanks to Pete Fritchman, grok also ships with an RPM spec so you can 'rpmbuild -tb grok-20091103.tar.gz' for simple build and deployment. The spec builds grok, grok-devel, and grok-ruby.

I'm using this version of grok myself with good success. It's also being used in the new logstash (log indexing tool) project for doing log parsing.

Full changelist since the last announced release:

 - New: ruby bindings are now really supported.
 - Change 'WORD' pattern to be word bounded (\b)
 - Move grok-patterns to patterns/base
 - update rpm spec to install patterns/base in /usr/share/grok

 - Add a bunch of tests, mostly in ruby, to exercise grok. This uncovered a
   few bugs which are fixed.
   All tests currently pass (both CUnit and Ruby Test::Unit) on:
   * FreeBSD 7.1. tokyocabinet 1.4.30, pcre 8.00, libevent 1.4.12
   * Ubuntu 9.04. tokyocabinet 1.4.35, pcre 7.8-2, libevent 1.3e-3
   * CentOS 5.3. tokyocabinet 1.4.9-1, pcre 7.8-2, libevent 1.1a-3.2.1
 - When making strings in ruby, we now make them tainted per ruby C docs.
 - "Too many replacements" error will now occur if you have cyclic patterns,
   such as defining 'FOO' to be '%{FOO}'. Max replacements is 500.

 - Make 'grok' main take a config for an argument.
 - Add grok rpm spec.
 - Updated Makefile to work on Linux and FreeBSD without modification.
 - Fixed bug introduced in 20091022 where capture_by_(name,subname) didn't
   work properly.
 - Add default values for match {} grok.conf blocks:
   shell: stdout
   reaction: "%{@LINE}"
 - Have grok exit nonzero if there were no reactions executed, akin to grep(1)
   not matching anything. 'reactions' are important here; matches with no
   reaction will not count as a reaction.

 - Fix libgrok accidentally sharing it's parser/lexer functions. Turns out,
   libgrok doesn't actually need to parse the grok.conf, so we don't build
   against it anymore for the library.

 - Convert to using tokyocabinet instead of berkeley db.
   * Berkeley DB isn't easy to target across platforms (4.x versions vary
     wildly in bugs)
   * tokyo cabinet should be faster
   * tokyo cabinet is less code to write, and slightly more readable in the
     author's opinion.
   * we don't have to serialize with xdr anymore

 - include pregenerated bison/flex output since gnu flex varies much from
 non-gnu flex, and many important platforms don't have gnu flex available
 easily from packages (freebsd, centos, etc) but come with the other flex.

 No functional changes.

Systems Administration Advent Calendar

I like the Perl Avent Calendar and have followed it for a few years now (even as my Perl usage has declined).

Some quick googling didn't find any hits for a similar thing for systems administration, so I'm starting one.

I need your help; interested in contributing ideas or content? Email me [email protected].

keynav with xinerama support

This is the same post I made to the keynav-users mailing list

I just finished working on the xinerama portion of multi-screen support for keynav.

If someone is interested, I could use some help testing. It's working for me, and there are a few odd behaviors that I'm not sure are the best. Let me know if you test it.

No new official release yet, but if you want to test, svn can be fetched with:
svn checkout

- Include support for multiple screens.
  * When 'start' happens, the region will be the size of the current display
    (wherever the mouse is)
  * Moving the region outside of the current display will move it to the next
    display (right or left). This currently only works with Xinerama.
  * History works as expected (move to the right display, history-back goes to
    the previous display, etc)
  * When multiple Screen (non-xinerama) are found, XGrabKey on all root windows.
  * Screens are sorted, if possible, from left-to-right based on x-coordinate
    origin. This unfortuntely means, for now, only left-to-right monitor
    configurations are known to be supported.

new keynav version available (20080522)

Hop on over to the keynav project page and download the new version.

The changelist from the previous announced release is as follows:

  - Sync xdotool library to 20080521.
  - Added 2 grid examples to keynavrc
  - Applied patches from Richard Kolkovich
    + Fix backwards math when calculating Nth cell when using 'cell-select N'
    + Fix dislexia when doing 'cell-select NxM'
    + Abort update() calls when app is inactive.
  - Now warns you if you try to execute an invalid command.
(Yes, 0522 is tomorrow. Turns out uploads to googlecode are write-once, so I'll need to come up with an additional versioning scheme that lets me push multiple releases in a single day in case I find something bad after I upload a release.)

new xdotool version available (20080521)

Hop on over to the xdotool project page and download the new version. This is the first version being served totally from googlecode, let me know if you have problems.

The changelist from the previous announced release is as follows:

  * Import patches from keynav's xdotool branch (From Lukas Mai) which make
    xdotool compile happily when building with -Wall and -pedantic (and others)
  * Finally convert to using Window type for (hopefully) everything instead of
    incorrectly typecasting Window to int sometimes.
  * Confirmed that the one (but mostly useless) test passes.

liboverride project page is up.

I finally got around to putting up a project page for liboverride.

Location: /projects/liboverride