Search this site

Page 1 of 2  [next]





logstash's first major release - 1.0.0

Ready for log and event management that doesn't suck or drain your budget? It's time to logstash.

After lots of refactoring and improvements to logstash since the first minor release last November, logstash is ready for wider usage now.

Read my announcement here.

The logstash site is also online and has docs, intros, slides, and videos.

Happy logstashing!

Introducing FPM - Effing Package Management

Having become fed up with dealing with rpmbuild, spec files, debian control files, dh_make, debuild, and the whole lot, I automated my way back to sanity.

The result is a tool I call "fpm" which aims to help you make and mangle packages however you choose, all (ideally) without having to care about the internals of your particular native package format.

The goal of this project is not to undermine upstream packaging but to grant everyone the ability to trivially build and edit packages. Why? Not all software is packaged. Not all software of the version you want is packaged. And further, not all users are willing or able to take the time to learn all the ins and outs of their package build tools.

For example, you can package up your /etc/init.d directory as an RPM by doing simply this:

% fpm -s dir -t rpm -n myinitfiles -v 1.0 /etc/init.d
Created /home/jls/rpm/myinitfiles-1.0.x86_64.rpm
fpm will create a simple package for you and put it in your current directory. The result:
% rpm -qp myinitfiles-1.0.x86_64.rpm -l

% rpm -qp myinitfiles-1.0.x86_64.rpm --provides
myinitfiles = 1.0-1
% rpm -qp myinitfiles-1.0.x86_64.rpm --requires
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
You can package up any directory. But there's more.

Above, I didn't specify a package summary, so how about fixing the rpm to include the description? You can use RPMs as the source (-s flag) in fpm. There's also a helpful '-e' (--edit) flag that'll let you edit the rpm spec (or debian control) file before building.

% rpm -qp myinitfiles-1.0.x86_64.rpm --info | grep Summary
Summary     : no summary given

% fpm -s rpm -t rpm -e myinitfiles-1.0.x86_64.rpm
... this opens up $EDITOR so you can edit the spec file it generated ...
... make some changes to the spec, including adding a proper 'Summary' ...
Created /home/jls/rpm/myinitfiles-1.0-1.x86_64.rpm

% rpm -qp myinitfiles-1.0-1.x86_64.rpm --info | grep Summary
Summary     : my /etc/init.d directory
The '-s dir' flag says the source of the package is a directory. There's also support for other package sources like rubygems, other rpms, debs, and more on the way.

With FPM, you can specify dependencies, architecture, maintainer, etc. All from a simple command line, and never forcing you to learn the pain and suffering that can come with rpm spec files or debian package building.

You can install fpm with: gem install fpm

The project page is here:

The wiki is here (has more examples):

Introducing: fingerpoken - a mobile device as a touchpad/remote/keyboard

I'm giving a presentation this week at the Puppet Bay Area user meetup and while working on slides, I wanted to be able to present while not being attached to my laptop.

Enter: fingerpoken

Fingerpoken lets you turn your iphone/ipad/itouch into a touchpad, keyboard, and remote for another computer. The only required piece on your iphone is Safari. No appstore stuff to download!

Under the hood, it uses websockets and touch events and sends JSON-encoded requests to your workstation and will move the mouse, type, scroll, and more.

Project page: fingerpoken on github.

A short demonstration of this project in action:

Introducing: Ruby Minstrel - a method instrumenter

The following tools are awesome: strace, truss, ltrace, dtrace, and systemtap.

Sometimes I'm trying to debug a ruby library or application, and I end up monkeypatching things just to see what arguments are being passed as a away of sanity checking configuration or correctness. Other times I want to profile the time spent only in a certain class, or method, etc. At a basic level, I'm looking for a simple ltrace equivalent in ruby.

Enter minstrel. There may be projects out there already that do this, but I don't know of one, so it got written tonight.

You can 'gem install minstrel' to get it (or here (rubygems) and here (github))

My standard path of debugging (without other options) is to sanity check my code and then dive into the code for whatever app/library I am using. It often requires root access to modify ruby libs on the system, which sucks for one-off debugging. Writing up monkeypatches guessing at methods that I should inspect is error prone and sucky. Monkeypatching for debugging is common for me and is about as efficient/productive as using LD_PRELOAD to hack in my own method calls. (see liboverride). It sucks.

Minstrel is something better than a bag of the usual hope+monkeypatch+time combination. For example. Let's use minstrel to debug why mcollective's 'mc-ping' is failing:

snack(~) % /usr/sbin/mc-ping
connect failed: Connection refused - connect(2) will retry in 5
Ok, connection refused. To what? Yes, I could use strace or tcpdump to debug this particular issue. But that's not the point, here. After looking at the mcollective code for a few minutes I came up with a few classes I want to instrument.
snack(~) % % RUBY_INSTRUMENT="MCollective::Connector::Stomp,Stomp::Connection,TCPSocket" minstrel /usr/sbin/mc-ping
Wrap of TCPSocket successful
Wrap of Stomp::Connection successful
Wrap of MCollective::Connector::Stomp successful
enter MCollective::Connector::Stomp#connect([])
enter Stomp::Connection#socket([])
class_enter TCPSocket#open(["localhost", 6163])
class_exit_exception TCPSocket#open(["localhost", 6163])
connect failed: Connection refused - connect(2) will retry in 5

Partials in HAML with Sinatra

HAML documentation mentions "partials" twice, in passing only. Doesn't actually say how to use them, best as I can tell. Lots of the documentation seems to be heavily Rails-focused (mentions of link_to and other methods). Frankly, if I didn't already have lots of Rails experience, I think some of the haml docs would be pretty confusing.

In Sinatra, you invoke haml by simply calling 'haml' with your template name.

  get "/" do
    haml :index
On that note, Sinatra (or HAML, I haven't checked) will let you use layouts, too, something I found hardly documented. Layouts are a way to basically wrap your entire template in another template called 'layout.haml' by default - useful if you want every page to have the same structure. You can also define different layouts.

What I wanted was a simple layout with a header in a separate template. Here's what I did:


    %title Hello!
      =haml :header, :layout => false
Thie 'yield' will invoke the original template called as 'haml :index', helping keep each page using the same layout. Additionally, I render the header by calling a separate haml template; calling haml with ':layout => false' tells it to ignore any layout files. This lets us invoke another haml template.

Also, since the 'haml' method only takes symbols, if you want to put your haml templates in different directories, you'll have to use this awkward syntax:

  haml :"some/path/to/template"
    #  ^ Note the ':' before the string. You can also probably the
    #  String#to_sym method.

Ruby metaprogramming will cost you documentation.

Ruby, like many other dynamic and modern languages, makes it easy for you to do fun stuff like metaprogramming.

Ruby, also like other nice languages, comes with a builtin documentation generator that scans your code for comments and makes them available in html and other formats.

... until you start metaprogramming.

Take a simple example, the Fizzler! The name of the class is unimportant; this class will simply provide a new way to define methods, simply for the sake of showing some metaprogramming and how ruby's rdoc fails on it.

class Fizzler
  def self.fizzle(method, &block)
    self.class_eval do
      define_method method, &block

class Bar < Fizzler
  # Print a worldly message
  fizzle :hello do
    puts "hello world!"
  # A simple test
  def test
    puts "testing, 1 2 3!"

# Now some sample code, let's invoke the new 'hello' method we generated with
# 'fizzle'.
bar =
The output looks like this:
% ruby fizzler.rb 
hello world!
All is well! We are generating new methods on the fly, etc etc, all features of metaprogramming. However, we can never make this 'hello' method obviously available to the world via rdoc, at least as far as I can tell. The rdoc generated looks like this:

Note the lack of any mention of 'hello' as a method. I cannot simply do what works for lots of other normal ruby code and ask for the documentation of hello by running 'ri Bar#hello' - because rdoc simply doesn't see it.

I recall in python, if you were dynamically generating methods and classes, you could also inject their documentation by simply setting the '__doc__' property on your class or method. Ruby doesn't appear to have such a thing.

Additionally, in some metaprogramming cases, the stack traces are actually harder to read. For example, ActiveRecord makes extensive use of 'method_missing' rather than dynamically generate methods. The output is the same, but the stacktraces are now littered with 'method_missing' and references to files and lines you don't own, rather than containing stacktraces to named functions and other useful pointers. This perhaps is a feature, but for cases like method_missing, being able to add other useful data onto the stack trace would greatly aid in debugging.

So, if long term necessities like documentation and easy debuggability (stack traces, etc), are hindered by metaprogramming, at least in ruby, what are we left to do? Metaprogramming is clearly a win in some places, but the automatic losses seem to detract from any value it may have.

Bringing test tools to Nagios monitoring

With all the TDD (test-driven design) and BDD (behavior-driven design) going around these days, it'd be a shame not to use these tools on monitoring applications.

You might have a boatload of tests that test your application before you roll a new version, but do you use those tests while the application is in production? Can you? Yes!

Let's take an important example of monitoring some complex interaction, like searching google and checking the results. Simple with a mouse, but perhaps complex in code. Even if you wrote a script to do it, using an existing testing framework gets you pass/fail testing automatically.

For this example, I'll use the following ruby tools: rspec and webrat. This fairly easy, though it took me a bit to find all the right documentation bits to clue me in to the right way.

require 'rubygems'
require 'webrat'

Spec::Runner.configure do |config|
  include Webrat::Methods

describe "google search for my name" do
  it "should include in results" do
    visit ""
    webrat.response.title.should =~ /Google/
    query = "jordan sissel"
    fill_in "q", :with => query
    webrat.response.title.should == "#{query} - Google Search"
    click_link ""
Now, we run this with the 'spec' tool:
% spec rspec-webrat.rb 

Finished in 0.578546 seconds

1 example, 0 failures
Seems ok. Let's break the test and see what happens. Change the 'visit' line to something else:
    visit ""
Now rerun the test, which was checking specifically for google things in the page and will now fail on yahoo's page:
 % spec rspec-webrat.rb

'google search for my name should include in results' FAILED
expected: /Google/,
     got: "Yahoo!" (using =~)

Finished in 0.186847 seconds

1 example, 1 failure
This output kind of sucks. Additionally, rspec failures seem to have exit code 1, not 2 as wanted by a nagios check reporting critical. Let's fix those. First, fixing the exit code can be hacked around directly in ruby if you want:
# Nagios checks expect exit code '2' to mean CRITICAL.
# Let's make any nonzero exit attempt always exit 2 (EXIT_CRITICAL).
module Kernel
  alias :original_exit :exit
  def exit(value)
    value = EXIT_CRITICAL if value != 0
Fixing the output just means telling spec to use a different output format. I like the 'nested' output. Rerun that test now:
% spec -f nested rspec-webrat.rb
google search for my name
  should include in results (FAILED - 1)

'google search for my name should include in results' FAILED
expected: /Google/,
     got: "Yahoo!" (using =~)

Finished in 0.017534 seconds

1 example, 1 failure

% echo $?
All set.

Even better is that you can include multiple checks in the same script, if you wanted to. RSpec lets you select any test to run alone, so your nagios checks for a given web application could be a very simple:

define command {
  command_name check_google_for_semicomplete
  command_line /usr/bin/spec -f nested -e "google search for my name" mytests.rb

Ruby: Finding subclasses in your world

Use the ObjectSpace class to find all ancestors of a given class.
class Foo; end
class Bar < Foo; end
class Baz < Foo; end

subclasses = ObjectSpace.each_object(Class).select do |klass|
  klass.ancestors.include?(Foo) and klass != Foo

# prints "[Baz, Bar]"
puts subclasses
Of course, you could always override Class#inherited instead, but if you don't want to override methods, the above is a reasonable choice.

Ruby's DateTime::strptime vs libc strptime

A project I'm working on has some odd slowness about it. Using ruby-prof, I found that String#scan was consuming most of the time, but ruby-prof didn't tell me where it was coming from. A quick hack that replaced String#scan with my own method showed who was calling it, DateTime.strptime -
class String
  def scan(*args)
I tried using the ruby debugger to break on String#scan, but it didn't seem to work. PEBCAK, probably, which is why I used the solution above to just toss an exception when that function was called.

Back at the point, DateTime.strptime is slow. Looking at the underlying code shows you why: date/format.rb - the _strptime_i method.

Lots of string shuffling, regular expressions to match field specifiers (%d, etc), string modification with more regexps, etc. The code is pretty easy to read, but it's still doing a lot of work it doesn't need to be doing. Luckily, libc comes with a method for parsing times in the same way: strptime.

So, I started working on an extension to the Time class that invokes libc's strptime and returns a Time instance: ruby-ctime. The usage is simple once you have the module:

require "CTime"

puts Time.strptime("%Y", "2009")
# outputs 'Wed Jan 00 00:00:00 +0000 2009'
The one major holdback from strptime is that there's no wide support for timezones. Format strings like %Z and %z work with strftime, but generally are unsupported by strptime; exceptions that do support %z are glibc, and freebsd appears to support both %Z and %z. Nothing reliably cross-platform. This is a historical problem due to the fact that the 'struct tm' structure has no timezone field (glibc and the bsds add 'long tm_gmtoff' to support timezones).

This means we'll have to correct for this by extending strptime to support it, but I'm not there yet.

Anyway, short benchmarking for features supported by both libc strptime and DateTime strptime shows libc a massive winner:

snack(~/projects/ruby-ctime) % ruby test.rb
Iterations: 10000
datetime: 7.680928 (1301.92601727291/sec)
my_strptime: 0.126583 (78999.5497025667/sec)
A 60x speedup using the new C code vs DateTime.strptime. This is a great start, but we still need timezone support. I need to hack timezone support into this, which probably means I'll start with glibc's strptime implementation.

Ruby Net::IMAP and Exchange

Exchange's server-side filters are pretty weak, so I decided to work around them by writing a tool that will fix my inbox and filter mail appropriately so that any client I use to view mail with (OWA, whatever) has the same view with no client-local filters. It's likely/possible there's already a tool that does this; let's ignore that possibility for now.

Ruby comes with Net::IMAP, but it doesn't come with an authenticator that supports 'PLAIN' auth, so we have to provide one:

# Learned the 'PLAIN' expected format from imapsync.
class PlainAuthenticator
  def process(data)
    # Net::IMAP takes care of base64 encoding the result of this...
    return "#{@user}\0#{@user}\0#{@password}"
  def initialize(user, password)
    @user = user
    @password = password

Net::IMAP::add_authenticator('PLAIN', PlainAuthenticator)
Now that we have that, let's try connecting.
imap ="", "imaps", usessl=true)
imap.authenticate("PLAIN", user, passwd)
This fails, because Exchange's IMAP server ignores the RFC:
/usr/lib/ruby/1.8/net/imap.rb:3122:in `parse_error': unexpected token CRLF (expected SPACE) (Net::IMAP::ResponseParseError)
        from /usr/lib/ruby/1.8/net/imap.rb:2974:in `match'
        from /usr/lib/ruby/1.8/net/imap.rb:1959:in `continue_req'
        from /usr/lib/ruby/1.8/net/imap.rb:1946:in `response'
Expected a space, not a crlf. The failure is in continue_req, which expects what the RFC says:
continue_req    ::= "+" SPACE (resp_text / base64)
However, Exchange's IMAP server doesn't send a space after the plus. Great, let's fix that by overriding the continue_req method:
# Copied/modified from net/imap.rb, don't modify that file, put this
# in your own code to override the continue_req method
module Net
  class IMAP
    class ResponseParser
      def continue_req
        #match(T_SPACE)   # Comment this line out to not expect a space.
        return, @str)
Once you've done that, everything else seems to work normally. I have only tested listing mail folders thus far, but the hacks above allow you to get this far.