This year's been pretty good, but the last two months were pretty lame.

In the last six weeks, I found out Caramel has lymphoma, got unemployed, and had emergency surgery to remove my appendix on Christmas Day. The unemployment caused me to lose an in-progress mortgage refinance.

I'll pick up the mortgage thing once I remedy the employment problem, but I'm staying quite happily unemployed until after my kid is born - should be any day now!

Most of my career-growing moves were outside of work: at meetups, in open source efforts, or in networking with folks on IRC or twitter. Lots of awesome folks out there, so go introduce yourself. Don't be a dick. :)

I didn't write much on this site, but mainly, that was due to an increase in my activities on IRC and twitter. Most of what I published this year was code and was less writing about said code. I'd like to fix that, though.

This years successes were topped by two new major projects, fpm and logstash. I also released some major improvements to xdotool and other tools.

The current implementation of logstash isn't very old, but prototypes, hacks, and other incarnations of pretty much the same thing date back to at least 2005 and probably earlier. This project has been a long-time-coming, and Pete Fritchman and I have been talking about logstash for years, so it's nice to finally have some code shipped and a community building around it.

FPM had a crazy positive response. I wrote it as a hack, and it's used all over the place now. Bonus that people are contributing patches and other improvements as well.

Sysadvent was another excellent success, the end of which marked the 4th year and 100th article posted to the project. It is awesome seeing such community involvement from so many different authors.

This year also cemented my move to git from svn. Why? Github, mostly, and not really the features of git itself. Sharing code and patches is so much easier on github than it is with other services.

I went to CarolinaCon and OSCON to talk about logstash. I also went to DevOps Days Mountain View and gave a lightning talk on logstash.

My OSCON talk was overflowing with people standing at the back of the room, etc; it went awesomely. I've also been able to do lunchtime logstash presentations at places like Square and others. I also gave talks at BayLISA meetings. It was a good year for getting out of the house and talking about code.

I tried to get a count of how much code I'd written this year, but I had lots of web-based projects that included third-party stuff like jquery, and I'm too lazy to pick through the results and trim that stuff out. I'm up to about 70 different projects on github now, some useful; some not; all fun!

Looking forward to 2012 :)

I never really liked C's assert() feature. If an assertion is violated, it'll tell you what assertion failed but completely lacks any context:

example: example.c:9: main: Assertion `i == 3' failed.

This is better:

Assertion failed insist.c:7 in main(), insist(i == 3): Something went wrong, wanted i == 3, got 4

The main difference here is that there's context about what failed. A message for humans looking to debug this. This is especially important on Linux these days because every distro I've used recently hates sysadmins and hates debugging - all libraries are stripped of debug symbols and coredumps are disabled by default.

What's the usage look like?

#include 
#include 
#include "insist.h"

int main() {
  int i = 4;
  //assert(i == 3);
  insist(i == 3, "Something went wrong, wanted i == 3, got %d", i);
  return 0;
} 

I also added a special 'return' version of this, 'insist_return' that lets you do error checking and early aborting like this:

insist_return(fd >= 0, START_FAILURE,
              "socket() returned %d, an error: %s", fd, strerror(errno));

Works just like insist() except returns START_FAILURE if 'fd > 0' is false and additionally logs the error formatted above.

Code here: insist.h

It's been about 8 months since the last xdotool release, and I think it's long overdue! This release has a ton of new feature and fixes.

Download: xdotool-2.20110530.1.tar.gz

As usual, if you find problems or have feature requests, please file bugs or send an email to the list.

Changelist since previous announcement:

2.20110530.*
  - New set_window feature: --urgency. This lets you set the urgency flag on a
    window Window managers will interpret this as something about your window
    needing attention. It might flash in the taskbar, pop up, or other.
    Original patch and suggestion by ervandew.
    Issue: http://code.google.com/p/semicomplete/issues/detail?id=39
  - New function: xdo_window_seturgency (see above)
  - Hack in OS X support as it is missing a proper clock_gettime.
    Should fix http://code.google.com/p/semicomplete/issues/detail?id=37
    Reported by ryandesign.com
  - Add support for typing UTF-8 characters. Patch from Joseph Krahn.
  - Make all output call fflush to send data immediately (for pipes). Reported
    by Andreas Wagner on the mailing list.
  - Make 'get_desktop_viewport' output usable with 'set_desktop_viewport'
    http://code.google.com/p/semicomplete/issues/detail?id=47
  - You can now make 'libxdo.a' for embedding libxdo into your binary
    (Requested by psc on the mailing list).
  - Fixed a typing bug where the keymap changes unnecessarily 
  - Should now build cleanly in C++ environments (Reported by psc on the
    mailing list)
  - bugfix: xdotool should use command names first before trying file scripts. 
    See https://bbs.archlinux.org/viewtopic.php?pid=938309 for original report.
  - Add a 'sleep' command. (Requested by Joseph Krahn via mailing list)
  - Add --relative flag to windowmve. (Requested by Anthony Thyssen via mailing
  - Add --desktop flag to the search command. This lets you search for windows
    on specific desktop. Requires a window manager that supports multiple
    desktops in a way that EWMH supports.
    Fixes http://code.google.com/p/semicomplete/issues/detail?id=38
  - Add --limit flag to search. This allows you to break the search early after
    a certain number of matches. (Requested by Anthony Thyssen)
  - New command 'getwindowgeometry' for fetching window position and size
    (Requested by Anthony Thyssen via mailing list)
  - Add --sync flag to search command; blocks until results are found.
    xdotool will search every 0.5 seconds for results.
    http://code.google.com/p/semicomplete/issues/detail?id=42
  - windowmove can now move windows along an axis. Give literal 'x' or 'y'
    instead of a coordinate and it uses the current position. (Requested by
    etnlIcarus via mailing list)
  - Add '--args N' and '--terminator TERMINATOR' to the 'exec' command.
    Default terminator unless specified (or --args is) is ':' (Requested by
    Joseph Krahn and Henning Bekel via mailing list)
  - set_desktop now supports --relative flag (+N or -N to move relative)
    (Requested by Anthony Thyssen)
  - The mouse cursor now changes during 'xdotool selectwindow' (Requested by
    Anthony Thyssen via mailing list)
  - Added '--args N' and '--terminator TERMINATOR' to the 'type' command.
  - Add 'getdisplaygeometry' command for querying the size of your screen.
    (Requested by @rrwo via twitter)
  - Add xdo_get_viewport_dimensions function.

Ready for log and event management that doesn't suck or drain your budget? It's time to logstash.

After lots of refactoring and improvements to logstash since the first minor release last November, logstash is ready for wider usage now.

Read my announcement here.

The logstash site is also online and has docs, intros, slides, and videos.

http://logstash.net

Happy logstashing!

I'm sure this is covered well elsewhere online, but that's never the point of these things ;)

I was helping with some capacity planning and run-rate math today at work and found that ec2 reserved instances are much cheaper compared to on-demand - If this is obvious to you, chill out, I have historically never really used EC2 nor have I ever been close to budgeting. ;)

I proved this conclusion with some math, but frankly I like visualizations better, so I decided to learn R. I wrote an R script that will graph an on-demand vs reserved pricing for one m1.large instance (code at end of the post).

The result is this graph:

The graph says it all, and definitely tells me that we need to be reserving all of our instances at Loggly - and it gives me a rule-of-thumb:

• If we're going to use one instance unit for at least 9 months, reserve for 3 years.
• If we're going to use one instance unit for at least 6 months, reserve for 1 year.
• Otherwise, stick with on-demand.

The "reserved instances" pay structure is you pay a one-time fee for access to a reduced hourly rate.

This also means that our random "debug something" deployments that are shutdown much of the time are probably best off being reserved instances as well- at least for a 1-year thing - since we are likely to use those deployments for more than half of a year.

A 3-year on-demand price for m1.large is just shy of $9000, which is twice as expensive as the 3-year reserve. Capaticy plan and maybe start buying reserved instances. Make your CFO happy.

And in case you were going to ask, I ran the same plot with data from EC2 "quaduple extra large" instances and the savings and break-even points were the same. I bet the rest of the prices flow similarly.

The R script is follows, run it with 'R --save < yourscript.r':

# Values taken from http://aws.amazon.com/ec2/pricing/
# for an m1.large ("Large") instance
on_demand_hourly = 0.34
reserve_hourly = 0.12
reserve_1year = 910       
reserve_3year = 1400

# quadruple extra large instances
#on_demand_hourly = 1.60
#reserve_hourly = 0.56
#reserve_1year = 4290
#reserve_3year = 6590

on_demand_daily = on_demand_hourly * 24
reserve_daily = reserve_hourly * 24
x <- c(0, 365)
y <- on_demand_daily * x

# Calculate day of break-even point reserve vs on-demand rates
break_1year_x = reserve_1year / (on_demand_daily - reserve_daily)
break_3year_x = reserve_3year / (on_demand_daily - reserve_daily)

png(filename = "ec2_m1large_cost.png", width = 500, height=375)
plot(x,y, type="l", col='red', xlab="", ylab="cost ($USD)")
title("EC2 cost analysis for m1.large", 
      sprintf("(days)\n1-year is cheaper than on-demand after %.0f days of usage,\n 3-year is cheaper after %.0f days", break_1year_x, break_3year_x))
text(60, 0, sprintf("on-demand=$%.2f/hour", on_demand_hourly), pos=3)

abline(reserve_1year, reserve_daily, col='green')
text(60, reserve_1year, sprintf("1-year=$%.0f+$%.2f/hour", reserve_1year, reserve_hourly), pos=3)

abline(reserve_3year, reserve_daily, col='blue')
text(60, reserve_3year, sprintf("3-year=$%.0f+$%.2f/hour", reserve_3year, reserve_hourly), pos=3)

point_y = reserve_1year + reserve_daily * break_1year_x
points(break_1year_x, point_y)
text(break_1year_x, point_y, labels = sprintf("%.0f days", break_1year_x), pos=1)

point_y = reserve_3year + reserve_daily * break_3year_x
points(break_3year_x, point_y)
text(break_3year_x, point_y, labels = sprintf("%.0f days", break_3year_x), pos=1)

dev.off()
quit()

Having become fed up with dealing with rpmbuild, spec files, debian control files, dh_make, debuild, and the whole lot, I automated my way back to sanity.

The result is a tool I call "fpm" which aims to help you make and mangle packages however you choose, all (ideally) without having to care about the internals of your particular native package format.

The goal of this project is not to undermine upstream packaging but to grant everyone the ability to trivially build and edit packages. Why? Not all software is packaged. Not all software of the version you want is packaged. And further, not all users are willing or able to take the time to learn all the ins and outs of their package build tools.

For example, you can package up your /etc/init.d directory as an RPM by doing simply this:

% fpm -s dir -t rpm -n myinitfiles -v 1.0 /etc/init.d
...
Created /home/jls/rpm/myinitfiles-1.0.x86_64.rpm

fpm will create a simple package for you and put it in your current directory. The result:

% rpm -qp myinitfiles-1.0.x86_64.rpm -l
/etc/init.d
/etc/init.d/.legacy-bootordering
/etc/init.d/NetworkManager.dpkg-backup
...

% rpm -qp myinitfiles-1.0.x86_64.rpm --provides
myinitfiles = 1.0-1
% rpm -qp myinitfiles-1.0.x86_64.rpm --requires
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1

You can package up any directory. But there's more.

Above, I didn't specify a package summary, so how about fixing the rpm to include the description? You can use RPMs as the source (-s flag) in fpm. There's also a helpful '-e' (--edit) flag that'll let you edit the rpm spec (or debian control) file before building.

% rpm -qp myinitfiles-1.0.x86_64.rpm --info | grep Summary
Summary     : no summary given

% fpm -s rpm -t rpm -e myinitfiles-1.0.x86_64.rpm
... this opens up $EDITOR so you can edit the spec file it generated ...
... make some changes to the spec, including adding a proper 'Summary' ...
Created /home/jls/rpm/myinitfiles-1.0-1.x86_64.rpm

% rpm -qp myinitfiles-1.0-1.x86_64.rpm --info | grep Summary
Summary     : my /etc/init.d directory

The '-s dir' flag says the source of the package is a directory. There's also support for other package sources like rubygems, other rpms, debs, and more on the way.

With FPM, you can specify dependencies, architecture, maintainer, etc. All from a simple command line, and never forcing you to learn the pain and suffering that can come with rpm spec files or debian package building.

You can install fpm with: gem install fpm

The project page is here: https://github.com/jordansissel/fpm

The wiki is here (has more examples): https://github.com/jordansissel/fpm/wiki

I was working on integrating jQuery Mobile stuff into fingerpoken and needed a way to make the content area of pages full-screen. By 'full screen' I mean still showing the header and footer, but otherwise the content needs to fill the rest.

I couldn't find an easy way to do this while googling, and even the jQuery Mobile demos didn't do it.

So here's a demo of what I came up with here: fullheight jQuery Mobile demo Javascript:

  var fixgeometry = function() {
    /* Some orientation changes leave the scroll position at something
     * that isn't 0,0. This is annoying for user experience. */
    scroll(0, 0);

    /* Calculate the geometry that our content area should take */
    var header = $(".header:visible");
    var footer = $(".footer:visible");
    var content = $(".content:visible");
    var viewport_height = $(window).height();
    
    var content_height = viewport_height - header.outerHeight() - footer.outerHeight();
    
    /* Trim margin/border/padding height */
    content_height -= (content.outerHeight() - content.height());
    content.height(content_height);
  }; /* fixgeometry */

  $(document).ready(function() {
    $(window).bind("orientationchange resize pageshow", fixgeometry);
  });

I'm giving a presentation this week at the Puppet Bay Area user meetup and while working on slides, I wanted to be able to present while not being attached to my laptop.

Enter: fingerpoken

Fingerpoken lets you turn your iphone/ipad/itouch into a touchpad, keyboard, and remote for another computer. The only required piece on your iphone is Safari. No appstore stuff to download!

Under the hood, it uses websockets and touch events and sends JSON-encoded requests to your workstation and will move the mouse, type, scroll, and more.

Project page: fingerpoken on github.

A short demonstration of this project in action:

Today starts the 3rd year of the SysAdvent calendar! How time flies!

What is SysAdvent? It's a 24-day event where I publish one excellent sysadmin article each day, starting December 1st. The articles are written by fellow sysadmins around the world.

Planning for this year has been quite a success. Many folks have contributed finished articles or drafts already and more have committed to writing about a topic. This is a huge step for the SysAdvent project.

SysAdvent is for sysadmins, so please share sysadvent with your coworkers, reddit, digg, twitter, and any other places with sysadmin communities.

The first article for this year is about Linux Containers (LXC). Go, read! Add sysadvent to your rss reader, too :)

Also, there are still 50ish articles from the past two years, all quite good. Have a look at the 2008 and 2009 years, too!

Truth should guide your configuration management tools.

Truth in this case is: what machines you have, properties of those machines, roles for those machines, etc. For example "foo-1.a.example.com is a webserver" is a piece of truth. Where and how you store truth is up to you and out of scope for this post.

My goal is to have truth steer everything about my infrastructure. Roles, jobs, and even long-term one-offs get put into the truth source (like a machine role, etc). That way, if the machine with a one-off dies, I can just add that machine role to another system and puppet will configure it - no pain and no fire to fight.

A simple example of truth in puppet is puppet's "node" type. A simple example:

node "foo-1.a.example.com" {
  include apache
}

Specifying each node in puppet doesn't scale very well for many people. Further, you may already have your node information in another tool (ldap, mysql, etc). To allow this, puppet lets you feed 'node' information from an external tool (called an external node classifier). However, I found that using an external node classifier also has its drawbacks (also out of scope for this post).

To avoid complex logic in a node classifier, I've got with a pure fact-based puppet configuration which I call "nodeless."

My puppet site.pp looks basically like this:

node default {
  include truth::enforcer
}

That's it. No extra nodes, no 'include' or conditionals randomly sprayed around.

From there, I have the truth::enforcer class include other classes, do sanity checking, etc, all based on facts (and properties if you use external node classifier)

Fully standalone example with code can be found here: https://github.com/jordansissel/puppet-examples/tree/master/nodeless-puppet/