Search this site


[prev]  Page 2 of 2

Metadata

Articles

Projects

Presentations

New fex version available (20071119)

Hop on over to the fex project page and download the new version.

Changelist:

20071119 -
  - Add nongreedy tokenizer. Same semantics of strtok_r(), but doesn't skip
    empty tokens.
  - Renamed tokenizer to split, since really that's what it was doing.
  - You can invoke the nongreedy tokenizer by using '?' as the first character
    of a {} set:
     args: :{?4,6}
     input: one:::four::six
     output: four:six

New fex version available (20071026)

Hop on over to the fex project page and download the new version.

Changelist:

20071026 - First major release
  - Added some tests
  - If you want to specify a different first split token, the first character 
    can be any non-digit character which is not '-' or '{'.

    These are now equivalent:
    % echo "foo/bar/baz" | fex 0/2
    bar
    % echo "foo/bar/baz" | fex /2
    bar

    Previously, this would give an error due to a design decision.

Overriding shared library functions

Long story short...

File: 'connect.over' contains

#include <netinet/in.h>

override(`connect', `
  {
    // code to inject before the connect() call is actually made
  }
')
Output is 'connect.so' which overrides libc's connect function.
% LD_PRELOAD=./connect.so nc google.com 80
stream connect: fd=3 host=64.233.187.99:80
% LD_PRELOAD=./connect.so nc -u 129.21.60.9 53 
dgram connect: fd=3 host=129.21.60.9:53
% LD_PRELOAD=./connect.so ssh scorn           
stream connect: fd=3 host=129.21.60.26:22
stream connect: fd=4 host=109.112.47.115:12148
scorn(~) %
The output by nc was due to my function above outputting this.

The strange ssh connection on fd=4 above is seemingly due to ssh calling connect() on a tty? fstat says:

jls      ssh         3221    4 /dev        122 crw--w----   ttypd rw
inode 122 on /dev is /dev/ttypd.

xdotool project page posted

I've gotten enough positive feedback about xdotool to convince me to put up a real project page for it. You can view it here at /projects/xdotool

grok 20070224 released.

It's been almost a year since the first release of grok. I've finally found some energy to put into the project and it's time for another release.

Download: grok-20070224.tar.gz

A quick summary of the changelist (which comes with the tarball):

  • Lots of doc updates. More examples in the manpage.
  • Lots of new builtin patterns
  • More new filters like strftime, ip2host, and uid2user.
  • Fancier syslog matching options
  • New flags -m and -r. See this post about this change
  • filelist, catlist, and filecmd thanks mostly to Canaan Silberberg.
  • More tests to make sure that it works. Find these in the 't' directory in the grok tarball.
Email me if the tests provided don't work.

New event recording database prototype

I finally managed to find time today to work on my events database project. In the processes of doing so, I found a few bugs in grok that needed to get fixed. Some of my regular expressions were being a bit greedy, so certain pattern expansion was breaking.

To summarize the event recording system, it is a webserver listening for event publish requests. It accepts the "when" "where" and "what" of an event, and stores it in a database.

To have my logs pushed to the database, I'll leverage the awesome power of Grok. Before I do that, I gathered all of the auth.log files and archives and compiled them into their respective files.

The grok.conf for this particular maneuver:

exec "cat ./logs/nightfall.auth.log ./logs/sparks.auth.log ./logs/whitefox.auth.log" {
   type "all syslog" {
      match_syslog = 1;
      reaction = 'fetch -qo - "http://localhost:8080/?when=%SYSLOGDATE|parsedate%&where=%HOST%/%PROG|urlescape|shdq%&what=%DATA:GLOB|urlescape|shdq%"';
   };
};
This is farily simple. I added a new standard filter, 'urlescape' to grok becuase I needed it. it will url escape a data piece. Hurray!

Run grok, and it sends event notifications to the webserver for every syslog-matching line. Using FreeBSD's command-line web client, fetch.

sqlite> select count(*) from events;
8085
Now, let's look for something meaningful. I want to know what happened on all sshd services between 1am and 3am this morning (Today, May 3rd):
nightfall(~/projects/eventdb) % date -j 05030100 +%s
1146632400
nightfall(~/projects/eventdb) % date -j 05030400 +%s
1146643200
Now I know the Unix epoch times for May 3rd at 1am and 4am.
sqlite> select count(*) from events where time >= 1146632400 
   ...> and time <= 1146643200 and location like "%/sshd" 
   ...> and data like "Invalid user%";
2465
This query is instant. Much faster than doing 'grep -c' on N log files across M machines. I don't care how good your grep-fu is, you aren't going to be faster.This speed feature is only the beginning. Think broader terms. Nearly instantly zoom to any point in time to view "events" on a system or set of systems. Filter out particular events by keyword or pattern. Look for the last time a service was restarted. I could go on, but you probably get the idea. It's grep, but faster, and with more features.

As far as the protocol and implementation goes, I'm not sure how well this web-based concept is going to prevail. At this point, I am not interested in protocol or database efficiency. The prototype implementation is good enough. From what I've read about Splunk in the past months in the form of advertisements and such, it seems I already have the main feature Splunk has: searching logs easily. Perhaps I should incorporate and sell my own, better-than-Splunk, product? ;)

Bear in mind that I have no idea what Splunk actually does beyond what I've gleaned from advertisements for the product. I'm sure it's atleast somewhat useful, or no one would invest.

Certainly, a pipelined HTTP client could perform this much faster than doing 10000 individual http requests. A step further would be having the web server accept any number of events per page request. The big test is going to see how well HTTP scales, but that can be played with later.

At this point, we have come fairly close to the general idea of this project: Allowing you to zoom to particular locations in time and view system events.

The server code for doing this was very easy. I chose Python and started playing with CherryPy (a webserver framework). I had a working event reciever server in about 30 minutes. 29 minutes of that time was spent writing a threadsafe database class to front for pysqlite. The CherryPy bits only amount to about 10 lines of code, out of 90ish.

The code to do the server can be found here: /scripts/cherrycollector.py

XML Presenter now supports printable slides

I haven't updated the codebase available from the xmlpresenter project page in a while, but development still continues on it as I do more presentations.

Today's update was to add printability to the slides. This is done by using the media="print" part of the <link> tag.

<link rel="stylesheet" type="text/css" media="screen,projection" href="presenter.css"/>

<link rel="stylesheet" type="text/css" media="print" href="printview.css"/>
When you print, a different css will be applied than 'presenter.css' - very cool.

Specific changes are:

  • all slides are shown
  • slide titles use a smaller font, and lack borders
  • slides with no titles are not printed
Useful, I suppose, if you want to print out your slides in "outline" form.

Check out the Unix Basics slides and look at it with "print preview" - Unix Basics slides

newpsm/newmoused update - merge into -CURRENT preparation

Lots of hours were spent today preparing the new moused and psm code for import into the FreeBSD source tree. I don't have a commit bit to CVS, so I'll have to wait on having it committed. This wait time will probably be spent fixing bugs, writing a decent rc script, and improving configuration options.

This update is only known to work for -CURRENT. The patch can be found on the newpsm project page. I had a friend test the patch against 6.0-RELEASE, and it seemed to apply cleanly with the exception that patch(1) got confused about sys/sys/mouse.h. Tell patch to not attempt to reverse-apply the patch, then tell it yes for trying the patch anyway. No guarantees if it doesn't build.

You can find almost all the information you need on the newpsm project page. If you find bugs, are interested in helping test, or have questions or comments, please contact me :)

Configuration file support added to moused and its modules

With the simple api that is getcap(3), moused and it's device modules will soon be able to be configurable from config files. The code is still in perforce to do this, however, and not on this site (at time of writing). A simple config file could be something like this:
synaptics:virtscroll:notouchpad:
This would modify options ONLY for the synaptics driver and enable "virtual scrolling" and disable the touchpad. Currently, only 'notouchpad' is supported (again, at time of writing).

I really want to add lots of options, mostly for synaptics support, but this would allow for some serious flexibility in mouse configuration. You can specify any kind of mouse you want and whatever config you please. Personally, I'd love to see an option to disable movement on the touchpad, and only allow the touchpad to be used for scrolling. Furthermore, allow me to decrease the sensitivity of the touchpad so that near-finger touches and palm touches don't trigger scrolling.

Wee! I love programming for fun. Makes me glad I'm not going to ever be a software developer. I like to code for myself and for my own goals.

pam_captcha 1.2 updated

This update fixes a few potential bugs and cleans up some other issues. I removed the 'cowsay' requirement and made the math and dda captchas compile-time options.

Check out the project page for information on pam_captcha and downloading it.
pam_captcha