Download grok

download grok-20070915

What is Grok?

Grok is simple software that allows you to easily parse logs and other files. You teach grok how to parse data through the config file. Grok is a system for reacting to events - those events being log entries.

Requirements: Perl 5.8.x, Parse::RecDescent, Unix::Syslog, Regexp::Common, Date::Parse (from TimeDate).

Why use Grok?

My servers get hit with brute-force ssh login attempts almost every day. They're also hit with several exploit attempts on varying protocols daily. Chances are all Internet-facing systems will see this on a regular basis. I've never had a successful break in, but why give script kiddies a chance?

My logs end up riddled with line after line of failed logins, etc. So I needed a system that could watch the logs

Failed login attempts are logged in /var/log/auth.log on FreeBSD, so that's where I'll be looking for patterns to match. An example of a failed login attempt is this:

Apr 30 22:19:32 kenya sshd[76279]: Failed password for illegal user test from 61.30.94.124 port 36884

This is an illegal login attempt for a username 'test' - this means there is no user 'test' on that server. I have made the assumption that no one will try to login to this machine with an invalid username more than once or twice in a short while, so that will be my criteria for blocking these script-kiddie login attempts.

Enter grok. This is a short perl program I wrote that allows you to configure reactions to certain patterns. The configuration file format is pretty straight forward, in my opinion.

A sample configuration follows:

file "/var/log/auth.log" {
  type "bad username" {
    match = "Illegal user %USERNAME% from %IP%";
    threshold = 4;
    interval = 300;
    reaction = "pfctl -t whores -T add %IP%";
  };
};

This will look for patterns matching the 'match' part of the configuration above. So, for example, if the following appears in the log file:

Apr 24 07:15:37 kenya sshd[44889]: Illegal user admin from 193.195.96.6

The match will be triggered and a count will be increased. If the threshold ever hits 4 in 300 seconds, the "reaction" will be executed. It will run pfctl -t whores -T add 193.195.96.6, which will add that IP to the 'whores' table in pf. Here is the pertinent section of my pf.conf:

table <whores> persist file "/etc/pf.whores"

# Block whores.
block in log quick on $ext_if from <whores> to any
block out log quick on $ext_if from any to <whores>

I find this to be pretty useful in helping to keep my machine free of brute-force attempts.

---

More advanced usage

The latest update to grok added support for more descriptively named pattern captures. This is an extremely necessary feature if you have multiple of the same patterns that need to be matched.

Example:

exec "tcpdump -li tl0 -n 2> /dev/null" {
  type "ssh-connect" {
    match = "%IP:SRC%.\d+ > %IP:DST%.22: S";
    reaction = "echo 'SSH connect(): %IP:SRC% -> %IP:DST%'";
  };
};

This will run tcpdump and grok the output for new connections to port 22. Sample output is as follows:

SSH connect(): 192.168.2.1 -> 192.168.50.4
SSH connect(): 192.168.4.8 -> 192.168.50.4
SSH connect(): 192.168.4.3 -> 192.168.50.4

This, immediately, may not seem useful. However, you'll note that grok is able to parse tcpdump output with very little effort on your part. Here's another example, where we look for port scans using the new 'key' feature:

exec "tcpdump -li tl0 -n 2> /dev/null" {
  type "port-scan" {
    match = "%IP:SRC%.%PORT% > %IP:DST%.%PORT:DST%: S";

    # Store each "hit" by source ip only
    key = "%IP:SRC%";

    threshold = 30;
    interval = 5;
    reaction = "echo 'Port scan from %IP:SRC%'";
  };
};

I then run 'nmap -sT' against this host:

Port scan from 129.21.61.220
Port scan from 129.21.61.220
Port scan from 129.21.61.220
Port scan from 129.21.61.220
Port scan from 129.21.61.220

grok is not a substitute for IDS software, nor is it IDS software, but the point of the above demonstrations are to show the usefulness of both the better capture naming and key generation. However, you may find it useful to use grok for IDS-type activities.

download grok-20070915

Mailing list

Grok has a mailing list hosted on Google Groups.

• URL: http://groups.google.com/group/grok-users/
• Email: grok-users@googlegroups.com