grok pattern match predicates
Posted Mon, 26 Feb 2007
I've added predicate tests to grok's pattern match system. These predicates allow you to specify an additional requirement on any matched patterns. Here's the grammar:
'%' pattern_name [ ':' subname ] [ operator value ] '%'The difference is that now you can put operator and values on the end of the pattern. The following are valid operators: < > <= >= == ~
- == < > <= >=
- Match equals, less than, etc. Should be obvious. One special note is that if both the match and predicate values are numbers, then the comparison is done using perl's numerical compare operators. Otherwise, string comparators are used (eq, lt, gt, etc).
- Regular expression match.
Still confused? Let's run through some examples.
- Let's find out what's going on in our auth.log on any day from 20:00 to 20:09:
% sudo cat /var/log/auth.log | ./grok -m '%TIME~/^20:0[0-9]/%' Sep 15 20:05:24 nightfall sshd: Server listening on :: port 22. Sep 15 20:05:24 nightfall sshd: Server listening on 0.0.0.0 port 22. Sep 15 20:07:31 nightfall login: login on ttyv0 as jls Nov 12 20:09:42 nightfall xscreensaver: FAILED LOGIN 1 ON DISPLAY ":0.0", FOR "jls" Nov 26 20:07:18 nightfall sshd: Server listening on :: port 22. Nov 26 20:07:18 nightfall sshd: Server listening on 0.0.0.0 port 22.
- How about looking through 'netstat -s' output for big numbers? Yes, you
can use awk for this particular example.
% netstat -s | ./grok -m "%NUMBER>100000%" 130632 total packets received 130465 packets for this host 114759 packets sent from this host
- Let's look in "all.log" (all syslog stuff goes here) for sshd lines with
an IP starting with '83.'
% ./grok -m "%SYSLOGBASE~/sshd/% .* %IP~/^83\./%" -r "%SYSLOGDATE% %IP%" < all.log Oct 17 09:54:37 22.214.171.124 Oct 17 09:54:53 126.96.36.199 Oct 17 09:56:02 188.8.131.52 <snip some output > Apr 16 06:54:52 184.108.40.206 Apr 16 06:54:53 220.127.116.11 Apr 16 06:54:54 18.104.22.168
This seems pretty powerful. Next feature I need to add is the ability to add predicates to patterns after they've been specified. Something like this would be sweet:
% ./grok -m "%APACHELOG%" -p "%NUMBER:RESPONSE==404%" < some output showing you all apache log entries with response code 404 >Something like that, which would let you modify the
%NUMBER:RESPONSE%pattern to add a predicate requiring that it be 404.