As a sysadmin, I get the privilege of sifting through piles of logs to find
anomalies. Logs are great. However, I don't have time to sift through every log
looking for data worth reading. I'd much prefer having only the data I want to
see displayed to me. Most of the time log audits are an all-or-nothing activity
- either you look at all of the data, or you look at none of the data. Looking
at all the data takes more time than it should, and ignoring data can be
hazardous (especially when tracking problems down).
Some time ago, I began a very long process of taking the massive quantities
of data and having a machine process them for me. Spend a bit of time up-front
to determine what data is definitely meaningful and let the computer handle the
rest. The computer needs to process the raw data and display the data to me in a
meaningful and readable format. Such formats include trend graphs, log
summaries, and anomaly detection. Trend graphs are simple to do, assuming you
have numeric data. Log summaries are easy if you know how and what you want to
summarize. Anomalies are easy to detect if you know what you're looking for, or
declare "anything unknown is badwrong... or badong."
Grok is the first step in having raw data turned into something easily readable.
The next step is writing some magic software piece that lets me store arbitrary
data (log entries, counters, key->value pairs, etc), possibly by date. This way,
you can take grok's parsing ability and turn it into stored content. Now that
you'll have super megatastic parsed log data, you'll want to turn it into
something more human-meaningful - graphs or summaries. That's the 3rd piece.
So anyway, I got bored and started playing with Visio (generously provided by
RIT's CS dept, ofcourse), and I came up with a little diagram of what I want
grok and it's sister tools to do. The yellowish items are things I'll be
writing. The rest aren't really software so much as stuff that happens. Here's a pretty diagram complete with a useful description of a "brick thing" -
In summary. Data (logs, etc) are extremely noisy. Use grok and other tools to
turn raw data into useful data for you. This will keep you reading your logs
aswell as keeping you sane. See how happy the sysadmins are? The smiley faces
indicate happiness... I promise.