Grok and other plans
Posted Sat, 18 Mar 2006
Some time ago, I began a very long process of taking the massive quantities of data and having a machine process them for me. Spend a bit of time up-front to determine what data is definitely meaningful and let the computer handle the rest. The computer needs to process the raw data and display the data to me in a meaningful and readable format. Such formats include trend graphs, log summaries, and anomaly detection. Trend graphs are simple to do, assuming you have numeric data. Log summaries are easy if you know how and what you want to summarize. Anomalies are easy to detect if you know what you're looking for, or declare "anything unknown is badwrong... or badong."
Grok is the first step in having raw data turned into something easily readable. The next step is writing some magic software piece that lets me store arbitrary data (log entries, counters, key->value pairs, etc), possibly by date. This way, you can take grok's parsing ability and turn it into stored content. Now that you'll have super megatastic parsed log data, you'll want to turn it into something more human-meaningful - graphs or summaries. That's the 3rd piece.
So anyway, I got bored and started playing with Visio (generously provided by RIT's CS dept, ofcourse), and I came up with a little diagram of what I want grok and it's sister tools to do. The yellowish items are things I'll be writing. The rest aren't really software so much as stuff that happens. Here's a pretty diagram complete with a useful description of a "brick thing" -