Storage utils, eventdb, etc.
Posted Mon, 26 Nov 2007
Again, I still don't have releaseworthy code, but here's a snippet of rrdtool-like behavior from this system:
% ./evtool.py create /tmp/webhits.db % ./evtool.py addrule /tmp/webhits.db http.hit agg.http.hit.daily total $((60*60*24)) time % time cat webhits.data | ./evtool.py update /tmp/webhits.db - 11.10s user 0.80s system 94% cpu 12.627 total % time ./evtool.py graph /tmp/webhits.db agg.http.hit.daily 0.49s user 0.11s system 96% cpu 0.624 totalThe result is exactly the same graph as mentioned in my previous post. Speed so far is pretty good. The input was 125000 entries, in 12.6 seconds; which equates roughly to 10000 updates per second. That kind of QPS seems pretty reasonable.
The primary difference today is that the aggregates are computed as data enters the system. 'Addrule' tells the database to schedule an aggregation for specific timestamps.
The goal is to be able to chain rules, and have N:M relationships between rule input and output. Those will happen soon. Chaining would've happened tonight, but I'm having some locking problems due to it being quite late ;)
The database code itself is designed to be reusable elsewhere. There are two primary classes: SimpleDB and FancyDB. SimpleDB lets you store and retrieve data based on row+timestamp => value pairs. FancyDB wraps SimpleDB and gives you operation listeners such as the rule used in the above example.
I've already used SimpleDB elsewhere; in the sms traffic tool I mentioned in my last post, I cache geocode data and traffic requests with this same database tool.