C vs Python with Berkeley DB
Posted Thu, 06 Dec 2007
Then again, how much should insert performance matter on a monitoring tool? For data that comes into it gradually, speed doesn't matter much. For bulk inserts, speed matters if you want to get your work done quickly. I haven't decided if bulk insertions are necessary use case for this tool. Despite that, I'm still interested in what the limits are.
I have experimented with many different implementations of parallelism, buffering, caching, etc in the name of making insertion to a fancydb with 10 rules fast. The fastest I've gotten it was 10000/sec, but that was on an implementation that wasn't threadsafe (and used threads).
My most-recent implementation (which should be threadsafe) can do reads and writes at 30000/sec. With evaluation rules the write rate drops to about 10000/sec.
The next task was to figure out what I was doing wrong. For comparison, I wrote two vanilla bdb accessing programs. One in C and one in Python. The output of these two follows:
# The args for each program is: insertions page_size cache_size % sh runtest.sh Running: ./test 2000000 8192 10485760 => 2000000 inserts + 1 fullread: 209205.020921/sec Running: ./py-bsddb.py 2000000 8192 10485760 => 2000000 inserts + 1 fullread: 123304.562269/secAs expected, C clearly outperforms Python here, but the margin is pretty small (C is 69% faster for this test). Given the 120000/sec rate from Python, the poor input rate of my tool seems to be blamed on me. Is my additional code here really the reason that I can only write at 30000 per second? I may need to revisit how I'm implementing things in python. I'm not clear right now where I'm losing so much throughput.
So I use hotshot (python standard profiler) and I find that most of the time is spent in my iterator method. This method is a generator method which uses yield and loops over a cursor.
It's important to note that my python bdb 'speed test' above did not use generators, it used a plain while loop over the cursor. So, I wrote another test that uses generators. First, let's try just inserts, no reading of data:
Running: ./test 1000000 8192 10485760 => 1000000 inserts: 261096.605744/sec Running: ./py-bsddb.py 1000000 8192 10485760 => 1000000 inserts: 166389.351082/secNow let's try with 3 different python reading methods: while loop across a cursor, generator function (using yield), and an iterator class (implementing __iter__):
Running: ./py-bsddb.py 4000000 8192 10485760 => 1 fullread of 4000000 entries: 8.660000 Running: ./py-bsddb_generator.py 4000000 8192 10485760 => 1 fullread of 4000000 entries: 9.124000 Running: ./py-bsddb_iterable_class.py 4000000 8192 10485760 => 1 fullread of 4000000 entries: 13.130000I'm not sure why implementing an iterator is so much slower (in general) than a yield-generator is. Seems strange, perhaps my testing code is busted. Either way, I'm not really closer to finding the slowness.