On 20/03/2006, at 2:24 AM, Michael Watkins wrote:

> Question for Jesus - have you any thoughts or experience with using
> other
> backends for Storage? Curious what the work effort and plusses/
> minuses would
> be for using something like Postgres.

I spent a couple of hours noodling around in Durus to better
understand it and in the process knocked together a very simple
SQLite storage engine which gives a rough idea of what might be the
pros and cons of storing in a SQL database.

* Startup and memory usage are very light since there is no need to
build an index to the on disk records.
* Given that Durus currently hands already packed records to the
storage engine there is a bit of packing and unpacking going on in
the sqlite storage engine.
* Packing to remove un-referenced records is the part that I spent
most time thinking about and didn't come up with a satisfactory
solution.

For me the bottom line is that a SQL data store is able to offer fast
start-up and little memory usage, traded off against that is the
added complexity and overhead of how to handle packing well.
Personally I think if I were to pursue improving the large object
count performance of Durus (which is where my travels started) I'd
focus on just the on disk index of individual records so maybe
somehow it can be made that such that it just doesn't need to be read
completely on start-up.

If you'd like to have a look at what I put together with SQLite
support (sqlite_storage.py, test/utest_sqlite_storage.py and test/
stress.py) just let me know and I'll send the files through.

Peter W