On Aug 14, 2006, at 5:29 AM, Terry Jones wrote:

> I note with interest that A.M. Kuchling, author of the ZODB/ZEO
> programming
> guide, is also active in Durus.

Andrew worked at the MEMS Exchange for several years, and we used
ZODB/ZEO for our applications during that time.  He is a kind of
grandfather of Durus.

> My main (initial) reason for considering using Durus is the
> availability of
> Jesus' bsddb storage. I have a lot of confidence in bsdbd and no
> experience
> with ZODB (so no basis for any strong opinion).

Jesus is the right person to ask about that, and it sounds like his
storage would be a good choice for you.

> Based on reading stuff
> online I am worried that ZODB files may grow too rapidly and/or
> that it may
> not do so well if a storage grows beyond 10-20Gb.

The basic storage for Durus, FileStorage, works like the one in ZODB,
in that
it just appends transactions to a file.  Both grow on every write,
and both
use packing to remove obsolete versions.  On the other hand, this
kind of storage
is very fast, simple, and reliable.  It is a good choice for some
applications,
but not far all.

Durus itself does not care about the size of the storage file, and I
assume this
is also true of ZODB.  The current Durus implementation does,
however, use a
plain dict in the FileStorage to hold an offset index for the current
version
of every instance.  This is very fast, but it does require RAM, about
100 bytes
for each instance.  It depends on your machine, of course, but this
could start
to be significant if you have several million instances.  A future
version of
FileStorage will offer the possibility of keeping most of the offset
index on disk,
so that the memory requirement will be reduced.

ZODB implements much more in C.  I assume ZODB's offset index and
BTrees are more
compact in RAM than they are in Durus.

In our applications, the overwhelming performance factor is the
client cache.
When the objects you need are already in RAM, the response is very fast.
Applications that can't hold a reasonable working set in RAM won't
perform
nearly as well in any Durus (or ZODB) environment.