On Feb 28, 2006, at 2:08 AM, Matthew Scott wrote:

> To tackle this, the Schevo team should work with the Durus team to
> identify a solution to this that can at least optionally be used to
> limit the amount of memory Durus is allowed to use for in-memory
> indices
> (e.g. if performance suffers slightly with a paged index vs. a
> completely in-memory index).
>
> Before this is done though, it would be good to perform some
> benchmarks
> to see if it is satisfactory to just let Durus's in-memory index grow
> unbounded, and let the operating system's swap facilities take care of
> paging it to/from disk.

The RAM required for the index is roughly 100MB for each
million persistent instances, so I don't think this is an issue at
all until you have at least a few million instances.

Still, it would be nice to be able to limit the server's memory
usage, and here is one idea that could be implemented
without major change in the durus file format.
Currently, when the database is packed, we also write
out a pickle of the index of all instances in the packed
database.  This allows for a faster startup.  Instead
of walking through the entire file and building the index
from scratch, the pickled index is just loaded.

The idea is to write the index in some format other than
pickle that can be used without loading the whole thing
first.  We would still keep an in-memory index of all of
the object records written since the last pack, but that would
be relatively small.