On Jan 6, 2005, at 2:48 PM, A.M. Kuchling wrote:

> The README and web page both include the following text:
>
>       Durus is best suited to collections of less than a
>       million instances with relatively stable state.
>
> Where does the figure of around a million come from?  Is it just
> pulled from the air, or does it stem from FileStorage having an
> in-memory dictionary mapping OIDs to file positions?


Almost thin air.  We haven't tested beyond that range.

The FileStorage does build an in-memory dictionary
mapping OIDS to file positions, and that starts to use a significant
amount of RAM eventually.  At a million objects this is no big deal,
but it might start to hurt at 100 million or a billion.

Also, the index is constructed when you start the server by seeking
through the records of the durus file.  The more objects you have, the
longer this takes.  If you double the number of objects, you probably
double the startup time (for the FileStorage).  The limit depends on
your
tolerance for startup delay and the speed of your system.

The storage file format puts some limits on the size of transaction
components
and on the oids and tids run out at 2^64.
The details of that are given in FileStorage.__doc__.

> I'm storing a simulation model.  Converting our largest model results
> in just over 100,000 objects, so that's already 10% of the way to the
> limit.  Toss in a few extra copies of the model for backups,
> versioning, or experimentation, and I'll probably reach the
> million-object mark fairly quickly.  Should I be worried?  Should I
> start thinking about how to make FileStorage more memory-efficient?

I would not worry.  If you want to be more confident, an experiment
might
help.  The MEMS Exchange servers run very nicely with something like
350,000
Persistent instances.  If it were me, I'd buy faster/bigger io/memory
before thinking about anything fancier than FileStorage.