On 28/06/2006, at 6:30 AM, David Binger wrote:

> I have some Durus changes in mind that will help with startup time
> and memory usage
> when managing a large number of objects.
>
> How large is your number of objects?

The dataset can get up to millions of objects but 250-500k is not
uncommon.

> Are you more concerned about startup time and memory usage of the
> storage server,
> or of clients?

Both, SQLite helps with the storage server part, instant startup is
nice but no memory for index is the biggy, running an async web
server helps with the client side of things. We're not completely
attached to the SQLite storage engine so if those 2 issues went away
we'd have a serious look at changing back to FileStorage. SQLite is a
well known quantity for us and so are quite comfortable with it and
the storage engine code for it is only 163 lines including comments
which we can live with maintaining separately. There are a couple of
other advantages of running SQLite (eg. we can replace records rather
than append to end of file so that file size is more constrained and
pack less frequent) but none of those are ones we can't live without.

We are in a position where we will likely be running at least 50 and
often 100 separate domains on a single machine. Getting
that running with the normal web server (we run lots of sites using
Apache and reverse proxying and will likely continue that with QP)
with at least 5 processes per domain starts getting interesting
memory wise. These sites aren't seeing huge traffic, the largest in
the order of 80k to 150k page or other dynamic content requests a day
most around the 15k to 30k mark, but do touch quite a few objects in
their life.

Peter W