On Jul 29, 2004, at 5:29 PM, Neil Schemenauer wrote:

> On Thu, Jul 29, 2004 at 10:17:39PM +0100, Jon Dyte wrote:
>> Given that using a PersistentDict seems pretty bad for this number
>> of objects, do you have a performance figure for same number of
>> things in the BTree prototype? Will they be released?

For a huge map, you can be sure that a commit of a single change  to a
BTree would be extremely fast compared to to the same operation
on a PersistentDict.   You can also be sure that we will release
BTree and any other general-purpose tools as soon as we have
them and like them well enough to share.  We'll be doing everything
we can to make Durus as useful as possible to everyone.

>
> I don't think David's BTree work has progressed very far yet.  We
> don't have a real need for them at this time.

That's true, but I really am working on it now because I think a BTree
is
sometimes a good choice in a Durus application.
As Neil says, though, it
depends entirely on what your application needs.
A PersistentDict is hard, and perhaps impossible to beat
if you have enough RAM to hold it in memory and if the change
rate is low, as is the case in most of our applications.

If your application has a mapping so large, that commits (when using
a PersistentDict) are too large and slow to tolerate, a BTree is an
alternative that I think would be worth considering because it
will tend to confine changes to smaller objects.
Other types of tree structures or linked lists will also have this
advantage.

>> In ZODB parlance I think what I need is an at a IOBTree.
>
> In that case you should be able to put together some sort of
> hierarchical structure that would give good performance.  For
> example, you could split the data into N sub-dictionaries.  You
> could find the sub-dictionary index by 'key % N'.  Details are left
> to the reader. :-)

When you know a lot about the set of keys, you have an advantage
that just might be worth using.

One feature of Durus (and ZODB) that I like is that it is pretty easy to
use code written for non-Persistent data structures by adding Persistent
here and there to __bases__ and sprinkling self._p_changed=1 in
the right places.