durusmail: durus-users: Re: PersistentDict vs BTree
PersistentDict vs BTree
2006-10-03
2006-10-03
2006-10-03
2006-10-03
2006-10-05
2006-10-05
2006-10-05
2006-10-05
2006-10-11
2006-10-11
2006-10-11
2006-10-12
2006-10-12
2006-10-12
2006-10-13
2006-10-13
2006-10-13
2006-10-13
2006-10-13
2006-10-13
Re: PersistentDict vs BTree
2006-10-23
2006-10-31
Re: PersistentDict vs BTree
Neil Schemenauer
2006-10-13
Mike Orr  wrote:
> Lowering the Durus cache to a measly 20 objects and making Chemical
> Persistent again (which required disabling .__slots__) brought the
> memory up to 190 MB (from 121), so that's not an option.
>
> The Chemical object has 153 slots, by the way, so slots are performing
> well even with a large number of attributes.

Hi Mike,

I haven't been following this thread closely but perhaps you are
running into a pickle issue.  I believe that when you unpickle
something, strings are not interned.  If you have many objects using
a small set of strings then the memory usage can be much higher than
necessary.

The trick to avoid this is to do something like the following:

    class Record(PersistentDict):
        def __setitem__(self, key, value):
            key = intern(key)
            PersistentDict.__setitem__(self, key, value)



reply