durusmail: durus-users: Durus and write intensive applications
Durus and write intensive applications
2004-12-08
2004-12-08
2004-12-08
2004-12-08
2004-12-08
2004-12-08
Durus and write intensive applications
David Binger
2004-12-08
On Dec 8, 2004, at 10:30 AM, Syver Enstad wrote:

>
> I saw that Durus was listed as not suitable for write intensive
> applications. Why is that? I ran a few tests and as long as I batched
> together the updates into one commit it seemed to be fairly quick.

It is space, not speed, that you need to worry about most.
The pickles involved in every change are appended to the
file on every commit.  Note that the important word here is
"appended".  The file will grow with every change and it
never shrinks, except during a pack operation.  Packing is
relatively slow and blocks other write operations, so you
don't want to pack very frequently.

It might be better to think about the maximum number of changes
(and the total pickle size of those changes) over a full day period.
If that size does not exceed the maximum size file you can
tolerate, then Durus is still a contender.  If not, you might
need a storage system that changes the stored information
in-place, or one that has a packing strategy that can work
incrementally.

Careful selection of data structures can make a big difference
in the space required for each transaction.  BTree, for example,
is much better than PersistentDict for this goal.


reply