durusmail: durus-users: Durus and write intensive applications
Durus and write intensive applications
2004-12-08
2004-12-08
2004-12-08
2004-12-08
2004-12-08
2004-12-08
Durus and write intensive applications
Syver Enstad
2004-12-08
David Binger  writes:

> On Dec 8, 2004, at 10:30 AM, Syver Enstad wrote:
>
> >
> > I saw that Durus was listed as not suitable for write intensive
> > applications. Why is that? I ran a few tests and as long as I batched
> > together the updates into one commit it seemed to be fairly quick.
>
> It is space, not speed, that you need to worry about most.
> The pickles involved in every change are appended to the
> file on every commit.  Note that the important word here is
> "appended".  The file will grow with every change and it
> never shrinks, except during a pack operation.

Just like ZODB, I know.

> Packing is relatively slow and blocks other write operations, so you
> don't want to pack very frequently.

>
> It might be better to think about the maximum number of changes
> (and the total pickle size of those changes) over a full day period.
> If that size does not exceed the maximum size file you can
> tolerate, then Durus is still a contender.

Good advice. I'll do a little spike on that.

> Careful selection of data structures can make a big difference
> in the space required for each transaction.  BTree, for example,
> is much better than PersistentDict for this goal.

I discovered that with an application where I used ZODB, really quite
enormous difference in added .fs file size per write.

Thank you David for helpful advice.



reply