durusmail: durus-users: Replication component?
Replication component?
2010-03-08
2010-03-09
2010-03-09
2010-03-10
Replication component?
Jesus Cea
2010-03-08
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I am the maintainer of Berkeley DB backend for Durus.

The private branch of my code includes support for replication, between
other features. Now I am considering porting some of the code from the
"closed" version to the GPL one, and I think that replication could be
done trivially at Durus level: just call a function with the modified
objects pickls for each transaction. The function can do something so
simple as storing the pickles on disk, on a increasing filenumber name.

"Replaying" those pickles in order would replicate the storage.

I am doing just this when Berkeley DB replication is not applicable,
like when upgrading the backend storage format:

* Stop the program.

* Using ZFS (Solaris), I do an "instant" snapshot of the storage
database. This take a couple of seconds only.

* Configure the program to store the commit history (the pickles of
transactions).

* Launch the program again. It will keep the story.

* Now I clone the snapshot, aside, to be able to modify it. I run the
upgrade scripts there. This can take a lot of time (proportional to
database size). I have quite a few multi terabyte databases. This can be
hours. Or days. Or weeks.

* After I do the upgrade, I apply the pickles with the changes since the
snapshot, in order. This accoumulated data can be pretty large, so I
keep the application running and storing new pickles.

* When I finish the pickles backlog, I stop the application, apply the
last few pickles and swap the new database in place (ZFS magic).

* Launch the aplication again, running with an upgraded database, and
pickles logging disabled.

Would Durus developers consider this addition?.

PS: I do this using an "stacked" approach to Durus storage backends. It
is useful, for instance, for having a big read-only and small read-write
storages at once.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea@jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea@jabber.org         _/_/    _/_/          _/_/_/_/_/
..                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBS5VJnJlgi5GaxT1NAQL8zQP7BL/PJC26TdY/KNldZ5m4b2OsDuplwxv8
ubxHywZlg7RWDRNDuTlCKk6Se9f3o76kk/jr4wvAX4qHF74I7A3UCsO/SoDQbHH5
VNU+KA9+jyZw6/MK7MNtCPAyt5YMw5JGCxnEoe+weoYmTCQB0s2TAdfQaDAwRvd6
4cdkHoHk4qQ=
=SHUQ
-----END PGP SIGNATURE-----
reply