This sounds like a capability that would be useful to us, too. When our site updates involve schema changes, such as a new attribute added to a class, we update a copy of the database and watch for new commits while the update is happening. If new commits occur, we just start over. If not, we switch to using the updated file and restart the application. Our downtime is just a few seconds. If I understand your idea correctly, we could keep a record of transactions that occur during the update and merge them into the update as a final step. That would be nice for us, especially if we ever have updates that take as much time as yours. The Durus server does something like this for packing, except that the list of accumulated records is held in memory instead of being written to disk. I'm not sure about the details. Why use separate files for all of the pickles? I think we could use the existing FileStorage code for this temporary store. It seems like the essential function required is a command that a client sends to the server to tell the server to start a transaction log. Is that right? - David On Mar 8, 2010, at 2:01 PM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I am the maintainer of Berkeley DB backend for Durus. > > The private branch of my code includes support for replication, between > other features. Now I am considering porting some of the code from the > "closed" version to the GPL one, and I think that replication could be > done trivially at Durus level: just call a function with the modified > objects pickls for each transaction. The function can do something so > simple as storing the pickles on disk, on a increasing filenumber name. > > "Replaying" those pickles in order would replicate the storage. > > I am doing just this when Berkeley DB replication is not applicable, > like when upgrading the backend storage format: > > * Stop the program. > > * Using ZFS (Solaris), I do an "instant" snapshot of the storage > database. This take a couple of seconds only. > > * Configure the program to store the commit history (the pickles of > transactions). > > * Launch the program again. It will keep the story. > > * Now I clone the snapshot, aside, to be able to modify it. I run the > upgrade scripts there. This can take a lot of time (proportional to > database size). I have quite a few multi terabyte databases. This can be > hours. Or days. Or weeks. > > * After I do the upgrade, I apply the pickles with the changes since the > snapshot, in order. This accoumulated data can be pretty large, so I > keep the application running and storing new pickles. > > * When I finish the pickles backlog, I stop the application, apply the > last few pickles and swap the new database in place (ZFS magic). > > * Launch the aplication again, running with an upgraded database, and > pickles logging disabled. > > Would Durus developers consider this addition?. > > PS: I do this using an "stacked" approach to Durus storage backends. It > is useful, for instance, for having a big read-only and small read-write > storages at once. > > - -- > Jesus Cea Avion _/_/ _/_/_/ _/_/_/ > jcea@jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ > jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ > ... _/_/ _/_/ _/_/ _/_/ _/_/ > "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ > "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQCVAwUBS5VJnJlgi5GaxT1NAQL8zQP7BL/PJC26TdY/KNldZ5m4b2OsDuplwxv8 > ubxHywZlg7RWDRNDuTlCKk6Se9f3o76kk/jr4wvAX4qHF74I7A3UCsO/SoDQbHH5 > VNU+KA9+jyZw6/MK7MNtCPAyt5YMw5JGCxnEoe+weoYmTCQB0s2TAdfQaDAwRvd6 > 4cdkHoHk4qQ= > =SHUQ > -----END PGP SIGNATURE----- > _______________________________________________ > Durus-users mailing list > Durus-users@mems-exchange.org > http://mail.mems-exchange.org/mailman/listinfo/durus-users