durusmail: durus-users: Replication - thoughts, patches, thinking out loud.
Replication - thoughts, patches, thinking out loud.
2009-06-21
Peter Wilkinson (4 parts)
Replication - thoughts, patches, thinking out loud.
2009-06-26
2009-07-02
2009-07-02
Replication - thoughts, patches, thinking out loud.
2009-07-07
Replication - thoughts, patches, thinking out loud.
Replication - thoughts, patches, thinking out loud.
2009-07-08
Replication - thoughts, patches, thinking out loud.
Peter Wilkinson
2009-06-21
Hi,
I've been thinking about replication of Durus for a while and being
bored decided to try out a theory I've come up with.

Firstly the issue that I've been stuck on is how to get a Durus
replication setup going such that the db is rarely more than a few
seconds out of date. Using rsync is nice and easy but has some fairly
well known issues (is the replica really the same as the master? how
to deal with packing? how often do you run it?)

My hope was to have a simple setup that would allow for master->slave
replication and deal with changing masters due to packing.

Attached are 2 files, a changed shelf.py and a new replicate.py which
work with the standard Durus 3.8 tar.gz sources.

The changes to shelf.py are fairly simple and basically boil down the
having a hash value calculated for each transaction that is committed
and tacking that on at the end of the transaction. ie. the format for
a transaction looks like: 8 bytes length, x bytes transaction data, 20
bytes hash. The 20 bytes are included in the transaction length.

This hash is calculated on the previous hash value concatenated with
the new transaction data. My thinking is that by calculating a hash
through the file like this a slave can be compared to the master by
just comparing the last 20 bytes of the slave with the 20 bytes at the
same index in the master file, any changes in any previous transaction
would show up as a different hash.

By being able to safely compare the 2 files the replicate script is
able to just append any transactions that are missing from the slave
after comparing the hash from the master with a calculated hash on the
slave.

I've tested the attached files and it works so far in the testing I've
done.

An example of firing a replication off is:
"replicate.py -d master_replica.durus -s sites/master/var/master.durus"
which replicates the file in sites/master/var/master.durus to
master_replica.durus or
"replicate.py -d master_replica.durus -s `pwd`/sites/master/var/
master.durus -n pfw@host"
which does the same thing over ssh.

I'd be very interested in any thoughts anyone might have, gaping holes
that I've missed, etc.

Regards,
Peter W.




Peter Wilkinson
pw@thirdfloor.com.au

Thirdfloor Software Works Pty Ltd
http://www.thirdfloor.com.au

Sydney       +61 2 8916 7366



reply