Thanks! FYI, after some more diagnosis, it turns out that this particular kind
of corruption seems to be correlated with running out of disk space.

Dave

On Aug 12, 2011, at 2:44 PM, David Binger wrote:

> This change looks good to me.  Nice work.
>
> On Aug 11, 2011, at 10:37 AM, David Hess wrote:
>
>> We have servers that deal with power outages (and dirty power in general) and
have ended up with some corrupted durus databases. We've used the normal
"repair" feature to handle a lot of the cases but we have another that's
occurring on occasion that is not handled by repair. It is failing as this
exception:
>>
>> Traceback (most recent call last):
>> File "/usr/local/bin/durus", line 22, in 
>>   client_main()
>> File "/usr/local/lib/python2.6/dist-packages/durus/client.py", line 108, in
client_main
>>   options.startup)
>> File "/usr/local/lib/python2.6/dist-packages/durus/client.py", line 35, in
interactive_client
>>   storage = FileStorage(file, readonly=readonly, repair=repair)
>> File "/usr/local/lib/python2.6/dist-packages/durus/file_storage.py", line 73,
in __init__
>>   self.shelf = Shelf(filename, readonly=readonly, repair=repair)
>> File "/usr/local/lib/python2.6/dist-packages/durus/shelf.py", line 91, in
__init__
>>   self.file, repair=repair)
>> File "/usr/local/lib/python2.6/dist-packages/durus/shelf.py", line 296, in
read_transaction_offsets
>>   file.seek(position + 8 + record_length)
>> File "/usr/local/lib/python2.6/dist-packages/durus/file.py", line 41, in seek
>>   self.file.seek(n, whence)
>> IOError: [Errno 22] Invalid argument
>>
>> We've come up with the following patch:
>>
>> --- /usr/local/lib/python2.6/dist-packages/durus/shelf.py    2007-04-24
16:10:16.000000000 -0500
>> +++ durus/shelf.py   2011-08-11 08:18:51.169288642 -0500
>> @@ -297,18 +297,17 @@
>>        if file.tell() != transaction_end:
>>            raise ShortRead
>>        return transaction_offsets
>> -    except ShortRead, e:
>> +    except (ShortRead, IOError), e:
>>        position = file.tell()
>>        if position > transaction_start:
>>            if repair:
>>                file.seek(transaction_start)
>>                file.truncate()
>>            else:
>> -                e.args = repr(dict(
>> +                raise ShortRead(repr(dict(
>>                    transaction_start=transaction_start,
>>                    transaction_end = transaction_end,
>> -                    position=position))
>> -                raise
>> +                    position=position)))
>>        return None
>>
>> Or test database started out as:
>>
>> -rw-r--r--  1 fishfinder fishfinder 49371713 2011-08-10 14:12 db.durus
>>
>> We ran with this patch and without the repair switch and got the following
(expected) report:
>>
>> durus.utils.ShortRead: {'position': 47501336L, 'transaction_start':
47501312L, 'transaction_end': 12061693972974895416L}
>>
>> We then ran again with --repair and the database was able to be opened. The
resulting database looked like this:
>>
>> -rw-r--r--  1 fishfinder fishfinder 47501312 2011-08-11 08:14 db.durus
>>
>> Can anybody comment on whether this patch makes the appropriate sense and is
safe enough? We (fortunately!) have a limited number of corrupted databases to
try this on.
>>
>> Thanks.
>>
>> Dave
>>
>> _______________________________________________
>> Durus-users mailing list
>> Durus-users@mems-exchange.org
>> http://mail.mems-exchange.org/mailman/listinfo/durus-users
>
>

------
David K. Hess
877.343.4947 x114
dhess@fishtechnology.com