durusmail: durus-users: building a large BTree efficiently
building a large BTree efficiently
2005-10-14
2005-10-14
2005-10-14
2005-10-17
2005-10-17
2005-10-17
2005-10-17
2005-10-17
2005-10-19
2005-10-20
2005-10-20
2005-10-20
2005-10-20
2005-10-20
2005-10-26
2005-10-26
2005-10-26
building a large BTree efficiently
Patrick K. O'Brien
2005-10-15
mario ruggier wrote:
> On Oct 14, 2005, at 10:44 PM, David Binger wrote:
>
>> On Oct 14, 2005, at 1:37 PM, mario ruggier wrote:
>>
>>>
>>> Can anything be done to alleviate this?
>>
>>
>> It might shed some light on this to run thus using the
>> hotshot module.  Also, what's happening with your process
>> size as this runs?
>
>
> Hotshot? Sounds trés cool... was not aware of that module. I will play
> with that.

This is the sort of thing I do with hotshot.  Some variation of this
might be useful for you:

def profile(func, *args):
    import hotshot
    import hotshot.stats
    import schevo
    filename = 'schevo.prof'
    prof = hotshot.Profile(filename)
    prof.runcall(func, *args)
    prof.close()
    stats = hotshot.stats.load(filename)
    # Print reports including all code, even dependencies.
    stats.sort_stats('cumulative', 'calls')
    stats.print_stats(50)
    stats.sort_stats('time', 'calls')
    stats.print_stats(50)
    # Print reports showing only Schevo code.
    stats.sort_stats('cumulative', 'calls')
    schevo_package_path = os.path.dirname(schevo.__file__)
    schevo_package_path = schevo_package_path.replace('\\', '\\\\')
    # Hotshot stores a lowercase form of the path so we need to
    # lowercase our path or the regular expression will fail.
    schevo_package_path = schevo_package_path.lower()
    stats.print_stats(schevo_package_path, 50)
    stats.sort_stats('time', 'calls')
    stats.print_stats(schevo_package_path, 50)

Hope that helps.

--
Patrick K. O'Brien
Orbtech       http://www.orbtech.com
Schevo        http://www.schevo.org
Pypersyst     http://www.pypersyst.org
PyDispatcher  http://pydispatcher.sourceforge.net

reply