durusmail: durus-users: Re: multithread and implicit cache management
RELEASED: Durus-3.5
2006-08-16
2006-09-21
multithread and implicit cache management (was: Re: [Durus-users] RELEASED: Durus-3.5)
2006-09-21
Re: multithread and implicit cache management
2006-09-21
2006-09-22
2006-09-22
2006-09-22
2006-09-26
2006-09-21
2006-09-21
2006-09-22
2006-09-22
Re: RELEASED: Durus-3.5
2006-09-21
Re: multithread and implicit cache management
Jesus Cea
2006-09-22
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Binger wrote:
> Here is a variation of the solution you suggested for
> just holding a plain dict of weakrefs.  Please review it.

Good.

> +    def __setitem__(self, key, value):
> +        self.mapping[key] = KeyedRef(value, self.remove, key)

What if that key was previously deleted and it is in "self.dead"?. You
add a new reference know, but it will vanish when we process
"self.dead". My original code has the same issue. If you do multiple
statements there, you probably have race-conditions when multithreading.

Studying the python "set" API, I think you can do a "set.discard()"
atomically even if the key doesn't exist. If you FIRST make sure that
self.dead is updated, that code would be safe.

But then you have the issue in the "self.dead" clearing, were you could
extract a key to delete and, before deleting it, other thread could
recreate the reference. Then you delete the reference, but the object is
still out there.

I'm starting to see that locking inside the dictionary object would be
appropiate. Locking primitives in python are fairly efficient. In my
machines, acquiring/releasing calls for "simple" locks are very fast:

=====
$ python
Python 2.5 (r25:51908, Sep 20 2006, 17:33:27)
[GCC 4.1.1] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> from threading import Lock
>>> from time import time
>>> def a() :
....   l=Lock()
....   t=time()
....   for i in xrange(1000000) :
....     l.acquire()
....     l.release()
....   return time()-t
....
>>> a()
1.3527078628540039
>>> a()
1.3447511196136475
>>> a()
1.3412230014801025
>>>
=====

So about 750.000 acquire/release pairs per second (Opteron 2.2Ghz),
including loop overhead. Lightweight enough to play safe, I guess.


> +    def __delitem__(self, key):
> +        self.dead.add(key)

Race condition if a thread is adding an element and other is deleting it.

These potential race-conditions would be nonexistant in current Durus
usage, nevertheless.

I will try-out your code in my machines.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea@argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea@jabber.org         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBRRQYgZlgi5GaxT1NAQK4rQP/Sgb4rSjAPgRnspUZNl0Wv1XtWrv80+RU
ZJJC+Sa1q2BgND4CL4vwxbh/MlHlkSh1IsZBQnELydkay5w4787CRA3zYiNdGuHN
jP/uhMjSLuFCv74wib6uJqrsgMdPIG6btEpCOITJc2Beub9u0EPcFGPf4UE2CRB8
2xQKpLGGEVk=
=vJmV
-----END PGP SIGNATURE-----
reply