durusmail: durus-users: RELEASED: Durus-3.5
RELEASED: Durus-3.5
2006-08-16
2006-09-21
multithread and implicit cache management (was: Re: [Durus-users] RELEASED: Durus-3.5)
2006-09-21
Re: multithread and implicit cache management
2006-09-21
2006-09-22
2006-09-22
2006-09-22
2006-09-26
2006-09-21
2006-09-21
2006-09-22
2006-09-22
Re: RELEASED: Durus-3.5
2006-09-21
RELEASED: Durus-3.5
Jesus Cea
2006-09-21
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

David Binger wrote:
>     * Fix a bug introduced in version 3.4 that could, under certain
> conditions, allow conflicts to be missed.

Nice fix.

>  * Revise the cache code.  It now uses a WeakValueDict instead of a
> plain dict to hold the references.  This simplifies the code because we no
> longer need to call the weakref instances directly.  It also helps the cache
> shrinking loop because the weakref callbacks have an immediate impact on the
> size of the mapping.

I just discovered a regressiĆ³n problem with this approach, related to
multithreading.

In previous Durus release, you could play safe in a multithreaded
environment if a) you don't share durus connections between threads or,
more interesting, b) you take great care when using persistent objects.

Let me explain last point:

You can share persistent objects/connection between threads if you take
steps to serialize access to persistent objects. The usual approach is
to use a global lock, and be sure you touch persistent objects ONLY when
having the lock under your wing.

This worked very nicely until Durus 3.5.

Previously the connection cache management was involved when a) you have
a persistent object "miss", or b) you do a connection commit/abort. Both
cases were *explicit* in the code, so you could be sure to adquire the
global lock.

With durus 3.5, nevertheless, there is a new point where cache
management comes around: object garbage collection. Worse, those points
are "implicit". That is, you can't be really sure where your persistent
objects will be collected.

Look at this code:

def example(s,key) :
  global_lock.adquire()
  conn.abort()  # Be sure your data is fresh
  root=conn.get_root()
  data=root["data"]
  value=data[key] # Retrieve the data
  conn.commit()
  global_lock.release()
  s.send(value) # Can take a lot of time
  return

Suppose that code is used in several simultaneous threads.

This code reads a data from the persistent storage (with a lock, to be
sure to serialize durus accesses) and then send the data to the user.
Since it already have the data in a non persistent variable, we can
release the lock, allowing other threads in durus while we are sending
the data. This is useful because the client can be very slow reading the
data.

If we upgrade to Durus 3.5, this code doesn't work!!.

Why?.

Because when the function finishes, "data" goes out of scope and, since
now the cache is managed as a weak-dictionary, the object "dissapears"
from cache "implicitelly"... while other thread can be potentially
inside the cache code, with the lock!!!.

God heaven!!.

I will expand this in next message.

>     * In the server, add support for a protocol verification command.
>       Use this in the client constructor to allow orderly behavior if
>       the client and the server do not implement the same protocol.

At last :-)

>     * Add a server command for allocating blocks of oids.

Remote performance vastly improved, when commiting new objects.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea@argo.es http://www.argo.es/~jcea/ _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea@jabber.org         _/_/    _/_/          _/_/_/_/_/
                               _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBRRLdQJlgi5GaxT1NAQKWEwP/aK00QI4T2JYoLHVoOZ7JcegxYPuQPLsD
oz+kLcGrghxr0i+F/z/iUW9l9RxlAU5997e22UvZuvccmZl8v0AO3Hs7gzIpzpuL
eCmL3HP3uo0RiFnbS9Q8BzZ8pR2CBXsoDHzhLQ838GALphtomSxgLDlfoviuLCfQ
HL1x9x1GwXY=
=0fyD
-----END PGP SIGNATURE-----