durusmail: quixote-users: scgi and __del__ cleanup in publisher
scgi and __del__ cleanup in publisher
2004-03-26
2004-03-27
2004-03-29
scgi and __del__ cleanup in publisher
David Hess
2004-03-26
Hi! I'm new to Quixote + scgi and really liking it. I've been building up the
infrastructure for an app based on Apache + SCGI + Python + Quixote + MySQLdb.
However, I ran into something with scgi that I'd like to ask the list about.

As I put the pieces together, I started seeing this in the scgi server log:

---
...
Exception exceptions.AttributeError: "'NoneType' object has no attribute
'close'" in > ignored
...
---

This would happen each time I HUP'd the scgi server to reload my apps Python
code. I followed the Quixote cookbook recipe for managing global publisher
resources (http://www.quixote.ca/qx/ReusingConnections) and added a __del__()
handler to my publisher sub-class with calls to MySQLdb cleanup methods and the
problem got worse! This message started showing up in addition to the previous
one.

---
Exception exceptions.AttributeError: "'NoneType' object has no attribute
'close'" in > ignored
---

After lot's of Google'ing and the judicious use of -d, -v, and -u Python command
line switches, I determined that the problem appears to be that the __del__
method is being called indirectly as a result of Python shutting down, and not
as a result of controlled scgi connection termination. I.e. at scgi_server.py:56

            try:
                os.write(self.parent_fd, "1") # indicates that child is ready
                fd = passfd.recvfd(self.parent_fd)
            except (IOError, OSError):
                # parent probably exited  (EPIPE comes thru as OSError)
                raise SystemExit

What ends up happening is Python begins doing all of its normal cleanup while
shutting down. Whether by design or due to some other problem, the MySQLdb
module gets cleaned out before the quixote module does. Whatever is happening,
something necessary is missing when the delete of the MySQLdb cursor is
attempted and those exceptions occur.

To me, this seems to be a weakness in the scgi server. The resource cleanup in
the __del__() of the cookbook recipe should be done directly as a result of
connection termination and not as an indirect result of Python exiting. So, I
modified the code above as follows:

            try:
                os.write(self.parent_fd, "1") # indicates that child is ready
                fd = passfd.recvfd(self.parent_fd)
            except (IOError, OSError):
                # parent probably exited  (EPIPE comes thru as OSError
                self.shutdown()
                raise SystemExit

I added a shutdown method to the Handler class that chains over to a shutdown
method in the Publisher which does what was in the __del__ method. All of my
ignored exceptions went away. :-)

This seems a more appropriate and controlled way to handle resource cleanup.
Getting exceptions unexpectedly as was happening before means my __del__ handler
wasn't properly executing and I could have ended up leaking other resources in
the cleanup code after that exception (temp files and the like).

Dave


reply