Jon, I hope you don't mind, but I am copying your question to the list.
There's a lot of expertise there, and they may have different answers to
your problem.

Jon Dyte wrote:

>Hi Graham, Jason
>
>I notice you are the two latest 'maintainers' ;-) of the twisted_http.py
>
>
>Are either of you using this with twisted's adbapi module as well.
>Basically I'm trying to execute queries via the adbapi module and then
>format the results for _q_index etc using the callbacks
>However, the queries are issued using this 'deferred' stuff in twisted
>which return immediately.
>
>I looks to me like I would have to make _q_index (or whatever other quixote
>method) block until the callbacks have fired.
>This doesnt sound quite right to me, so I'm just wondering whether either
>of you have any more general solutions?
>
>so in short something like
>
>def _q_index(request):
>       ## goes off and executes dbpool.runQuery and formats the results in the
>callbacks
>       getAndFormatRequest(request)
>       ## this will now return immediately which is a problem....
>
>
>
>I've only really tinkered with quixote/twisted.Right now I'm wishing that I
>had used scgi, which I understood more fully.
>
>Hoping this makes some sort of sense,
>
>Cheers
>
>Jon
>
>

Jason Sibre responded to Jon's question:

>Sorry Jon,
>I haven't tried using the adbapi module for twisted.  I have written
>something for quixote to facilitate long running requests under Medusa and
>Twisted (spawns another thread to handle the long request, puts the browser
>in a request loop until the response is ready to be served), but it doesn't
>sound like what you need.  (In fact it sounds like a hacked answer to the
>same problem you're trying to answer more elegantly.)
>
>You can see the module in the list at:
>http://mail.mems-exchange.org/pipermail/quixote-users/2004-February/002637.html
>
>

And here's my two cents...  :-)

Sorry, Jon -- I haven't tried this kind of thing either.

Just my opinion, but I would be hesitant to use any approach that
wouldn't work if you had to switch to another adapter (SCGI, for
example). Sometimes you can be certain you'll always be working with a
single adapter, but it's better not to limit yourself unless there's a
significant win by doing so.

Since Quixote doesn't have any notion of a defered request, it seems to
me that you would want to do one of two things:

    * block until the db call had returned. Not too hard to implement
      but it delay processing of other requests until the return of the
      db call.
    * dispatch the db call, hold some kind of object in memory (in
      session, perhaps, or in some kind of "defered pool"), return an
      immediate response to the user ("please wait while we fetch your
      data..."). Use a Refresh directive to have the client reload the
      page every so often; the response handler checks to see whether
      the defered db call is finished, and if so present the data,
      otherwise respond with another "please wait...". If you're
      expecting your db calls to be very long running, this isn't a bad
      idea, since it lets your other responses get handled in short order.

The "block-until-return" approach makes more sense if the db calls are
expected to be relatively brief, of course.

Having said all that, I have sometimes wished for a request.defer()
method in Quixote to accommodate this kind of thing. I'm not sure if
there's a way to implement it that such that it would work sensibly
under the wide range of adapters that Quixote handles. Under CGI, it
would have to be a blocking call; under a threaded model, it could yield
to another active thread; under Twisted/Medusa, it could cache the
request, and run the select() loop again (but you'd need a way to
register the defered request with that loop, in order to get it handled
later; a bit easier with Twisted than Medusa, I think).

In short, blocking until the db call is finished is probably your best
bet. Or, write the defer() method to solve the general problem (and meet
us at the next PyCon so I can buy you a beer. Actually, I'll buy it for
you right away, but I'll have to defer delivery until next year. ;-)

-- Graham