> -----Original Message-----
> From: quixote-users-bounces@mems-exchange.org
> [mailto:quixote-users-bounces@mems-exchange.org]On Behalf Of Graham
> Fawcett
> Sent: Friday, July 18, 2003 12:33 PM
> To: quixote-users@mems-exchange.org
> Subject: [Quixote-users] Re: How do you handle long running requests?
>
>
...
>
> When you say 'multi-process', do you mean running one or more
> Quixote/Medusa processes (I think you mentioned Medusa as your
> front-end)? Or do you mean calling external processes to do the heavy
> lifting?

Actually, I've been using FastCGI (configured in Dynamic mode), and meant
multiple processes in that sense (Apache spawns processes of the Quixote
driver script as it needs them).  That keeps development very simple,
because I don't have to be aware of what other processes are doing, provided
I don't do anything that precludes multiple processes sharing a resource
(such as using the shelve module to implement session persistance).

> The latter seems cleaner, IMO. Here's one way to do it: create a pool of
> processes, each connected to your DB and ready to go; dispatch the query
> to the process pool, returning a JobStatus object to your app. Jobs are
> submitted to processes via a Queue, and their state is updated when the
> job is taken and again when completed.
>
> Upon queuing the job, redirect the end-user (immediately) to a URL
> representing the JobStatus object. End-user can refresh until the job is
> ready; in the meantime, he gets just gets a status report. When it is
> ready, your Web app fetches data from the job-process, renders it, and
> the job-process is put back in the pool for a subsequent request.
> Results could be piped back between processes, or issued via shared
> memory, or what have you.

I really like this idea, and hadn't considered it in that light (though I
had considered something kinda along those lines - maintaining a thread pool
for request handlers).  This does add some complexity to the UI layer of the
app, as it has to handle dispatching jobs, keeping track of what job is
associated with which browser, etc (and near as I could tell, would produce
some un-bookmarkable URLs, unless you rely on the user's session to track
that info - another no-no, IMO), but considering Mark's idea about only
doing that on the SELECTS that warrant it, and putting it in the db (or app)
layer, I guess it's really not that bad.  It just has to track whether the
request is an initial request or a follow-up, and if it's a follow-up,
checks the status of the job, and either returns the results (report, etc)
or returns a "processing, please wait" page...  I guess I could keep the
other parts of the UI simple, and just process them using the master
process.

I suppose that same could be done with threads, but then thread safety
becomes a concern again...


> You get to keep a single-process Web app, so session persistence is a
> no-brainer, yet you can scale and distribute to your heart's content.
>
> Or, instead of a process pool, you could also use a tuple space,
> connecting JobProcessor agents to the space which are hungry for new
> jobs to complete. It could make your concurrency issues much easier to
> implement. (I guess you could use a ZODB instead of a tuple space, and
> use ZEO to connect each of your agents to it; I'm not experienced with
> ZEO so I don't know what the overhead would be like.)
>
> -- Graham

I found this link on tuple spaces (http://www.infolets.com/1006361585/) but
must confess I don't see how to implement it in python, *without* using
something like ZODB, or BSDDB, or the filesystem, etc...

Thanks for the input!

Jason