On Friday 18 July 2003 1:33 pm, Graham Fawcett wrote:
> Jason Sibre wrote:
> > I think I'll plan on going the multi-process route.


> Here's one way to do it: create a
> pool of processes, each connected to your DB and ready to go;
> dispatch the query to the process pool, returning a JobStatus
> object to your app. Jobs are submitted to processes via a Queue,
> and their state is updated when the job is taken and again when
> completed.
>
> Upon queuing the job, redirect the end-user (immediately) to a URL
> representing the JobStatus object. End-user can refresh until the
> job is ready; in the meantime, he gets just gets a status report.
> When it is ready, your Web app fetches data from the job-process,
> renders it, and the job-process is put back in the pool for a
> subsequent request. Results could be piped back between processes,
> or issued via shared memory, or what have you.

This is a cool idea.

If your backend doesn't support transactions, and all your data update
operations go fast, you could further refine this so only selects
participate in the pool and one thread is dedicated to operations
that change the data.  In any case, the multi-process stuff, which is
required because your database queries are slow, goes in the database
layer.  seems like a nice fit.

> You get to keep a single-process Web app, so session persistence is
> a no-brainer, yet you can scale and distribute to your heart's
> content.
>
> Or, instead of a process pool, you could also use a tuple space,
> connecting JobProcessor agents to the space which are hungry for
> new jobs to complete. It could make your concurrency issues much
> easier to implement. (I guess you could use a ZODB instead of a
> tuple space, and use ZEO to connect each of your agents to it; I'm
> not experienced with ZEO so I don't know what the overhead would be
> like.)

what's a "tuple space?"  actually, i don't understand what you meant
in the entire first sentance.  care to elaborate?

Regards,

Mark