durusmail: quixote-users: OT - zodb best practices with quixote
OT - zodb best practices with quixote
2003-01-06
2003-01-07
OT - zodb best practices with quixote
Michael Watkins
2003-01-06
This is a little off topic but since there are ZODB & Quixote users among
the group, I thought this may be interesting for other Quixote users who
may not have looked at ZODB yet.

I'm in the middle of trying to transition my mindset away from relational
database-think into ZODB-think. Its tough for me, I've found.

Initially I spent a bunch of time trying to come to grips with losing SQL
(specifically the query-language). I looked at IndexedCatalog and while its
interesting I'm trying to avoid adding the overhead it imposes unless I
really need it.

The solution I'm building is a news & document publishing system for a
political organization. Most of the objects which Quixote will be
publishing will be Documents or Folders (which contain other Folders and
Documents).

With SQL I'd typically build one or more objects, frequently differing only
by some attribute such as "type", or if I need a high level of
categorization something like subject->category hierarchy. Producing
filtered and ordered lists of such things is trivial in SQL.

i.e.    select * from view_document where status = 'published' order by
date desc
          select * from view_document where type='folder' and parent=334939
          select * from view_document where category='press_release' and
status='published'

          (I wouldn't do this exactly as illustrated, just an example )

But doing any of this is not trivial when using dictionaries

When using ZODB is it a common practice to embed attribute info into a key?
I could imagine something like:

(date-type-counter)
     20030106-MA-7
     20030106-MA-8
     20030107-SD-9

which would allow for easy sorting and filtering to deal with some common
requirements, without having to build separate lists of documents by attribute.

Since I would never do this (embed info in a key) with a SQL database, I'm
finding it a bit odd during this mind-set transition phase.

I expect there to be tens of thousands of documents (or forum messages,
etc) over time and based on some testing I've been doing it seems worth my
time to avoid schemes which will have to resort to brute-force methods such
as grabbing all instances from the zodb in order to check attribute values.

Any comments or suggestions welcome.

Mike

PS: I think the idea floated earlier about an example Quixote-ZODB
application is a very good one.

PSPS: Regarding the thread I started a few days ago re SCGI and ZEO/ZODB,
Neil has resolved the issue and I'll leave him to comment on it since its
OS mumbo jumbo which I have no knowledge of. A single line of code resolves
the blocking issue for FreeBSD, and it doesn't appear that the issue is
present on Linux.

Grab the document contents from ZODB using ClientStorage:

ab -c 2 -n 1000 http://testserver/doc/759/

Document Path:          /doc/759/
Document Length:        19573 bytes
Concurrency Level:      2
Time taken for tests:   1.938 seconds
Complete requests:      1000
Failed requests:        0
Broken pipe errors:     0
Total transferred:      19782000 bytes
HTML transferred:       19573000 bytes
Requests per second:    516.00 [#/sec] (mean)
Time per request:       3.88 [ms] (mean)
Time per request:       1.94 [ms] (mean, across all concurrent requests)
Transfer rate:          10207.43 [Kbytes/sec] received

I can live with that.


reply