[ Rod Senra ]:
> > I wonder if C.J.Date mentioned anything about databases for storing
> > large dense graphs in that book ? I suspect this is an area where OODB
> > would be more suitable than Relational DBs.
[ David Binger ]:
> I'd be interested to hear about the storage design for a database of
> that size.
[ Mario Ruggier]:
> I would also be very intrigued by the design choices of such a storage.
So am I ;o) In fact that problem was proposed by a friend of mine, and
it caught my attention. I'll probably do some research to see how (if) it
has been tackled before. I'll be glad to share here what I eventually find
out about it.
[ Mario Ruggier]:
> One thing I would like to understand better is what characteristics of
> a problem make for an OODB would perform (human + performance) better
> than a relational implementation.
IMHO, the relationship between Zope and ZODB is a great example.
Even though, the recommended way to go is to store Zope applications
in the file system. You *can* make them TTW and store them inside ZODB.
But choosing to store such applications in a Relational DB, even though
feasible, would be a nightmare.
Some possible rules of thumb (far from an exaustive list):
Relational - when you have a huge volume of data, sharing a
small set of stable data schemes.
- when you need ad hoc queries that could be
written as SQL+operators
OODB - when you have a considerable diversity of data schemes,
when you have to index textual content (==Information Retrieval),
when browsing a structure is enough as a query language
(== when you can do without cross products and joins).
But I afraid you knew all of that already ;o)
[ Mario Ruggier]:
> An aspect of this comparison I have always wondered about, but never measured,
> is that correlation of data must be done:
> (c) or initialising rich domain-savvy objects, using an ORB or an OODB
> directly
> Which one of these approaches is:
> - easier to design, implement, maintain, evolve?
> - gives better performance?
> Option (c) begs another question... in addition to the potential
> impedance mismatch problems introduced by the OR mapping layer, what
> gives better performance:
> - SQL query -> creation of rich objects ?
> - "Reviving" persisted rich objects ?
I believe Relational DBs performance is tightly coupled to the Locality
principle, meaning that stuff you may want is placed close to each other
in storage. When this holds, cache efficiency and fetching data from disks
is naturally improved.
I'm not sure the locality principle holds for an object persistence system.
Perhaps only if you could ensure that closely related objects (in behaviour not
in structure) where stored together. That is why I have doubts about the
performance of O-R mapping.
I am no expert though, so take this comments with a pinch of salt.
[Rod Senra]:
> > By the way, I helped another federal project to build a Python desktop
> > app to record info from more than 150.000 public schools. While working
> > off-line, such app used Twisted + **Durus**. But unfortunately, due to the
> > rampant corruption unveiled recently, the project is in the freezer.
> > If it ever gets out I'll let you know.
[ Mario Ruggier]:
> Wow... is there any chance that you could say more about this
> experience? How did Durus hold up? How did you use it?
Sure. The architecture is inspired by Bruce Eckel's article
"Browser as a Desktop UI" [1]. The client wanted a portable
(Windows and Linux) application to record info about teachers,
students and schools in order to better control budget distribution.
The team in charge used a variation of Bruce Eckel's design.
I came later as a consultant, suggesting an alternative implementation
using Twisted Web + Durus + Ajax. There where some **huge** forms
(more than 400 fields, completely insane, but client == Gov so...)
and Durus handled the stress tests just fine. But the Durus version
was not yet released for production use, due to some political battles.
If that changes I'll let you know.
> Little things that help,
Since the forms where huge and ever changing, we used Python reflective
abilities to build the database scheme automatically by parsing the
HTML forms (defined by the client, and modified every week). The scheme
itself was stored inside the database, and used to detect scheme evolution.
> things to avoid?
Clients like this.
best regards,
Ro Senra
[1] http://mindview.net/WebLog/log-0045