[om-list] FramerD

[Mark Butler]

Luke,

I glanced at the FramerD documentation once before, and had same impression. 
However, on closer examination it looks like a very good match.  

FramerD is not written in LISP, it is written in C.  It just happens to be a
database designed to store semantic networks like the ones we are dealing with
very efficiently.  It has a small Scheme interpreter, but you are not required
to use it.  It is also distributed, which is great for very large database
clusters, like the type Google uses (I read that they use a *4000* node Linux
cluster for their web index). 

FramerD was developed precisely to overcome the objections you raise below, in
particular scalability and interoperability with other programming languages. 
I certainly wouldn't use it to run a bank quite yet, but I can certainly see
it being used in terabyte scale data analysis applications.

You would like an architecture good for five to seven years.  I don't think
any relational database will give it to us.  We have chosen a field that
relational databases are notoriously poorly suited for.  OODMS manufacturers
regularly advertise how their databases help customers obtain performance
10-20 times higher than their previous relational databases on the same
application.

The main reasons for this are: 

1. OODMSs cluster related records together on disk, RDBMSs generally cannot.
2. The SQL parser & query planner are mostly dead weight for simple traversal
operations
3. Typical relational databases are optimized for table scans, which work best
if you have a large number of very specialized tables, not OO style all
purpose tables.  
4. Relational database query protocols require too many network round trips to
perform well with graph structured data.

Of course, in principle, one could re-engineer the internals and network
protocols of a relational database to cluster records efficiently on disk,
implement distributed client record caches, and arrange to transfer records
from different tables in one burst, but no one has done it yet.

I am trying to help a little with Postgres, but it is going to be a long time
before it will replace any decent dedicated OODMS for graph traversal
operations.  Right now they have a lot of work to do to get to where Oracle
was a decade ago, and object oriented features aren't much of a priority.  

I have recent experience implementing a genealogy database in Oracle, with the
freedom to customize the tables for that particular application.  The two core
tables were person and person_event, each with two or three dozen attributes.
To generate a pedigree chart or a family group sheet, we had to run twenty or
thirty SQL queries because of the way the records are related in a graph. 
Performance is okay, but many times slower than the equivalent flat file
application I wrote for a related family history research firm.

Now if we took that same model and split it up so that every attribute value
was in its own record, it is hard for me to imagine that even a single user
could get any work done unless the application was written with a very large
RAM object cache.  Even then, performance would crawl until the current
working set of objects was brought into memory, something that never quite
happens with ad hoc queries.


The only major problem I can see with FramerD is that it does not yet have
transaction support, something that would need to be added before using it for
a mission critical system, and something the lack of which can sometimes be a
major annoyance.  The bright side is, as in the case of MySQL, leaving out
transaction support can add a factor of 5 to update performance.  For many
applications, that is a worthwhile tradeoff - I sure wish I could turn off
transaction logging when loading a million row table in Oracle.


- Mark


Luke Call wrote:
> 
> But how would lisp handle terabytes
> of data, and our requirements? We've been over this in detail, remember?
> I have used lisp (the scheme dialect, taught by one of the leading
> lights at the time) and couldn't seem to find ways to map it to the real
> world, though I asked the professors about it a lot back then (10 yrs
> ago). Like how to do object systems (write your own), persistence (not
> discussed), access to operating system facilities, databases, or other
> necessary utililties (well, didn't know, weren't so interested in that
> anyway), etc. Great for AI and education, they kept saying--so I say we
> may use it for that, as described below--as an embedded language to
> express things within the model that rows/columns can't so well (as
> discussed before), like formulas, algorithms, and actions performed by
> entities in the model, and perhaps more.