Flatfile vs. database

[Edward Glowacki]

Databases seem to be a popular topic these days, and I was wondering at
what point it makes sense to move from flat files to an actual database.  
I'm trying to create a database (or database-like) collection of
documents, and each document has associated keywords, categories, cross
references, and creation/modification information (authors, date written,
etc.).  

In some sense, I guess this will be like an encyclopedia, with some
entries being a few lines, others being large articles (like information
on a whole country), plus images and sounds maybe?  I'd like a fair amount
of cross-referencing between articles, ala hyperlinks probably.  The
viewing format will probably be web-based, but I'll want to edit the
documents with a real editor (i.e. emacs) as opposed to a web textbox.

My current incarnation is to have each document in a custom file format
consisting of a header block like this:

AUTHOR="Ed Glowacki",bob
KEYWORDS=something,"something else",blah
CATEGORIES=stuff,things,"other stuff"
END

followed by the body of the document, which is the rest of the file.
I parse this with perl to create an index for the important header
fields, which I can later search against, for instance to find out which
document belong to a given category.  

I plan on putting a lot of stuff in this "database", and I imagine the
text files I have now will get hard to manage at some point. I need to be
able to access it indefinately, so I don't want some obscure data format
I'll be unable to read in 10 years.  As long as I can do some conversions
along the way, I should be OK, but I don't want the conversion routine to
take 2 weeks of programming. =)

I'm sure I can do a lot of the stuff I want to add using my perl script
and flat text files, but before I spend too much time on it, I'd like to
make sure I'm travelling down a reasonably good path.  Any thoughts?


-- 
Edward Glowacki				glowack2@msu.edu