Is a flatfile database a terrible idea?
As I talked about in my previous Gemlog, the idea of a database being built on flatfiles intrigues me.
A history lesson
Flatfile databases have been around since the days of the terminals and tapedecks, we of course quickly went away from it, once we had relational databases, with indexes, caching in ram and other fancy hacks to deliver data at lightning speed.
In my opinion, we mainly moved to relational databases, to escape the limitations of spinning rust (mechanical hard drives), but as time went on, we got solid state storage and later solid state storage connected directly to the pc backbone, hard drive speeds isn't exactly a limitation anymore.
Why do I care?
It brings us back around to flatfile databases, a lot of people use them every day through Excel, which in spite of all it's bells and whistles is still just a flatfile database under the hood.
So if people still use it, the bigger question becomes, could it be better than relational databases?
It's not better than relational databases... is it?
Well yes and no, it most likely will never be able to live up to the speed at which a ram cached relational database can deliver relational data, but could it deliver the same data in an object/document format faster?
The answer is, I honestly have no idea, but I want to find out. A similar idea is MongoDB, built on an object/document format, but still retaining the database engine part.
If we have MongoDB, why would we want a flatfile version?
See, now we are cooking with Gasoline! I mentioned hard drives and these hard drives have filesystems on them, which most people will never even know or hear about. But what I find intriguing is filesystems like ZFS and BtrFS, that are built for handling extremely large file counts, retaining versioning, making backups easy and even having error correction.
These features are something databases have to implement themselves, which they in most cases haven't or it's a paid feature or it is a pain in the rear to deal with. Plus trying to add these filesystems on top of a database file, will not give you these abilities, it will give you a corrupted database.
By now, I have already ranted for far too long and I hope you get the general gist of what I am getting at, so we shall continue this idea in another episode!
Created 2024-09-30 - Updated 2024-10-09
Source