Re: Is a Flatfile database a terrible idea? and Episode 2
I did not expect to be writing Episode 2 so soon, but I did get my first reply ever, so that is a celebrateable occasion.
Getting a reply
So as mentioned this is my first ever reply, so I want to thank you very much for that, it being constructive criticism just makes it even better. I would also like to thank you for some of the work you did testing and explaining some of the concepts, as that will definitely be a big help.
Episode 2
This is partially an answer, but also more of an explanation of the idea. I figured flatfiles would be a bit more of a lower barrier to entry for the tech uninitiated, as well as requiring a lower footprint and also not database software to configure.
My idea isn't really a traditional flatfile per se, but more focused on object/document storage. I think we need an example for it to make more sense! So lets take a user profile on Station or BBS, what if that user profile was basically just a Gemtext file, lets say we have Lars, Martin and Yvonne.
We'd have a folder with an index.gmi file, looking something along these lines
=> lars.gmi Lars => martin.gmi Martin => yvonne.gmi Yvonne
Then each Gemtext file would contain a user profile kinda like this
# Yvonne ## Bio My name is Yvonne and I love tennis. ## Hash 3A2F4C23E632
may have row-level locking so that a whole table or file need not be locked for a row update
So it becomes a distributed flatfile system so to speak, with human editability/readability in mind. Then whoever writes the code could index the index file or just look up in it whenever needed. The user profiles themselves would be client servable, and likewise a user updating their profile, would only lock their file while writing, so you'd get a similar idea to rows in SQL.
This could be used in HTTP land as well, for JSON/YAML docs served directly to Javascript from the proxy/webserver or even a static website with minimal server code.
tradeoffs as to where the complexity ends up and how much trouble there is when something fails, what about backups
The general idea here being that for larger applications, you would ideally have to use something like ZFS, as it would provide, backup, encryption, caching, load balancing and much more.
Outtro musings
I know Gemtext probably isn't the best idea, because what I just described is basically just a functioning Capsule. 😂
So there are cases where you can get away with something custom
I just felt it would be a fitting example and maybe if there was a spec on how to write "data" documents, this could also be parsed by crawlers or even the capsule itself, to provide different information.
There could also be a template document in the folder, which could be used to bulk update or correct current documents. There are quite a few ideas that could exist, but doesn't yet, I've been in the idea stage for a while now.
Until next time, may the god emperor bless you with music of creation!
Created 2024-10-01 - Updated 2024-10-09
Source