After looking at ways to go about this, I'm a bit overwhelmed, having never done this before, so here I am. I'm looking into ways to store my data for a personal project. I've never used so much data before, and I usually just stored it in a .txt or .csv, but I'm curious if there would be a better way to go about it. I'm currently looking into json at the moment, as I feel the structure could prove useful in the future (not that important).
In terms of the data, just think of a dictionary: idx | string | string | string | string ...
Would it be worth the investment to store all the information into a DBMS if I just want to grab it by the index to display? I'm more curious about why you'd take one route over the other.
CodePudding user response：
With lightweight options like SQLite the investment per se will be pretty much limited (in as well complexity as extra footprint), and you will be able to offload a whole bunch of concerns to the library: memory management, data retrieval by criterion, etc. With JSON you will either have to keep the entire dataset in RAM (but that doesn't scale plus on each start-stop you will have to load and save it) and you will have to figure out querying all by yourself.
There are other options as well but in my opinion SQLite is very mature, very well known and pretty solid, and if all you need is pure ISAM style access, the SQL you will need to understand is minimal.
CodePudding user response：
There are thousands of already implemented standard solutions.
As already mentioned by others: Connect to a data base with a library of your choice. But this is typically the slowest method as in all libraries I know, all and everything will be converted to text to transfer it to the database core and there sometimes parsed back to native data. This consumes a lot of time and memory.
Use your own data structures and use a common serializer.
Use a library to describe your data structures and the IO facilities for them like
protobuf. There are a lot of implementations.Or simply do it handcrafted... but my advice is: Take a already implemented solution as it saves your time and especially helps to not reinvent the wheel and bugs.