Recently, I am working on a project which will have thousands of small files to manage. The whole system will randomly read/write those files. And parts of them are downloaded from networks at the moment that it is needed. For performance and system limitations, I could only keep 10~20 file descriptors/handles. So I decided to create a large single file as my 'disk'. And then dynamic read and write files into it.

Here is my design:

1. A file header for serialization of meta-info
2. An index to save offset of every little file
3. A bunch of 'pre-allocate segments' to save files
4. Support multi-thread accessing

Finally, It looks like a virtual file system. But I'm not sure is it safe to read/write the same file at the same time on Android and iOS. And I'm totally no idea how to manage the fragment of my 'pre-allocate segments'.

Is there any best practice or frameworks that I can reference? Am I on the right way?

CodePudding user response：

I am not an expert on Android/Ios but I think that in your current case the problem will be to use multiple threads on a io operation. My advice if you want to follow down this place would be to add an intermediate memory buffer that could handle the update/deletion and asynchronously write on disk. Another advice would be to try to use an sqlite db (a db with a single file) that is already doing all the heavy work for you.

CodePudding user response：

Put all files in a database. Sqlite for Android. Then they are in one file.