Can you read and delete chunk of a file?-CodePudding

I am using ofstream and ifstream to read a chunk from a file post it over middleware (DDS) to another process and the other process writes that chuck of the file.

Basically transferring a file. The two components are unaware of each other and may live on the same hardware or on different hardware (DDS takes care of the transfer either way).

This is all working, however when I try to do this with a large file (>500MB) and if the destination component is on the same board, then I run out of RAM (since 500 x 2 = 1GB which is my limit).

So I am thinking of reading a chunk from a file deleting that chunk of the file and then sending the chunk. So I end up with:

A        B
12345 ->
2345  -> 1
345   -> 12
45    -> 123
5     -> 1234
      -> 12345

Where each number is a chunk of a file.

I am using linux, so I can use any linux APIs directly, but would probably prefer a pure c approach. I can't really see any good options here. i/ostream does not appear to let you do this. Options like sed will (I think) end up using more memory by copying.

Are there any better mechanisms for doing this?

Update

The files are stored in RAM via a tmpfs partition

CodePudding user response：

I am using linux, so I can use any linux APIs directly, but would probably prefer a pure c approach. I can't really see any good options here. i/ostream does not appear to let you do this. Options like sed will (I think) end up using more memory by copying.

Are there any better mechanisms for doing this?

There is no standard mechanism in C or Linux for shortening a file in-place by deleting data from the beginning or middle. Most file systems don't work in a way that would support it. When one wants to delete data from such a position, one has to make a new copy of the file, omitting the data that are to be deleted.

You can shorten a file by removing a tail, but that does not serve your purpose unless possibly if you send chunks in reverse order, from tail to head. However, the most natural ways I can think of to support that in an application such as yours would involve pre-allocating the full-size destination file, and that would have the same problem you are trying to solve.

CodePudding user response：

Can you read ... chunk of a file?

You can seek to an offset to start reading from anywhere of a file, and you can stop reading once you've read the chunk entirely.

Can you ... delete chunk of a file?

Operating systems present files with an interface similar to a vector. Deleting a chunk from the end of a file is trivial like it is trivial to erase elements from the end of a vector. You can simply call std::filesystem::resize_file with the new size which is the chunk size subtracted from the original size.

Deleting a chunk from elsewhere is more complex. You must first seek to the beginning of the chunk, then copy the content from after the chunk into the start of the chunk and forward. The complexity of this operation is linear in distance from the start of the chunk to the end of the file. When you have copied all of the trailing content, you can resize to remove the excess.