Read from "edge" of file written by another process (which used fallocate)-CodePudding

I have two processes: a "writer" and a "reader"

The writer uses fallocate(2) to first create a fixed length file (filled with zeros) and the writer writes data to the file.

I want the reader process to read the file up to the "edge" (file offset where the writer last wrote data) and then wait for the writer to add more data.

Here are simple example programs for the writer and reader. Note the reader reads just a little bit faster so at some point will hit the edge of the file. When it does, it should seek back one data element and try again until it reads a non-zero data element.

It doesn't work. Seems the reader can seek and read successfully only once. After that it always reads zeros from the current file position. (see log below)

But... if I use another terminal window to copy the file somewhere else, all of the expected data is present. So it seems the data is being written but it is not read back reliably.

Any suggestions?

EDIT / SOLUTION

John Bollinger's suggestion solved the problem:

Adding this line after fopen and before starting to read:

setvbuf(fp, NULL, _IONBF, 0);

WRITER PROCESS

/*
  g   fWriter.cpp -o fWriter -g -pg
*/

#include <stdio.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>

int
main(int argc, char **argv)
{
    printf("WRITER LAUNCHED\n");

    const int N = 100;

    FILE *fp = fopen("/tmp/someFile", "w");
    if (!fp)
    {
        perror("fopen");
        exit(1);
    }

    int   itemLen = sizeof(uint32_t);
    off_t     len = itemLen * N;

    int fd = fileno(fp);
    if (fallocate(fd, 0, 0, len) == -1)
    {
        fprintf(stderr, "fallocate failed: %s\n", strerror(errno));
        exit(1);
    }

    for (uint32_t i = 1; i <= N; i  )
    {
        int numBytes = fwrite((void*)&i, 1, itemLen, fp);
        if (numBytes == itemLen)
            printf("[%u] Wrote %u\n", i, i);
        else
            printf("Write error\n");

        fflush(fp);

        sleep(1);
    }

    fclose(fp);

    printf("WRITER COMPLETE\n");
}

READER PROCESS

/*
  g   fReader.cpp -o fReader -g -pg
*/

#include <stdio.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <unistd.h>

int
main(int argc, char **argv)
{
    printf("READER LAUNCHED\n");

    FILE *fp = fopen("/tmp/someFile", "r");
    if (!fp)
    {
        perror("fopen");
        exit(1);
    }

    
    // New!
    setvbuf(fp, NULL, _IONBF, 0);

    int   i = 1;
    int   itemLen = sizeof(uint32_t);

    while (!feof(fp))
    {
        uint32_t val;

        int numBytes = fread((void*)&val, 1, itemLen, fp);

        if (numBytes != itemLen)
        {
            printf("Short read\n");
            continue;
        }

        printf("[%d] Read %u\n", i, val);

        // wait for data
        if (val == 0)
        {
            off_t curPos = ftello(fp);
            off_t newPos = curPos - itemLen;
            printf("Seek and try again. newPos %d\n", newPos);
            fseeko(fp, newPos, SEEK_SET);
        }

        i  ;

        //usleep(1100000);
        usleep(900000);
    }

    fclose(fp);
}

READER LOG

READER LAUNCHED
[1] Read 1
[2] Read 2
[3] Read 3
[4] Read 4
[5] Read 5
[6] Read 6
[7] Read 0
Seek and try again. newPos 24
[8] Read 7
[9] Read 8
[10] Read 9
[11] Read 10
[12] Read 11
[13] Read 0
Seek and try again. newPos 44
[14] Read 0
Seek and try again. newPos 44
[15] Read 0
Seek and try again. newPos 44
[16] Read 0
Seek and try again. newPos 44
[17] Read 0
Seek and try again. newPos 44
[18] Read 0
Seek and try again. newPos 44
[19] Read 0
Seek and try again. newPos 44
[20] Read 0
Seek and try again. newPos 44
[21] Read 0
Seek and try again. newPos 44
[22] Read 0
Seek and try again. newPos 44

CodePudding user response：

Streams (anything represented by a stdio.h FILE object) can be fully buffered, line buffered, or unbuffered. Streams connected to a regular file are fully buffered by default. This is surely why your reader, having once read a bunch of zeroes at the current file position, doesn't notice that the writer has overwritten some of those. The reader will not consult the underlying file again for data from the region of the file that is already buffered.

Under the circumstances, the reader can and should use setvbuf() to change the buffering mode of the file to unbuffered.