I have two types of files in go which can be represented by the following strings:

const nonewline := 'hello' # content but no newline
const newline := `hello\nworld' # content with newline

My goal is just to read all the content from both files (it's coming in via a stream, so I cannot use something built in like ReadAll, i'm using stdioPipe) and include newlines where they appear.

I'm using Scanner but it APPEARS that there's no way to tell if the line terminates with a newline, and if I use Scanner.Text() it auto-splits (making it impossible to tell if a line ends in a newline, or the line just terminated at the end of the file).

I've also looked at writing a custom Split function, but isn't that overkill? I just need to split on some fixed length (I assume the default buffer size - 4096), or whatever is left in the file, whichever is shorter.

I've also looking at Scanner.Split(bufio.ScanBytes) but is there a speed up by chunking the read?

Anyhow, this seems like something that should be really straightforward.

CodePudding user response：

Use this loop to read a stream in fixed size chunks:

chunk := make([]byte, size) // Size is the chunk size.
for {
    n, err := io.ReadFull(stream, chunk)
    if n > 0 {
        // Do something with the chunk of data.
        process(chunk[:n])
    }
    if err != nil {
        break
    }
}