How to make write operations to a file faster-CodePudding

I am trying to write a large amount of data to a file but it takes quite some time. I have tried 2 solutions but they both take same amount of time. Here are the solutions I have tried;

Solution A:

f, err := os.Create("file.txt")
if err != nil {
    log.Fatal(err)
}
defer f.Close()
w := bufio.NewWriter(f)
for _, d := range data {
  bb, err := w.WriteString(fmt.Sprint(d   "\n"))
  if err != nil {
      fmt.Println(err)
  }
}
err = w.Flush()
if err != nil {
    log.Fatal(err)
}

Solution B:

e, err := os.OpenFile(filePath, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
if err != nil {
    panic(err)
}
defer e.Close()
for _, d := range data {
    _, err = e.WriteString(d)
    err = e.Sync()
    if err != nil {
        return err
    }
}

Any other suggestion on how I can make this write operation faster?

CodePudding user response：

I think bufio is your friend, as it can help to reduce the number of sys calls required to write the data to disk. You are already using it as part of solution A, however note the default buffer size is 4K. If you want to try larger buffer sizes you can use NewWriterSize() to create a writer with a larger buffer.

See https://pkg.go.dev/bufio#NewWriterSize

CodePudding user response：

Combine all your data into a single string, and write that in one operation. This will avoid the overhead of filesystem calls.