Why is writing files witht syscall.O_DIRECT flag make writing files slower in go?-CodePudding

I've got a small peice of code named test.go. It counts time(ns) when doing two writings that write a same byte slice to 2 files, one with the flag syscall.O_DIRECT and the other not.

The code is below:

package main;

import (
        "os"
        "time"
        "fmt"
        "strconv"
        "bytes"
        "syscall"
        // "os/exec"
)
func main() {
        num, _ := strconv.Atoi(os.Args[1]);
        writeContent:= bytes.Repeat( ([]byte)("1"), num );
        t1:= time.Now().UnixNano();
        fd1, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_DIRECT | syscall.O_TRUNC, 0);
        syscall.Write(fd1, writeContent);
        if err != nil {panic(err);}
        t2:= time.Now().UnixNano();
        fmt.Println("sysW1:", t2-t1);

        t1= time.Now().UnixNano();
        fd2, err := syscall.Open("abc.txt", syscall.O_WRONLY | syscall.O_TRUNC, 0);
        syscall.Write(fd2, writeContent);
        if err != nil {panic(err);}
        t2= time.Now().UnixNano();
        fmt.Println("sysW2:", t2-t1);
}

The program is runned in linux command line like this:(after being compiled with go build ./test.go)

./test 1024

I had expected writing file with syscall.O_DIRECT flag to be faster, but the result showed that writing files with syscall.O_DIRECT flag was about 30 times slower than writing without it :(

result:

sysW1: 1107377
sysW2: 37155

Why? I tought writing with syscall.O_DIRECT does less copying and would be faster, but it now turns out to be much slower. Please help me explain it :(

PX: I will not provide playground link since the result is always 0 when running the program on the playground in some reason.

CodePudding user response：

O_DIRECT doesn't do what you think. While it does less memory copying (since it doesn't copy to the cache before copying to the device driver), that doesn't give you a performance boost.

The filesystem cache ensures that the system call can return early before the data is written to the device, and buffer data to send data in larger chunks.

With O_DIRECT, the system call waits until the data is completely transferred to the device.

From the man page for the open call:

O_DIRECT (since Linux 2.4.10)

Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user-space buffers. The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred.

You don't need to manually release the cache after using it. The cache is considered free available memory by the Linux kernel. If a process needs memory that is occupied by the cache, the kernel will flush/release the cache at that point. The cache doesn't "use up" memory.