How to redirect output of background bash pipeline in real time-CodePudding

I have a long-running bash script (like a daemon), which may emit output to both stdout and stderr.

I have these requirements:

prepend every output line with a string (e.g. date, time of the log entry and other info)
redirect all output into a log file
output redirection must be real-time (i.e. log file must be filled while process is running)
the process needs to run in background

This is what I've achieved so far:

myscript="script_name"
log_file="/tmp/blabla"

prepend_info() {
    xargs -d '\n' printf "$(date  '%b %d %T') $HOSTNAME $myscript: %s\n"
}

script -q -c "$myscript" /dev/null 2>&1 | prepend_info > "$log_file" 2>&1 &

The problem is that the log file gets filled only after my script has terminated, but I want to see output while it's running instead.

If I remove |& prepend_info it works as expected, but I need that additional info into the log file as well.

It seems like the pipe only gets executed after the first command terminates.

Is there some way to modify the output of a background script and redirect it into a file while it's running?

I need to be as compatible as possible, and I can only use simple bash commands. For example, I cannot use ts because it's not always available and also I don't need only the timestamp but other info as well.

UPDATE: The only solution I found so far (it solves everything, also the issue with date) is the following.

myscript="script_name"
log_file="/tmp/blabla"

exec_script() {
    rm -f "$log_file"
    local out_log=<($myscript 2>&1)
    while read -r line; do
        echo "$(date  '%b %d %T') $HOSTNAME $myscript: $line" >> $log_file
    done < "$out_log"
}

exec_script &

If anyone has a better solution, I'm all ears.

CodePudding user response：

I would like to understand where my snippet is wrong

Remember to check your scripts with shellcheck. $myscript and $log_file are not quoted. Also see https://wiki.bash-hackers.org/scripting/obsolete - you are using function NAME(), just use NAME(), and both |& and >& shouldn't be used.

Commands in pipeline are fully buffered, which in your case flushes the output on the command termination, as the buffer is big enough. Both script_name and tr are fully buffered, which gives a big buffer before your command reaches xargs at all.

tr '\n' '\0' | xargs -0 just sounds very strange. Why change newline for zero to use zero? Just use newline xargs -d '\n' from the start.

Additionally command substitution expands before running xargs, so $(date '%b %d %T') will expand once to the date before xargs has started, not the time the current line has been written.

In Bash, $HOSTNAME is faster than $(hostname).

Moreover, tr output is always fully buffered, as tr program itself uses an algorithm with full buffering. Sooo.. do not use tr.

Is there some way to modify the output of a background script and redirect it into a file while it's running?

There are many, many, many solutions available on the net on how to prefix command output, like https://unix.stackexchange.com/questions/26728/prepending-a-timestamp-to-each-line-of-output-from-a-command , https://unix.stackexchange.com/questions/592215/best-way-to-prefix-output-of-a-command-and-check-its-return-value .

Subjectively, the best is not to use shell, and write a python script that will subprocess.popen the command and implement what you want by reading from file descriptors. That way the performance will be acceptable.

CodePudding user response：

With bash 4.2 you can avoid forking a date command per line of input with printf '%( ... )T' and the global variable SECONDS

#!/bin/bash

myscript="script_name"
log_file="/tmp/blabla"

SECONDS=$(command -p awk 'BEGIN{srand();print srand()}')

{
    "$my_script" 2>&1 |
    while IFS='' read -r line
    do
        printf '%(%b %d %T)T %s %s: %s\n' "$SECONDS" "$HOSTNAME" "$myscript" "$line"
    done > "$log_file" 2>&1
} &