How to tail dynamically created files following a regex-CodePudding

I am trying to tail dynamically created files in bin/bash using command

tail -f /data/logs*.log

But its not tailing the files created at runtime.

For eg if there are already 2 files logs1.log and logs2.log present and after some time if logs3.log is created at runtime.

It is not tailing logs3.log

What is the way to tail such dynamically created files ?

CodePudding user response：

This does not work because the bash wildcard * is resolved only once. It produced the list of files that are already existing. This list is then not updated any more. So you whole line tail -f /data/logs*.log is replaced by something like tail -f /data/logs/logs1.log /data/logs2.log /data/logs3.log. And this command is then executed. Normally you must understand wildcards as a preprocessing before a command is executed.

What you want needs a bit more effort. You command already works for files that already exist. That is good so far. But you need more. So you must send your tail command into the background by adding a &. Try it:

tail -f /data/logs*.log &
sleep 2s
echo something more

But instead of writing "something more" you want to listen for new files and also tail -f them. How to do this you can find here: https://unix.stackexchange.com/questions/24952/script-to-monitor-folder-for-new-files

Over the time you will have more and more processes. Assuming that you normally have only a few new files, this won't be a problem. But in case you have hundreds or thousands of new files you have to spend more effort for your solution.

CodePudding user response：

You can try something like this, but it has some issues (see below):

#!/bin/bash

pid=
dir="$1"

handle_sigint() {
    kill $pid 2>/dev/null
    exit
}

trap handle_sigint SIGINT SIGTERM

while true; do
    tail -n1 -f "$dir"/*.log &
    pid=$!
    inotifywait -q -e create "$dir"
    kill $pid 2>/dev/null
done

Run it by giving the wanted directory as the first parameter.

Sadly, even if you remove the -n1 argument in the tail command, you may miss some log lines, notably if the new files have many lines written directly on creation.