using wget to download log file every 5 mins & detect changes-CodePudding

i am writing a bash script to accomplish the following.

script runs wget every five minutes to download a small log from a static url.
script uses diff to see if there are any new entries made to the log file (new entries are made at the end of log file).
if new log entries are found - extract the new entries to a new file, format them properly, send me an alert, return to #1.
if no new log entries are found, go back to #1.

wget "https://url-to-logs.org" -O new_log
if diff -q new_log old_log; then
echo "no new log entries to send."
else
echo "new log entries found, sending alert."
diff -u new_log old_log > new_entries

#some logic i have to take the output of "new_entries", properly format the text and send the alert.

rm -rf old_log new_entries
cp new_log old_log
rm -rf new_log
fi

there is one additional thing - every night at midnight the server hosting the logs deletes all entries and displays a blank file until new log entries are made for the new day.

i guess i could always run a cron job at midnight to run "rm -rf" and "touch" the old_log file, but curious if an easier way to do this exists.

thanks in advance for any/all input and help.

CodePudding user response：

If your logs are not rotating - i.e. the old log is guaranteed to be the prefix of the new log, you can just use tail to get the new suffix - something like this:

tail -n $(( $(wc -l old_log)   1 )) new_log > new_entries

If there are no new lines in new_log, the new_entries file will be empty, which you can check using stat or some other way.

If your logs are rotating, you should first use grep to check if the last line from the old log exists in the new log, and if not - assume the entire new log is new:


if ! egrep -q "^$(tail -n1 old_log)\$" new_log; then cat new_log > new_entries; fi

CodePudding user response：

If all the lines in your log file are unique then you could use grep:

wget "https://url-to-logs.org" -O new_log || exit 1

if new_entries=$(grep -vxFf old_log new_log)
then
    # format and send alert
    printf '%s\n' "$new_entries"
fi

mv -f new_log old_log