Redirecting s3 cp logs into a single file using while loop and bash redirection is not working

I am trying to save s3 cp command output to a single log file using bash shell script. My script takes the source and destination bucket information form another file called output.tsv which is actually inputs to a while loop. like below. I am using &>> to redirect the output to a file with the timestamp

while read SOURCE DESTINATION
do
    aws s3 cp $SOURCE $DESTINATION &>> /shared_data/logs/logs-$(date  "%b-%d-%Y-%H-%M-%Z").log
done < output.tsv

The s3 cp logs are supposed to be redirect/append to a single log file but What I see that the log files get created every single minute instead of redirecting it to a single file. FYI i am trying to copy bulk amount of files and my scripts runs for an hour.

-rwxrwxr-- 1 user1 group1 33K Oct 28 06:13 logs-Oct-28-2021-06-12-PDT.log

-rwxrwxr-- 1 user1 group1 33K Oct 28 06:14 logs-Oct-28-2021-06-13-PDT.log

-rwxrwxr-- 1 user1 group1 53K Oct 28 06:15 logs-Oct-28-2021-06-14-PDT.log

CodePudding user response：

(I'm not an AWS S3 user but) regarding your use case that just relates to a shell issue, you could just define the filename outside of your while loop:

log="/shared_data/logs/logs-$(date  "%b-%d-%Y-%H-%M-%Z").log"
while read SOURCE DESTINATION; do
    aws s3 cp "$SOURCE" "$DESTINATION" &>> "$log"
done < output.tsv

BTW, note that you might want to change your date format string to use a more standard (ISO-8601 like) encoding:

log="/shared_data/logs/logs-$(date -u  "%Y-%m-%d_%H-%M-%S").log"

The advantages of this date command being:

the fact that lexicographically speaking, the string sorting order matches that of the dates;
and uses the UTC timezone (thanks to option -u):

$ date -u  "%Y-%m-%d_%H-%M-%S"
2021-10-28_16-42-00