Are delete events considered close_write in inotifywait?-CodePudding

I have a simple inotifywait script that watches for FTP file uploads to be closed and then moving them to a aws s3. It seems to be working except that in the inotify logs, it indicates that the file was not found ( although the file was indeed uploaded to s3 ). The s3 move command moves the file to the cloud and deletes it locally. Could this be because inotifywait detects deleting the file as a close_write event ? Why is inotify seems to be executing the commands twice ?

TARGET=/home/*/ftp/files

inotifywait -m -r -e close_write $TARGET |
    while read directory action file
    do
        if [[ "$file" =~ .*mp4$ ]]
        then
            echo COPY PATH IS "$directory$file"
            aws s3 mv "$directory$file" s3://bucket
        fi
    done

example logs:

Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
COPY PATH IS /home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4
COPY PATH IS /home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4
COPY PATH IS /home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4
move: ../user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4 to s3://bucket/user-cam-1_00_20220516114055.mp4
upload: ../user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4 to s3://bucket/user-cam-1_00_20220516114055.mp4
move failed: ../user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4 to s3://bucket/user-cam-1_00_20220516114055.mp4 [Errno 2] No such file or directory: '/home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4'
rm: cannot remove '/home/user/ftp/files/2022/05/16/user-cam-1_00_20220516114055.mp4': No such file or directory

CodePudding user response：

Cleaned-up your script and added some safety with quotes and check for already processed file in case the filesystem triggers duplicate events for same file.

#!/usr/bin/env bash

# Prevents expanding pattern without matches
shopt -s nullglob

# Expands pattern into an array
target=(/home/*/ftp/files/)

# Creates temporary directory and cleanup trap
declare -- tmpdir=
if tmpdir=$(mktemp -d); then
  trap 'rm -fr -- "$tmpdir"' EXIT INT
else
  # or exit error if it fails
  exit 1
fi

# In case no target matches, exit error
[ "${#target[@]}" -gt 0 ] || exit 1

s3move() {
  local -- p=$1
  local -- tmp="$tmpdir/$p"
  printf 'Copy path is: %s\n' "$p"
  # Moves the file to temporary dir
  # so it is away from inotify watch dir ASAP
  mv -- "$p" "$tmp"

  # Then perform the slow remote copy to s3 bucket
  # Remove the echo onces it is ok
  echo aws s3 mv "$p" s3://bucket
  
  # File has been copied to s3, tmp file no longer needed
  rm -f -- "$tmp"
}

while read -r -d '' p; do
  # Skip if file does not exist, as it has already been moved away
  # case of a duplicate event for already processed file
  [ -e "$p" ] || continue
  s3move "$p"
done < <(
  # Good practice to spell long option names in a script
  # --format will print null-delimited full file path
  inotifywait \
    --monitor \
    --recursive \
    --event close_write \
    --includei '.*\.mp4$' \
    --format '%w%f%0' \
    "${target[@]}" 2>/dev/null
)