Unix Bash script - How to Limit for loop through files to only process 10 at a time-CodePudding

I am banging my head against a wall and I'm certain I'm just being dense.

I am attempting to run a for loop with a lftp command against files in a watched folder basically a backup and move process. But due to the way LFTP and the loop works if I process files on the destination before everything is done I get a failure.

So what I want to do is read all files with *.log and then if there are say 20 files only process 10, then on its next run process 10 files again eventually it will be at a point of 1 file at a time. I can loop through everything fine. just can not seem to figure out how to only read the first 10 files as I say.

for FILE in *.log; do
lftp -p 2252 -u $FTPUser,$FTPPass $Location <<EOF
set ftp:ssl-allow no
set xfer:use-temp-file on
set xfer:temp-file-name *.tmp
set log:file/xfer /log/LFTP_$FILE.log;
mput $LogPath/$FILE
quit
EOF
   if [ $? == "0" ]; then
        rm $LogPath/$FILE
    else
        echo "Error"
    fi
done;

CodePudding user response：

You only need one lftp connection per batch.

Also, don't use upper case for private variables, and quote your variables; see also Why is testing “$?” to see if a command succeeded or not, an anti-pattern?, and take care to send diagnostic messages to standard error, not standard output.

files=(*.log)
for ((i=0; i<${#files[@]}; i =10)); do
    if { cat <<________EOF
        set ftp:ssl-allow no
        set xfer:use-temp-file on
        set xfer:temp-file-name *.tmp
        set log:file/xfer /log/LFTP_$FILE.log;
________EOF
        printf "mput $LogPath/%s\n" "${files[@]:$i:10}" 
        echo quit; } |
      lftp -p 2252 -u "$FTPUser,$FTPPass" "$Location"
    then
        rm "$LogPath/$FILE"
    else
        echo "$0: Error for ${files[@]:$i:10}" >&2
    fi
done

CodePudding user response：

Using an array:

cd "$LogPath"
files=( *.log )

if lftp -p 2252 -u "$FTPUser,$FTPPass" "$Location" <<EOF
    set ftp:ssl-allow no
    set xfer:use-temp-file on
    set xfer:temp-file-name *.tmp
    set log:file/xfer /log/LFTP_$FILE.log;
    mput ${files[@]:0:10}
    quit
EOF
then
    rm ${files[@]:0:10}
fi

Explanations

${files[@]:0:10}

is a parameter expansion that retrieve only the first 10 elements