Bash script: read 30000 records from file and run multiple process in parallel-CodePudding

txt with more than 30000 records. All records are one for line and is an IP like this:

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.0.7
192.168.0.8
192.168.0.9
192.168.0.10

I read each row in a bash script, and I need to run a curl like this:

while IFS= read -r line || [[ -n "$line" ]]; do
    #check_site "$line"
    resp=$(curl -i -m1 http://$line 2>&1)
    echo "$resp" | grep -Eo "$ok" > /dev/null

    if [ $? -ne 0 ]; then
        #echo -e "failed: $line" >> "${logfile}"
        echo -e "Command: curl -i -m1 http://$line 2>&1" >> "${outfile}"
        echo -e "failed: $line:\n\n \"$resp\"\n\n" >> "${outfile}"
        echo "$line" >> "${faillog}"
    fi
done < "${FILE}"

Is there a method to run multiple lines simultaneously in my file to reduce the execution time?

CodePudding user response：

I solved for the multiprocess in this way:

# create function:
check_site() {
  ip=$1
  resp=$(curl -i -m1 http://$ip 2>&1)
  echo "$resp" | grep -Eo "$ok" > /dev/null

  if [ $? -ne 0 ]; then
    echo -e "Command: curl -i -m1 http://$ip 2>&1" >> "${outfile}"
    echo -e "Block failed: $ip:\n\n \"$resp\"\n\n" >> "${outfile}"
    echo "$ip" >> "${faillog}"
  fi
}

# call the function:
export -f check_site
parallel -j 252 -a "${FILE}" check_site

But I have a problem, that now don't enter in the IF and don't write to the log file ${outfile} and ${faillog}.

environment: line 6: : No such file or directory
environment: line 7: : No such file or directory
environment: line 8: : No such file or directory

Any help for this?

CodePudding user response：

This article describe approach to resolve parallel execution, it may help you: Parallel execution in Bash

Example from the article:

#!/bin/bash

RANDOM=10
JOBS_COUNTER=0
MAX_CHILDREN=10
MY_PID=$$

for i in {1..100}
do
    echo Cycle counter: $i
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    while [ $JOBS_COUNTER -ge $MAX_CHILDREN ]
    do
        JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
        echo Jobs counter: $JOBS_COUNTER
        sleep 1
    done
    sleep $(($RANDOM % 30)) &
done
echo Finishing children ...
# wait for children here
while [ $JOBS_COUNTER -gt 1 ]
do
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    echo Jobs counter: $JOBS_COUNTER
    sleep 1
done
echo Done