How can I run the body of the for loop parallely in bash-CodePudding

I have 4 commands that are all dependent on each other within the body of a nested for loop (Hence they should be executed sequentially). I would like to speed up the execution time, so I was thinking to run those 4 commands parallely with the next 4 commands and so on. I have read that the & and wait should be used in this case. However, I am not exactly sure where to place them since placing them after each of the 4 command won't speed up anything as far as I understand.

I have around 10 cores to use.

for p in {50,60,70,80,90,100,200,300}
do      
  for min_p_len in {1,2,3,4,5}
  do      
    for min_s_rem in {1,2,3}
    do 
      for length_threshold in {2,3,4}
      do
        command 1
        command 2
        command 3
        command 4
      done
    done    
  done         
done

CodePudding user response：

Replace your inner loop with:

for length_threshold in {2,3,4}
do
  {
    command 1
    command 2
    command 3
    command 4
  } &
done

This may push your system to its performance limits.

CodePudding user response：

To better manage backgroud processes, use xargs. Pass the state to xargs with stdin.

for p in {50,60,70,80,90,100,200,300}
do      
  for min_p_len in {1,2,3,4,5}
  do      
    for min_s_rem in {1,2,3}
    do 
      for length_threshold in {2,3,4}
      do 
        # output variables on one line properly qouted
        declare -p p min_p_len min_s_rem length_threshold | base64 -w0
      done
    done    
  done         
done |
{
   work() {
    eval "$(base64 -d <<<"$1")"   # read the variables from arguments
    command 1
    command 2
    command 3
    command 4
  }
  export -f work
  xargs -d '\n' -P $(nproc) -n1 bash -c '"$@"' -- work
}