I am wondering how to wait for any process to finish in macOS, since wait -n doesn't work. I have a script doing several things, and in some point it will enter a loop calling another script to the background to exploit some parallelism, but not more than X times since it wouldn't be efficient. Thus, I need to wait for any child process to finish before creating new processes.

I have seen this question but it doesn't answer the "any" part, it just says how to wait to a specific process to finish.

I've thought of either storing all PIDs and actively checking if they're still running with ps, but it's very slapdash and resource consuming. I also thought about upgrading bash to a newer version (if that's ever possible in macOS without breaking how bash already works), but I would be very disappointed if there was no other way to actually wait for any process to finish, it's such a basic feature... Any ideas?

A basic version of my code would look like this:

for vid_file in $VID_FILES
do
    my_script.sh $vid_file other_args &
    ((TOTAL_PROCESSES=TOTAL_PROCESSES 1))
    if [ $TOTAL_PROCESSES -ge $MAX_PROCESS ]; then
        wait -n
        ((TOTAL_PROCESSES=TOTAL_PROCESSES-1))
    fi
done

My neither elegant nor performant approach to substitute the wait -n:

NUM_PROCC=$MAX_PROCESS
while [ $NUM_PROCC -ge $MAX_PROCESS ]
do
    sleep 5
    NUM_PROCC=$(ps | grep "my_script.sh"| wc -l | tr -d " \t")
    # grep command will count as one so we need to remove it
    ((NUM_PROCC=NUM_PROCC-1))
done

PS: This question could be closed and merged with the one I mentioned above. I've just created this new one because stackoverflow wouldn't let me comment or ask...

PS2: I do understand that my objective could be achieved by other means. If you don't have an answer for the specific question itself but rather a workaround, please let other people answer the question about "waiting any" since it would be very useful for me/everyone in the future as well. I will of course welcome and be thankful for the workaround too!

Thank you in advance!

CodePudding user response：

It seems like you just want to limit the number of processes that are running at the same time. Here's a rudimentary way to do it with bash <= 4.2:

#!/bin/bash

MAX_PROCESS=2
INPUT_PATH=/somewhere

for vid_file in "$INPUT_PATH"/*
do   
    while [[ "$(jobs -pr | wc -l)" -ge "$MAX_PROCESS" ]]; do sleep 1; done
    my_script.sh "$vid_file" other_args &     
done
wait

---

Here's the bash >= 4.3 version:

#!/bin/bash

MAX_PROCESS=2
INPUT_PATH=/somewhere

for vid_file in "$INPUT_PATH"/*
do   
    [[ "$(jobs -pr | wc -l)" -ge "$MAX_PROCESS" ]] && wait -n
    my_script.sh "$vid_file" other_args &     
done
wait

CodePudding user response：

GNU make has parallelization capabilities and the following Makefile should work even with the very old make 3.81 that comes with macOS. Replace the 4 leading spaces before my_script.sh by a tab and store this in a file named Makefile:

.PHONY: all $(VID_FILES)
all: $(VID_FILES)

$(VID_FILES):
    my_script.sh "$@" other_args

And then to run 8 jobs max in parallel:

$ make -j8 VID_FILES="$VID_FILES"

Make can do even better: avoid redoing things that have already been done:

TARGETS := $(patsubst %,.%.done,$(VID_FILES))
.PHONY: all clean
all: $(TARGETS)

$(TARGETS): .%.done: %
    my_script.sh "$<" other_args
    touch "$@"

clean:
    rm -f $(TARGETS)

With this last version an empty tag file .foo.done is created for each processed video foo. If, later, you re-run make and video foo did not change, it will not be re-processed. Type make clean to delete all tag files. Do not forget to replace the leading spaces by a tab.

CodePudding user response：

Suggestion 1: Completion indicator file

Suggesting to add task completion indication file to your my_script.sh

Like this:

 echo "$0.$(date  %F_%T).done" >> my_script.sh

And in your deployment script test if the completion indicator file exist.

rm "my_script.sh.*.done"
my_script.sh "$vid_file" other_args &     
while [[ ! -e "my_script.sh.*.done" ]]; do
    sleep 5
done

Don't forget to clean up the completion indicator files.

Advantages for this approach:

1. Simple
2. Supported in all shells
3. Retain a history/audit trail on completion

Disadvantages for this approach:

1. Requires modification to original script my_script.sh
2. Requires cleanup.
3. Using loop

Suggestion 2: Using wait command with pgrep command

Suggesting to learn more about wait command here.

Suggesting to learn more about pgrep command here.

my_script.sh "$vid_file" other_args &     
wait $(pgrep -f "my_script.sh $vid_file")

Advantages for this approach:

1. Simple
2. Readable

Disadvantages for this approach:

1. Multiple users using same command same time
2. wait command is specific to Linux bash maybe in other shells as well. Check your current support.