Run commands in parallel in bash script-CodePudding

I have a script.sh file something like this:

python3 utility.py query-1.json 
python3 utility.py query-2.json
python3 utility.py query-3.json
.
.

I am currently using sh script.sh to run the script, this however runs each command in a sequential order. I am new to the concepts of parallelism, I would really appreciate any pointers to study more about it in this context, also is there any bash way of achieving multiprocessing or parallelism here ?

CodePudding user response：

You could ask each statement to run in the background which could achieve what you want:

python3 utility.py query-1.json&
python3 utility.py query-2.json&
python3 utility.py query-3.json&
...
wait

The wait command makes the script run in the foreground until all background tasks have completed.

If you want more control, for example run one statement per CPU core, you could take a look at GNU parallel:

cat script.sh | parallel