I have the bash file -

cat input.txt | parallel -j4 'python {}'

cp -r outputs /home/usr/my_outputs

I want to copy outputs to my_outputs only after all jobs finish executing. Presently, it looks like parallel returns immediately (after starting the jobs) and then cp -r is executed immediately, but I want to wait for the jobs to finish executing before I copy. How do I do this? Thanks!

EDIT: input.txt is like -

run1.py -n 5 
run2.py -n 5 
run3.py -n 5 
run4.py -n 5 

CodePudding user response：

Try:

parallel --colsep ' '

otherwise you are effectively trying to run:

python 'run1.py -n 5'

which means the Python interpreter is looking for a script called run1.py -n 5

CodePudding user response：

I've done something similar where I have 4 python scripts running at once and had the same issue, though I was setting them off an pushing them to the background. These scripts wrote to files that were then used later in my bash script. I solved it by using wait $(jobs -rp) -

python script1.py & 
python script2.py & 
python script3.py & 
python script4.py & 
wait $(jobs -rp)

# other bash code that needs to run after the above

This works with gnu parallel as well. Try this:

cat input.txt | parallel -j4 'python {}'
wait $(jobs -rp)
cp -r outputs /home/usr/my_outputs