running multiple external commands in parallel in Python-CodePudding

I want to run multiple external commands (like opening notepad.exe) at once in parallel through Python. I know how to do it by using the subprocess module. However, I also want to run multiple such commands at once in parallel and ensure that at any time, n commands are running out of a total of m commands until all the m commands are finished running. Below is the example code/ explanation to illustrate this where n = 3 and m = 10

import subprocess

commands = []
batch_size = 3

for i in range(10):
    commands.append('notepad.exe')

def run_in_batches(batch_size):

    # run multiple( = batch_size = 3) commands in parallel through subprocess.Popen
    # command1 = subprocess.Popen(commands[0])
    # command2 = subprocess.Popen(commands[1])
    # command3 = subprocess.Popen(commands[2])
    # command1.wait()
    # command2.wait()
    # command3.wait()
    # if command1.poll() == 0 or commad2.poll() == 0 or command3.poll == 0:
    #   queue the next command in commands
    #   perform this check repeatedly so that 3 commands are running at any time untill all commands are finished running

As you can see, I have been trying to do this with subprocess.Popen but I am not able to code it properly. I am really struggling with the portion wherein I have to ensure that 3 commands are running all the time and if any of them finishes, only then the next command in queue is executed until all the commands have been run successfully. Any help will be highly appreciated, thanks!

CodePudding user response：

you should use a Threadpool for this, with subprocess.run as it blocks the running thread.

import subprocess
from multiprocessing.pool import ThreadPool
commands = []
batch_size = 3

for i in range(10):
    commands.append('notepad.exe')

def run_in_batches(batch_size, commands_to_run):
    with ThreadPool(batch_size) as pool:
        pool.map(subprocess.run, commands_to_run)

run_in_batches(batch_size, commands)