Add element to list using multiprocessing python-CodePudding

I have defined a function that takes single integer input and will return the output integer.

def get_output(n):
  output = # process the integer
  return output

now I have defined an input list that has to be processed using the function defined above.

input_list = [1,2,3,5,6,8,5,5,8,6,5,2,5,2,5,4,5,2]

Now I have defined an empty output list that will store the output from the function.

output_list = []

Now I want to go through every single item of input_list and append it to the output_list. I know how to achieve this using a sequential manner but I want to know how to parallelize this task. Thanks in advance.

CodePudding user response：

IIUC you need:

If your integer process is more IO bound, threads might work better.

Threads are more IO intensive, therefore if that's what you need you could try:

from concurrent.futures import ThreadPoolExecutor
def get_output(n):
    output = n ** 2
    return output

input_list = [1,2,3,5,6,8,5,5,8,6,5,2,5,2,5,4,5,2]
output_list = []

if __name__ == '__main__':
    with ThreadPoolExecutor(max_workers=6) as pool:
        output_list.extend(pool.map(get_output, input_list))
        
print(output_list)

This processes the list and squares all the elements, it applies this to every 6 elements parallelly, as you can see I specified max_workers=6.

If your integer process is more CPU bound, go with multiprocessing.

With the virtually same code:

from concurrent.futures import ProcessPoolExecutor
def get_output(n):
    output = n ** 2
    return output

input_list = [1,2,3,5,6,8,5,5,8,6,5,2,5,2,5,4,5,2]
output_list = []

if __name__ == '__main__':
    with ProcessPoolExecutor(max_workers=6) as pool:
        output_list.extend(pool.map(get_output, input_list))
        
print(output_list)

This does the same, it processes and squares all elements for every 6 elements parellelly.

Both codes output:

[1, 4, 9, 25, 36, 64, 25, 25, 64, 36, 25, 4, 25, 4, 25, 16, 25, 4]

CodePudding user response：

Append the output into the 'input_list' first.

output_list = numpy.zeros_like(input_list)
output_list  = input_list

Array programming is happening here.