How to pass list and variable to the thread pool?-CodePudding

How should I pass multiple arguments specifically list and string variable to the threading pool:

def activate_item (list, object_id):
      do smth


thread_pool = ThreadPool(parallelism)

with open('examples/1000_PSOrthoTile_ids.txt') as f:
    item_ids = f.read().splitlines()[:400] # only grab 100

thread_pool.map(activate_item, item_ids)

How can I pass arguments to the Pool? I have reviewed several sources, but all mention both arguments as a list.

CodePudding user response：

If you have to send two values to function then you have to create list with pairs -

[ [data, id1], [data, id2], ....]

and use this list in map() or rather in starmap()

Minimal working code:

from multiprocessing.pool import ThreadPool

def activate_item(var1, var2):
    # create one string with '\n' to display it without strings from other theads
    print(f'var1: {var1} | var2: {var2}\n', end='')

pool = ThreadPool()

ids = [1,2,3,4,5]

data = [ (ids, x) for x in ids ] 

results = pool.starmap(activate_item, data)

Result:

var1: [1, 2, 3, 4, 5] | var2: 1
var1: [1, 2, 3, 4, 5] | var2: 2
var1: [1, 2, 3, 4, 5] | var2: 3
var1: [1, 2, 3, 4, 5] | var2: 4
var1: [1, 2, 3, 4, 5] | var2: 5

And if you use map() then you get all values as single list and you have to assign them to variables

def activate_item(all_vars):
    var1, var2 = all_vars

from multiprocessing.pool import ThreadPool

def activate_item(all_vars):
    var1, var2 = all_vars
    # create one string with '\n' to display it without strings from other theads
    print(f'var1: {var1} | var2: {var2}\n', end='')

pool = ThreadPool()

ids = [1,2,3,4,5]

data = [ (ids, x) for x in ids ] 

results = pool.map(activate_item, data)

CodePudding user response：

I assume you are trying to pass to your worker function, activate_items, two arguments: the complete item_ids list and an individual element from that list. If so, use functools.partial:

from functools import partial

def activate_item (list, object_id):
      do smth


thread_pool = ThreadPool(parallelism)

with open('examples/1000_PSOrthoTile_ids.txt') as f:
    item_ids = f.read().splitlines()[:400] # only grab 100

thread_pool.map(partial(activate_item, item_ids), item_ids)