converting thread to multiprocess pool-CodePudding

basically i have a thread code right now but i wish to try to convert to multiprocessing pool for testing the performance

user_dict = {
            'users': [
                {
                    'username': 'test1',
                    'password': 'opop1',
                },
                {
                    'username': 'test2',
                    'password': 'opop2',
                }
            ]
        }

        thread_list = []
        for user in user_dict['users']:

            ct = threading.Thread(
                target=task,
                args=(
                    some_object,
                    some_other_object,
                    another_object,
                    user['username'],
                    user['password']
                )
            )
        ct.start()
        thread_list.append(ct)

        for thread in thread_list:
            thread.join()

something basic, my task function will do some login base on each user's username and password and proceed to assign some information to call an api endpoint to fetch data and write into a json file.

but i have no idea how to convert this into multiprocessing pool as i am not sure how to pass my user_dict and other objects information required for this.

CodePudding user response：

You can create a Pool and feed it with your function and arguments in a list with starmap. With arguments reduced to usr/pwd:

from multiprocessing import Pool

user_dict = {
            'users': [
                {
                    'username': 'test1',
                    'password': 'opop1',
                },
                {
                    'username': 'test2',
                    'password': 'opop2',
                }
            ]
        }

def task(usr, pwd):
    print(usr, pwd, sep=", ")
    return (usr,pwd)

if __name__ == '__main__':
    pool = Pool(processes=len(user_dict['users']))
    results = pool.starmap(task, [tuple(el.values()) for el in user_dict['users']])

results contains all returned values for each pool member:

[('test1', 'opop1'), ('test2', 'opop2')]