multiprocessing cannot create dictionary structure-CodePudding

The problem is very similar to: Adding values in dictionary multiprocessing

I want to use multiprocessing to write a dictionary faster, but for some reason, the multiprocessing function can only write the first layer of the dictionary. This means, the dictionary works totally fine, when using the example above.

But instead of the original output with random numbers as the values:

{1: 169, 2: 520, 3: 637, 4: 559, 5: 497, 6: 470, 7: 113, 8: 221, 9: 946, 100: 69}

I want the function to create a dictionary inside these dictionaries. However, the output I get is an empty dictionary, as if the subdictionary creation dic[user][i] = random.randint(0, 1000) did not happen and as a consequence, I get a Key error in the next line print(dic[user][i]).

{3: {}, 1: {}, 8: {}, 5: {}, 2: {}, 9: {}, 100: {}, 7: {}, 4: {}, 6: {}}

Why is the function not able to create the subdictionary during multiprocessing? Using the function normally does give the correct output.

import multiprocessing
import random

users = [1, 2, 3, 4, 5, 6, 7, 8, 9, 100]
users2 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 100]


def random_number_check(user, dic, list2):
    dic[user] = {}
    print(list2)
    for i in list2:
        dic[user][i] = random.randint(0, 1000)
        print(dic[user][i])


if __name__ == '__main__':
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
jobs = []
for i in range(10):
    # Pass the user at position i instead of the whole list
    p = multiprocessing.Process(target=random_number_check, args=(users[i], return_dict, users2))
    jobs.append(p)
    p.start()

for proc in jobs:
    proc.join()

print(return_dict)

CodePudding user response：

I got it working by creating a temporary dict d and then assiging d to dict[user].

def random_number_check(user, dic, list2):
    d = {}
    for i in list2:
        d[i] = random.randint(0, 1000)
    dic[user] = d

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    return_dict = manager.dict()
    jobs = []
    for i in range(10):
        # Pass the user at position i instead of the whole list
        p = multiprocessing.Process(target=random_number_check, args=(users[i], return_dict, users2))
        jobs.append(p)
        p.start()

    for proc in jobs:
        proc.join()

    print(return_dict)

CodePudding user response：

you should use dic[user] = manager.dict() or d = {} and dic[user] = d as @Albin Paul said.