I am trying to update a class that is supposed to be used as a custom type for the multiprocessing.manager and imitate a basic dictionary. All works well on Linux, but things fail on Windows and I understood that the problem lies in a possibly suboptimal creation mechanism that it uses that involves a closure. With forking, Linux gets around serializing something that pickle cannot cope with, while this does not happen on Windows. I am using Python 3.6 and feel like it is better to improve the class rather than force a new package dependency that has more robust serialization than pickle.

An example that I think demonstrates this is presented below. It involves a class that is meant to act like a dict, but have an additional method and a class attribute. These are bound in a factory method that the code calls and passes to multiprocessing.manager.register. I get AttributeError: Can't pickle local object 'foo_factory.<locals>.Foo' as a result here.

import abc
import pickle


class FooTemplate(abc.ABC, dict):

    bar = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.foo = 'foo'

    @abc.abstractmethod
    def special_method(self, arg1, arg2):
        pass


def foo_factory(dynamic_special_method):

    class Foo(FooTemplate):
        bar = 'bar'

        def special_method(self, arg1, arg2):
            print(self.foo, ' ', self.bar, ' ', dynamic_special_method(arg1, arg2))

    return Foo


def method_to_pass(a1, a2):
    return a1   a2


if __name__ == '__main__':
    foo = foo_factory(method_to_pass)()
    pickle.dumps(foo)

I attempted to fix the problem by creating a class dynamically, but this throws a new error that I am not sure I understand and it makes things look even worse with all honesty. Using the main part from above with the code below produces error _pickle.PicklingError: Can't pickle <class '__main__.Foo'>: attribute lookup Foo on __main__ failed.

class FooTemplate(dict):

    bar = None
    method_map = None

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.foo = 'foo'

    def special_method(self, arg1, arg2):
        print(self.foo, ' ', self.bar, ' ', self.method_map[self.bar](arg1, arg2))


def foo_factory(dynamic_special_method):
    return type('Foo', (FooTemplate,), {'bar': 'bar', 'method_map': {'bar': dynamic_special_method}})

Error above aside, I feel like I am missing something fundamental and that I took a wrong direction. Even if this worked, it feels wrong to introduce a new attribute with a nested structure to simply keep a method which avoids calls to this method as a class method with self in the front...

Maybe someone can suggest a better direction how to create a preferably serializable class which imitates a dictionary and that can also get parameters dynamically? An explanation of the error that I get would be very useful too, but I think this is not the biggest problem I am facing here. Thank you for any help in advance.

CodePudding user response：

With a lot of help and encouragement from @CharchitAgarwal, I have figured out two solutions to the problem.

1. If you are looking for the solution to the error in my last attempt, then Pickle a dynamically parameterized sub-class can be consulted. Tried it, all worked with minimal alterations, but the final result was harder to follow and not satisfying. Due to ease of copy/pasting and the bitter taste afterwards, I will not share the solution here.

2. Better answer to the question is to stop for a moment and to reflect if you are on the right path. Had this doubt when posting the question and, fortunately, I am not committed to the structure above. I think it is better to use composition instead of inheritance. Code is shorter and much more simple. If necessary, one can bind the special parameters with functools.partial and objects are still going to be serializable in Python 3.6 that was used here.

import pickle


class Foo(dict):

    def __init__(self, *args, special_method=None, bar=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.foo = 'foo'
        self._special_method = special_method
        self.bar = bar

    def special_method(self, arg1, arg2):
        print(self.foo, ' ', self.bar, ' ', self._special_method(arg1, arg2))


def method_to_pass(a1, a2):
    return a1   a2


if __name__ == '__main__':
    foo = Foo({'foo': 'bar'}, special_method=method_to_pass, bar='bar')
    print(foo)
    foo.special_method(4, 2)
    pickle.dumps(foo)