I'm having trouble with trying to get a Thread (threading.Thread) subclass to work within a Process (multiprocessing.Process) subclass.
Here is the simplest working example to demonstrate the issue.
I make one "SubProcess" (instance of multiprocessing.Process), which will contain a child "WorkerThread" (instance of threading.Thread). Execution terminates at subProcess.start()
import multiprocessing
import threading
class SubProcess(multiprocessing.Process):
def __init__(self):
multiprocessing.Process.__init__(self, daemon=True)
#Instantiate this one worker
self.workerThread = WorkerThread()
def run(self):
#Start the worker
self.workerThread.start()
class WorkerThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self, daemon=True)
def run(self):
#Real work code goes here
pass
if __name__ == '__main__':
#Program starts here
#Instantiate the SubProcess class - works fine
subProcess = SubProcess()
#Start the subProcess - Execution stops here - see Traceback below
subProcess.start()
subProcess.join()
Here is the output traceback:
Traceback (most recent call last):
File "[***]\simplestExampleError.py", line 31, in <module>
subProcess.start()
File "C:\Python39\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Python39\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Python39\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Python39\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:\Python39\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
[***]>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python39\lib\multiprocessing\spawn.py", line 107, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
File "C:\Python39\lib\multiprocessing\reduction.py", line 79, in duplicate
return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Access is denied
I have reviewed this similar question, but the key difference is that they are defining the "WorkerThread" to point to a function, with predefined arguments (not a subclass of 'threading.Thread').
Example: Within the subProcess, they define workerThread = Thread(target=print_all_the_things, args=("a", self.num))
I require the ability to change the WorkerThread's class variables while it is running.
Example: In the subProcess, I would do something like self.workerThread.input = "INPUT"
while both are running
Any help would be greatly appreciated!
CodePudding user response:
I don't see any reason with the code you posted (of course, you only posted very minimal code) why the instantiation of the worker thread needs to be done in the SubProcess.__init__
method. I would simply do the initialization of the workerThread
attribute in the run
method:
class SubProcess(multiprocessing.Process):
def __init__(self):
multiprocessing.Process.__init__(self, daemon=True)
def run(self):
#Instantiate this one worker
self.workerThread = WorkerThread()
#Start the worker
self.workerThread.start()
I don't think it is a fatal flaw to derive classes from Process
and Thread
classes but it is not very flexible. What if the code using your SubProcess
class wanted to assign a name attribute to the process? In your current implementation it can't without your redesigning the __init__
method. The following implementation is just cleaner and makes the classes reusable in non-multiprocessing, non-multithreading scenarios (for which I would possibly choose different method names other than run
that better describe the processing done by the method):
import multiprocessing
import threading
class SubProcess:
def __init__(self):
# set whatever attributes are required (that can be pickled)
...
def run(self): # or a more descriptive method name
#Instantiate this one worker
worker = Worker()
self.workerThread = threading.Thread(target=worker.run, daemon=True)
#Start the worker
self.workerThread.start()
class Worker:
def __init__(self):
# set whatever attributes are required
...
def run(self): # or a more descriptive method name
#Real work code goes here
pass
if __name__ == '__main__':
subprocess = SubProcess()
# In this context, the Process does not need to be a daemon process
# since we are waiting for its complettion:
p = mulitprocessing.Process(target=subprocess.run, name='SubProcess')
p.start()
p.join()