Getting information back from a process with a multiprocessing Queue-CodePudding

I am trying to play around with multiprocessing and I would like to communicate between Python's main thread and a subprocess with a Queue. Here is a quick test code I wrote that should get periodically some results generated by the subprocess:

from multiprocessing import Process, Queue
import time

def calculate(queue):  
    n = 0
    while n < 10:   
        n  = 1
        queue.put(n)
        time.sleep(1)
    queue.put(0)

def queue_getter(queue):
    executing = True
    while executing:     
        while queue.qsize():
            n = queue.get()
            print(n)
            if n == 0:
                executing = False
        time.sleep(0.1)
    print('done')

queue = Queue()
p = Process(target=calculate, args=(queue,))
p.start()
queue_getter(queue)
p.join()
print('DONE')

This program just hangs forever, while replacing Process with threading.Thread gives the expected result:

1
2
3
4
5
6
7
8
9
10
0
done
DONE

How to make Process behave the same way as Thread in this situation?

CodePudding user response：

Your program works fine on POSIX (UNIX-like) systems.

However, for it to work properly on ms-windows and macOS, you will need to put the program itself inside a main block, so the file can be imported without side effects.

This is due to the way multiprocessing has to work on ms-windows and macOS. Read the programming guidelines for multiprocessing.

Modify your code like this:

from multiprocessing import Process, Queue
import time


def calculate(queue):
    n = 0
    while n < 10:
        n  = 1
        queue.put(n)
        time.sleep(1)
    queue.put(0)


def queue_getter(queue):
    executing = True
    while executing:
        while queue.qsize():
            n = queue.get()
            print(n)
            if n == 0:
                executing = False
        time.sleep(0.1)
    print("done")


if __name__ == "__main__":
    queue = Queue()
    p = Process(target=calculate, args=(queue,))
    p.start()
    queue_getter(queue)
    p.join()
    print("DONE")

CodePudding user response：

Here's a simplified and more robust approach which is (almost) functionally identical to the OP's original except that is does not print the zero:

from multiprocessing import Manager
from concurrent.futures import ProcessPoolExecutor
import time

def calculate(q):
    for n in range(1, 11):
        q.put(n)
        time.sleep(1)
    q.put(0)

def queue_getter(q):
    while (n := q.get()):
        print(n)

def main():
    with Manager() as manager:
        q = manager.Queue()
        with ProcessPoolExecutor() as executor:
            executor.submit(calculate, q)
            queue_getter(q)

if __name__ == '__main__':
    main()