Python Multiprocessing.Queue.get - timeout and blocking not working?-CodePudding

I'm building an application that runs multiple processes. These processes pass around objects through queues. All works fine. Except for the fact that I need processes to stick around a while when their input queues are empty. The queue.get() method is implemented in an infinite loop, but I need this loop to run every once in a while even though they have no input in the queue to be able to run some housekeeping code.

So essentially it's an infinite loop that needs to block at the queue.get() for at least a second, and do some stuff when there is input otherwise do some other stuff.

The problem is: the timeout doesn't seem to be working:

  while True:
            # Process input queue, but make sure to continue the loop once every second if the queue is empty
            wd_input = None
            try:
                wd_input = self.input_queue.get(block=True, timeout=1)
            except Empty:                    
                logging.debug(f"Watchdog queue empty. (interval: {interval.microseconds} microseconds)")

I have removed some code for clarity. Just believe me I'm recording the interval in between the Empty exceptions. I get this output:

[2022-05-04 09:35:27,648] DEBUG: Watchdog queue empty. (interval: 1412 microseconds)
[2022-05-04 09:35:28,650] DEBUG: Watchdog queue empty. (interval: 1306 microseconds)
[2022-05-04 09:35:29,651] DEBUG: Watchdog queue empty. (interval: 1417 microseconds)
[2022-05-04 09:35:30,652] DEBUG: Watchdog queue empty. (interval: 1329 microseconds)
[2022-05-04 09:35:31,654] DEBUG: Watchdog queue empty. (interval: 1323 microseconds)
[2022-05-04 09:35:32,655] DEBUG: Watchdog queue empty. (interval: 1318 microseconds)
[2022-05-04 09:35:33,656] DEBUG: Watchdog queue empty. (interval: 1324 microseconds)
[2022-05-04 09:35:34,658] DEBUG: Watchdog queue empty. (interval: 1322 microseconds)
[2022-05-04 09:35:35,659] DEBUG: Watchdog queue empty. (interval: 1308 microseconds)

Which is far from the 1 second I was wanting to achieve. Essentially, that's also what the documentation says:

If timeout is a positive number, it blocks at most timeout seconds and raises the queue.Empty exception if no item was available within that time. (https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue)

So maybe this is expected. What I want is not to wait AT MOST timeout seconds but AT LEAST timeout seconds. Is this even possible? How would one do this?

CodePudding user response：

I don't know what this interval.microseconds is that you are printing out. But have you noticed the timestamp on the actual debug message? These messages seem to be outputted every second within a couple of milliseconds.

import logging
import time
from queue import Queue, Empty

logging.basicConfig(format='%(asctime)s %(message)s')

q = Queue()

for _ in range(5):
    try:
        t0 = time.time()
        msg = q.get(timeout=1)
    except Empty:
        t1 = time.time()
        logging.warning(f'is when this queue empty event was logged. Elapsed = {t1 - t0}')

Prints:

2022-05-04 07:24:05,768 is when this queue empty event was logged. Elapsed = 1.000537395477295
2022-05-04 07:24:06,769 is when this queue empty event was logged. Elapsed = 1.0004637241363525
2022-05-04 07:24:07,770 is when this queue empty event was logged. Elapsed = 1.000412940979004
2022-05-04 07:24:08,771 is when this queue empty event was logged. Elapsed = 1.000406265258789
2022-05-04 07:24:09,772 is when this queue empty event was logged. Elapsed = 1.000260591506958