I have a class that starts a thread in its __init__ member and I would like join that thread when the instance of that class is not needed anymore, so I implemented that clean-up code in __del__.

It turns out that the __del__ member is never called when the last reference of the instance gets deleted, however if I implicitly call del, it gets called.

Below is a shorter modified version of my implementation that shows the issue.

import sys
from queue import Queue
from threading import Thread

class Manager:

    def __init__(self):
        ''' 
        Constructor. 
        '''
        # Queue storing the incoming messages.
        self._message_q = Queue()
        # Thread de-queuing the messages.
        self._message_thread = \
            Thread(target=process_messages, args=(self._message_q,))
        # Start the processing messages thread to consume the message queue.
        self._message_thread.start()

    def __del__(self):
        ''' 
        Destructor. Terminates and joins the instance's thread.
        '''
        print("clean-up.")
        # Terminate the consumer thread.
        # - Signal the thread to stop.
        self._message_q.put(None)
        # - Join the thread.
        self._message_thread.join()

def process_messages( message_q):
    ''' 
    Consumes the message queue and passes each message to each registered
    observer.
    '''
    while True:
        print("got in the infinite loop")
        msg = message_q.get()
        print("got a msg")
        if msg is None:
            # Terminate the thread.
            print("exit the loop.")
            break
        # Do something with message here.

mgr = Manager()
print("mgr ref count:"   str(sys.getrefcount(mgr) - 1)) # -1 cause the ref passed to getrefcount is copied. 
#del mgr

The console outputs the following for this code:

mgr ref count:1
got in th infinite loop

The execution hangs since the thread is still running. For some reason I don't understand __del__ is not called, and as a result of that the thread is not terminated.

If I uncomment the last line del mgr to explicitly delete the instance, then __del__ gets called and the thread clean-up occurs.

mgr ref count:1
clean-up.
got in the infinite loop
got a msg
exit the loop.
Press any key to continue . . .

Does anyone have an explanation for this?

CodePudding user response：

Silvio's answer is correct, but incomplete. In fact, it's actually guaranteed that mgr won't be deleted in this case because:

1. The message thread won't exit until mgr's deleter has been invoked, and
2. The main thread doesn't actually begin the part of process shutdown that clears module globals until after all non-daemon threads have completed

This ends up with a cyclic problem:

1. The thread won't complete until mgr is destroyed
2. mgr won't be destroyed until the thread completes

The explicit del mgr here works, assuming no other references to mgr exist (implicitly or explicitly). You could get a safer, and automatic version of this cleanup by putting the code in a function, e.g. a standard design (that actually makes things run quicker by replacing use of dict-based globals with function array-based locals) is to put the main functionality in a function, then invoke it:

def main():
    mgr = Manager()
    print("mgr ref count:"   str(sys.getrefcount(mgr) - 1)) # -1 cause the ref passed to getrefcount is copied. 

if __name__ == '__main__':
    main()

It's still not perfect though; an exception can end up holding the frame object after main exits leading to non-deterministic cleanup (similarly, only the CPython reference interpreter uses reference-counting as its primary GC mechanism; on other Python interpreters cleanup is not deterministic). The only way to make this completely deterministic is to make your object a context manager, and use a with statement with it, e.g.:

import sys
from queue import Queue
from threading import Thread

class Manager:
    def __init__(self):
        # unchanged, except for one added line:
        self._closed = False  # Flag to prevent double-close

    # Give named version for rare cases where cleanup must be triggered
    # manually in some other function and therefore with statements can't be used
    def close(self):
        '''Signal thread to end and wait for it to complete'''
        if not self._closed:
            self._closed = True
            print("clean-up.")
            # Terminate the consumer thread.
            # - Signal the thread to stop.
            self._message_q.put(None)
            # - Join the thread.
            self._message_thread.join()

    __del__ = close  # Call it for users on best-effort basis if they forget to use with

    def __enter__(self):
        return self    # Nothing special to do on beginning with block
    def __exit__(self, exc, obj, tb):
        self.close()  # Perform cleanup on exiting with block

def process_messages( message_q):
    # unchanged

with Manager() as mgr:
    print("mgr ref count:"   str(sys.getrefcount(mgr) - 1)) # -1 cause the ref passed to getrefcount is copied. 
# Guaranteed cleanup of mgr here

CodePudding user response：

From the official documentation on __del__

It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.

You have a reference at module-level called mgr to your object. That reference exists when your program terminates, so __del__ may or may not be called.

To elaborate on some of the comments on the question, __del__ should not be thought of as a resource deallocator. That is to say, if you're coming from C , __del__ is not the equivalent of a destructor. __del__ may or may not run (for instance, as indicated above it will not run at program exit), and even if it does, it may run far later in the program than you anticipated, depending on how the garbage collector feels about you.

If you're looking for resource allocation that gets freed when you say so, you want a context manager.

I don't know your exact use case, but if you want to create a mgr object that's freed at the end of a particular block of code, you can write something like

class Manager:

    def __enter__(self):
        # Resource allocation goes here
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        # Resource deallocation goes here; you
        # can use the arguments to determine
        # if an exception was thrown in the 'with'
        # block
        pass

with Manager(...) as mgr:
    # __enter__ is called here
    ... some code ...
    # __exit__ is called at the end of this
    # block, EVEN IF an exception is thrown.