A multiprocessing Queue object managers multiple resources:

• a multiprocessing Pipe
• a thread
• (a lock and a semaphore)

If a Queue is not cleaned up properly, your application may leak many resources.

Try attached queue_leak.py to see an example "leaking a thread".

I suggest to emit a ResourceWarning warning in Queue destrutor. I don't know what should be the test to decide if a warning must be emitted?

• if the queue wasn't closed yet?
• if the thread is alive?
• if the queue wasn't closed yet and/or the thread is alive? (my favorite choice)

Other examples of objects emitting ResourceWarning:

• io files: io.FileIO, io.TextIOWrapper, etc.
• socket.socket
• subprocess.Popen: I recently added a ResourceWarning on that one
• asyncio transports and event loops

Example:

haypo@selma$ ./python queue_leak.py
number of thread diff: +1
dangling threads!
before: [<_MainThread(MainThread, started 139814961067072)>]
after: [<_MainThread(MainThread, started 139814961067072)>]

Note: queue_leak.py resource check is based on test.support.reap_threads.

Oh, I forgot that I hitted this issue while analyzing issue bpo-30131: test_logging leaks a "dangling" thread. It took me a while to find multiprocessing queues in the big test_logging file!

See also issue bpo-30244: Emit a ResourceWarning in concurrent.futures executor destructors.

The thread seems to be stopped when the Queue object is finalized:

        # Send sentinel to the thread queue object when garbage collected
        self._close = Finalize(
            self, Queue._finalize_close,
            [self._buffer, self._notempty],
            exitpriority=10
            )

I don't think the other resources (pipe, lock, semaphore) need explicit cleaning.

Another example:
http://bugs.python.org/issue30886#msg298014

"The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess."

The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess.

I don't understand what this means. Can you clarify a bit?

Specifically "the thread wasn't started by a subprocess"...

Specifically "the thread wasn't started by a subprocess"...

I'm talking about this check in Queue._start_thread() of multiprocessing.queues:

        created_by_this_process = (self._opid == os.getpid())
        if not self._joincancelled and not created_by_this_process:
            self._jointhread = Finalize(
                self._thread, Queue._finalize_join,
                [weakref.ref(self._thread)],
                exitpriority=-5
                )

Let's discuss created_by_this_process in bpo-30886.

This issue is more about adding or not a ResourceWarning.

I don't think a ResourceWarning should be emitted. There is no risk of data loss or resource leak if you don't close a multiprocessing Queue explicitly. Actually, in the real world, I don't think I've ever seen code that closes queues explicitly.

I'm willing to close this issue, as I don't think a ResourceWarning is appropriate here.