classification
Title: Emit ResourceWarning in multiprocessing Queue destructor
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: davin Nosy List: davin, haypo, pitrou, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-04-26 13:52 by haypo, last changed 2017-07-22 10:21 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
queue_leak.py haypo, 2017-04-26 13:52
Messages (12)
msg292346 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-26 13:52
A multiprocessing Queue object managers multiple resources:

* a multiprocessing Pipe
* a thread
* (a lock and a semaphore)

If a Queue is not cleaned up properly, your application may leak many resources.

Try attached queue_leak.py to see an example "leaking a thread".

I suggest to emit a ResourceWarning warning in Queue destrutor. I don't know what should be the test to decide if a warning must be emitted?

* if the queue wasn't closed yet?
* if the thread is alive?
* if the queue wasn't closed yet and/or the thread is alive? (my favorite choice)

Other examples of objects emitting ResourceWarning:

* io files: io.FileIO, io.TextIOWrapper, etc.
* socket.socket
* subprocess.Popen: I recently added a ResourceWarning on that one
* asyncio transports and event loops
msg292347 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-26 13:54
Example:

haypo@selma$ ./python queue_leak.py 
number of thread diff: +1
dangling threads!
before: [<_MainThread(MainThread, started 139814961067072)>]
after: [<_MainThread(MainThread, started 139814961067072)>]

Note: queue_leak.py resource check is based on test.support.reap_threads.
msg292348 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-04-26 13:55
Oh, I forgot that I hitted this issue while analyzing issue #30131: test_logging leaks a "dangling" thread. It took me a while to find multiprocessing queues in the big test_logging file!
msg292850 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-03 07:18
See also issue #30244: Emit a ResourceWarning in concurrent.futures executor destructors.
msg293001 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-05-04 17:03
The thread seems to be stopped when the Queue object is finalized:

        # Send sentinel to the thread queue object when garbage collected
        self._close = Finalize(
            self, Queue._finalize_close,
            [self._buffer, self._notempty],
            exitpriority=10
            )

I don't think the other resources (pipe, lock, semaphore) need explicit cleaning.
msg298015 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-10 00:31
Another example:
http://bugs.python.org/issue30886#msg298014

"The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess."
msg298034 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-10 08:56
> The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess.

I don't understand what this means.  Can you clarify a bit?
msg298035 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-10 08:56
Specifically "the thread wasn't started by a subprocess"...
msg298036 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-10 09:15
> Specifically "the thread wasn't started by a subprocess"...

I'm talking about this check in Queue._start_thread() of multiprocessing.queues:

        created_by_this_process = (self._opid == os.getpid())
        if not self._joincancelled and not created_by_this_process:
            self._jointhread = Finalize(
                self._thread, Queue._finalize_join,
                [weakref.ref(self._thread)],
                exitpriority=-5
                )
msg298039 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-10 09:28
Let's discuss created_by_this_process in bpo-30886.

This issue is more about adding or not a ResourceWarning.
msg298045 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-10 09:50
I don't think a ResourceWarning should be emitted.  There is no risk of data loss or resource leak if you don't close a multiprocessing Queue explicitly.  Actually, in the real world, I don't think I've ever seen code that closes queues explicitly.
msg298662 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-19 10:17
I'm willing to close this issue, as I don't think a ResourceWarning is appropriate here.
History
Date User Action Args
2017-07-22 10:21:01pitrousetstatus: pending -> closed
2017-07-19 10:17:30pitrousetstatus: open -> pending
resolution: not a bug
messages: + msg298662

stage: resolved
2017-07-10 09:50:21pitrousetmessages: + msg298045
2017-07-10 09:28:20hayposetmessages: + msg298039
2017-07-10 09:15:35hayposetmessages: + msg298036
2017-07-10 08:56:49pitrousetmessages: + msg298035
2017-07-10 08:56:20pitrousetmessages: + msg298034
2017-07-10 00:31:04hayposetmessages: + msg298015
2017-05-04 17:03:16pitrousetnosy: + pitrou
messages: + msg293001
2017-05-03 07:18:27hayposetmessages: + msg292850
2017-04-26 15:11:54rhettingersetassignee: davin
2017-04-26 13:55:18hayposetmessages: + msg292348
2017-04-26 13:54:09hayposetmessages: + msg292347
2017-04-26 13:52:50haypocreate