This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Add multiprocessing.queues.SimpleQueue.close()
Type: resource usage Stage: resolved
Components: Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: arigo, davin, iritkatriel, miss-islington, pitrou, vstinner, xiang.zhang
Priority: normal Keywords: patch

Created on 2017-07-19 08:53 by arigo, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2760 closed vstinner, 2017-07-19 10:31
PR 2776 closed vstinner, 2017-07-20 13:20
PR 19735 merged vstinner, 2020-04-27 15:20
PR 19738 merged vstinner, 2020-04-27 18:30
Messages (11)
msg298645 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2017-07-19 08:53
multiprocessing.queues.SimpleQueue should have a close() method.  This is needed to explicitly release the two file descriptors of the Pipe used internally.  Without it, the file descriptors leak if a reference to the SimpleQueue object happens to stay around for longer than expected (e.g. in a reference cycle, or with PyPy).

I think the following would do:

diff -r 0b72fd1a7641 lib-python/2.7/multiprocessing/queues.py
--- a/lib-python/2.7/multiprocessing/queues.py  Sun Jul 16 13:41:28 2017 +0200
+++ b/lib-python/2.7/multiprocessing/queues.py  Wed Jul 19 10:45:03 2017 +0200
@@ -358,6 +358,11 @@
             self._wlock = Lock()
         self._make_methods()

+    def close(self):
+        # PyPy extension: CPython doesn't have this method!
+        self._reader.close()
+        self._writer.close()
+
     def empty(self):
         return not self._reader.poll()
msg298649 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-07-19 09:44
Get this solved then we could also solve #23267, which reports pipes leak in pool due to using SimpleQueue.
msg298651 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-19 09:46
On Python 3.6+, regrtest is able to detect file descriptor leaks when using --huntrleaks. Is someone interested to backport this feature to 3.5 and/or 2.7 which would mean fix all FD leaks in our test suite?

If someone wants to work on that, I would suggest to first fix all bugs, and when the test suite is fine: modify regrtest.
msg298657 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-19 10:06
See also my bpo-30171: "Emit ResourceWarning in multiprocessing Queue destructor".
msg298660 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-19 10:14
This issue is marked as also affecting Python 3.7. I don't see a SimpleQueue.close() method in master, but I don't remind any resource warning when running tests on master neither.

Does it mean that our test runner miss such file descriptor leak? Or worse, that SimpleQueue is not tested? I see a TestSimpleQueue test case in Lib/test/_test_multiprocessing.py.

haypo@selma$ ./python -m test test_multiprocessing_spawn -m TestSimpleQueue -R 3:3
(...)
Tests result: SUCCESS


> This is needed to explicitly release the two file descriptors of the Pipe used internally.  Without it, the file descriptors leak if a reference to the SimpleQueue object happens to stay around for longer than expected (e.g. in a reference cycle, or with PyPy).

Oh ok, the garbage collector is able to close the file descriptors. The bug is when a SimpleQueue is not collected.

So again, I consider that a ResourceWarning is needed here. The purpose of a ResourceWarning is to warn when a leak may be kept alive longer than expected if it's not closed explicitly.
msg298661 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-19 10:16
Anyone who thinks those objects stay alive too long can submit a PR for SimpleQueue and Pool to close them eagerly.
msg298667 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-19 10:31
> Anyone who thinks those objects stay alive too long can submit a PR for SimpleQueue and Pool to close them eagerly.

I started with SimpleQueue for the master branch:
https://github.com/python/cpython/pull/2760
msg367432 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-27 15:47
bpo-23267 is marked as a duplicate of this issue.
msg367435 - (view) Author: miss-islington (miss-islington) Date: 2020-04-27 16:11
New changeset 9adccc1384568f4d46e37f698cb3e3a4f6ca0252 by Victor Stinner in branch 'master':
bpo-30966: Add multiprocessing.SimpleQueue.close() (GH-19735)
https://github.com/python/cpython/commit/9adccc1384568f4d46e37f698cb3e3a4f6ca0252
msg367445 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-27 18:53
New changeset 1a275013d1ecc2e3778d64fda86174b2f13d6969 by Victor Stinner in branch 'master':
bpo-30966: concurrent.futures.Process.shutdown() closes queue (GH-19738)
https://github.com/python/cpython/commit/1a275013d1ecc2e3778d64fda86174b2f13d6969
msg396246 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-06-21 13:49
This seems complete, can it be closed?
History
Date User Action Args
2022-04-11 14:58:49adminsetgithub: 75149
2021-06-21 14:00:05iritkatrielsetversions: + Python 3.9, - Python 3.10
2021-06-21 13:51:54pitrousetversions: + Python 3.10, - Python 2.7, Python 3.5, Python 3.6, Python 3.7
2021-06-21 13:50:44pitrousetstatus: pending -> closed
type: resource usage
stage: patch review -> resolved
2021-06-21 13:49:22iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg396246

resolution: fixed
2020-04-27 18:53:43vstinnersetmessages: + msg367445
2020-04-27 18:30:11vstinnersetpull_requests: + pull_request19060
2020-04-27 16:11:20miss-islingtonsetnosy: + miss-islington
messages: + msg367435
2020-04-27 16:02:38vstinnersettitle: multiprocessing.queues.SimpleQueue leaks 2 fds -> Add multiprocessing.queues.SimpleQueue.close()
2020-04-27 15:47:07vstinnersetmessages: + msg367432
2020-04-27 15:20:29vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19057
2017-07-20 13:20:43vstinnersetpull_requests: + pull_request2830
2017-07-19 10:31:58vstinnersetmessages: + msg298667
2017-07-19 10:31:22vstinnersetpull_requests: + pull_request2819
2017-07-19 10:21:12pitroulinkissue23267 superseder
2017-07-19 10:21:12pitrouunlinkissue23267 dependencies
2017-07-19 10:16:37pitrousetmessages: + msg298661
2017-07-19 10:14:20vstinnersetversions: - Python 3.3, Python 3.4
2017-07-19 10:14:13vstinnersetmessages: + msg298660
2017-07-19 10:06:24vstinnersetnosy: + pitrou, davin
2017-07-19 10:06:13vstinnersetmessages: + msg298657
2017-07-19 09:47:59xiang.zhanglinkissue23267 dependencies
2017-07-19 09:46:47vstinnersetmessages: + msg298651
2017-07-19 09:45:13vstinnersetnosy: + vstinner
2017-07-19 09:44:08xiang.zhangsetnosy: + xiang.zhang
messages: + msg298649
2017-07-19 08:53:22arigocreate