This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author crazycasta
Recipients crazycasta
Date 2020-12-26.23:03:40
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1609023821.09.0.189504509221.issue42752@roundup.psfhosted.org>
In-reply-to
Content
Didn't feel like necroing #33081, but this is basically that problem. The trouble is the cleanup that appeared to fix #33081 only kicks in once something has been put in the queue. So if for instance a Process function puts something in the queue and the parent gets it, then calls q.close() the writer on the parent side doesn't get culled until the object does. This is particularly a problem for PyPy and isn't exactly great for any weird corner cases if anyone holds onto Queue objects after they're closed for any reason (horders!).

Attached file test_queue.py is an example of how to trigger this problem. Run it without a command line argument "python test_queue.py" and it won't crash (though it will take a very long time to complete). Run with an argument "python test_queue.py fail" and it will fail once you run out of file descriptors (one leaked per queue).

My suggestion on how to handle this is to set self._close to something that will close self._writer. Then, when _start_thread is called, instead of directly passing the self._writer.close object, pass a small function that will switch out self._close to the Finalize method used later on and return self._writer. Finally, inside _feed, use this method to get the _writer object and wrap the outer while 1 with a contextlib.closer on this object.

This is a fair bit of stitching things together here and there so let me know if anyone has any suggestions on this before I get started.
History
Date User Action Args
2020-12-26 23:03:41crazycastasetrecipients: + crazycasta
2020-12-26 23:03:41crazycastasetmessageid: <1609023821.09.0.189504509221.issue42752@roundup.psfhosted.org>
2020-12-26 23:03:41crazycastalinkissue42752 messages
2020-12-26 23:03:40crazycastacreate