classification
Title: multiprocessing.Process leaves read pipes open (Process.sentinel)
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: davin Nosy List: Roman Bolshakov, Winterflower, davin, jnoller, josh.r, pitrou, sbt, socketpair
Priority: normal Keywords:

Created on 2016-05-29 10:59 by Roman Bolshakov, last changed 2017-07-22 10:20 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
leak.py Roman Bolshakov, 2016-07-06 15:22
Messages (6)
msg266598 - (view) Author: Roman Bolshakov (Roman Bolshakov) Date: 2016-05-29 10:59
There's no code that closes read pipe(Process.sentinel) when a Process is joined. That creates issues in long running programs as the pipe's file descriptors are effectively leaked.
msg269886 - (view) Author: Roman Bolshakov (Roman Bolshakov) Date: 2016-07-06 15:22
multiprocessing.Queue._writer is affected by similar issue. _writer FD is left open after .close() is invoked on a Queue.
msg295890 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-13 10:34
The Process issue will be fixed with Process.close() in issue30596.
msg295899 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-13 11:16
As for Queue._writer, I don't think this is true.  When you call Queue.close(), it calls the _close() method, which sends a sentinel to the feeder thread (see _finalize_close), and the feeder thread then pops the sentinel, closes the writer and exits.
msg295901 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-13 11:17
As for Process, it doesn't have a close() method right now, so you should lose all references to a Process object once you are done with it.  When a Process is garbage-collected, its Popen object is garbage-collected too, and the Popen object's Finalizer then closes the sentinel fd.
msg297211 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 19:13
Roman, do you think my analysis above is wrong? Otherwise, I'm inclined to close the issue.
History
Date User Action Args
2017-07-22 10:20:22pitrousetstatus: pending -> closed
resolution: rejected
stage: resolved
2017-06-28 19:13:49pitrousetstatus: open -> pending

messages: + msg297211
2017-06-13 11:17:49pitrousetmessages: + msg295901
2017-06-13 11:16:15pitrousetmessages: + msg295899
2017-06-13 10:34:44pitrousetnosy: + pitrou
messages: + msg295890
2017-03-08 04:04:50josh.rsetnosy: + josh.r
2017-03-08 01:43:49rhettingersetassignee: davin
2017-03-07 20:53:50Winterflowersetnosy: + Winterflower
2016-07-06 15:22:59Roman Bolshakovsetfiles: - leak.py
2016-07-06 15:22:38Roman Bolshakovsetfiles: + leak.py

messages: + msg269886
2016-06-05 17:54:07ned.deilysetnosy: + davin
2016-05-29 20:03:12socketpairsetnosy: + socketpair
2016-05-29 16:26:50SilentGhostsetnosy: + jnoller, sbt

versions: + Python 3.6, - Python 3.3, Python 3.4
2016-05-29 10:59:54Roman Bolshakovcreate