classification
Title: subprocess leaves open fds on construction error
Type: Stage:
Components: Library (Lib) Versions: Python 3.0, Python 3.1, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: georg.brandl, ocean-city, pitrou
Priority: high Keywords: patch

Created on 2009-02-07 20:04 by georg.brandl, last changed 2009-03-03 22:56 by ocean-city. This issue is now closed.

Files
File name Uploaded Description Edit
sp-patch.py georg.brandl, 2009-02-07 20:17
fix_subprocess_leak_on_failure_on_windows.patch ocean-city, 2009-03-03 21:04
Messages (12)
msg81347 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-02-07 20:04
The test case below will (on Linux here) eventually quit with "OSError:
[Errno 24] Too many open files".  I assume that some additional cleaning
up is in order.

-------------------------------------------------------------------

from subprocess import Popen, PIPE

while 1:
    try:
        Popen(['nonexisting'], stdout=PIPE, stderr=PIPE)
    except OSError, err:
        if err.errno != 2:  # ignore "no such file"
            raise
msg81348 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-02-07 20:17
Proposed patch attached.
msg81387 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-08 16:06
Perhaps you could add a test?
msg82001 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-02-14 08:49
I would, but how this fails is likely to be highly platform-specific. 
Can you try it on Windows and tell me what the resulting exception is?
msg82028 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-14 12:18
Well, I'm not under Windows. I'll try to launch a VM if nobody beats me
to it...
msg82082 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-02-14 16:00
On second thought, if the patch is applied, there *shouldn't* be an
exception.  And simply running the loop for a fixed number of
repetitions isn't a good test either, since different platforms may have
a different maximum for open file descriptors.

Hmm, is there a way to get the current number of open file descriptors?
msg82083 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-14 16:12
> Hmm, is there a way to get the current number of open file descriptors?

Under Unix, you can use resource.getrlimit():

(1024, 1024)

But 1024 is a very common value, so you could simply loop 1024 times if
it's not too slow.
Besides, you definitely don't want to loop 2**31 times if 2**31 happens
to be the current limit.
msg82084 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-14 16:13
Wow, the Roundup email gateway borks code snippets:

>>> import resource
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(1024, 1024)
msg82092 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-02-14 17:01
OK, Windows has 2048. Added test and fixed in r69620.
msg83097 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-03 21:04
This issue is not fixed on windows yet. (test_leaking_fds_on_error fails)
I think attached patch will fix this.

======================================================================
ERROR: test_writes_before_communicate (__main__.ProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_subprocess.py", line 398, in test_writes_before_communicate
  File "e:\python-dev\py3k\lib\subprocess.py", line 632, in __init__
  File "e:\python-dev\py3k\lib\subprocess.py", line 746, in _get_handles
IOError: [Errno 24] Too many open files
msg83099 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-03-03 21:05
Since I can't test on Windows, I'll leave that in your hands :)
msg83108 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-03 22:56
Thanks, fixed in r70137(trunk), r70142(py3k), r70146(release30-maint),
r70147(release26-maint)
History
Date User Action Args
2009-03-03 22:56:40ocean-citysetstatus: open -> closed
resolution: fixed
messages: + msg83108
priority: release blocker -> high
2009-03-03 21:05:40georg.brandlsetmessages: + msg83099
2009-03-03 21:04:27ocean-citysetstatus: closed -> open
files: + fix_subprocess_leak_on_failure_on_windows.patch
versions: + Python 3.0, Python 3.1
keywords: + patch
nosy: + ocean-city
messages: + msg83097
resolution: fixed -> (no value)
priority: high -> release blocker
2009-02-14 17:01:53georg.brandlsetstatus: open -> closed
resolution: fixed
messages: + msg82092
2009-02-14 16:13:59pitrousetmessages: + msg82084
2009-02-14 16:12:58pitrousetmessages: + msg82083
2009-02-14 16:00:45georg.brandlsetmessages: + msg82082
2009-02-14 12:18:40pitrousetmessages: + msg82028
2009-02-14 08:49:30georg.brandlsetmessages: + msg82001
2009-02-08 16:06:50pitrousetnosy: + pitrou
messages: + msg81387
2009-02-07 20:17:55georg.brandlsetfiles: + sp-patch.py
messages: + msg81348
2009-02-07 20:04:37georg.brandlcreate