classification
Title: subprocess.Popen causes socket to remain open after close
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: kevinwatters, mamulsow, neologix, ngrilly, pitrou
Priority: high Keywords:

Created on 2008-05-29 21:02 by mamulsow, last changed 2013-05-20 21:41 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
socketTest.py mamulsow, 2008-05-29 21:02 socketTest.py
socketinherit.py kevinwatters, 2008-08-01 17:42 Snippet that patches socket.socket to create uninheritable sockets
Messages (5)
msg67511 - (view) Author: Matt Mulsow (mamulsow) Date: 2008-05-29 21:02
On Windows, when a suprocess.Popen command is issued while
a socket connection is being handled the socket connection
will not close until the output of the subprocess is consumed.
The connection remains open even though the request.close()
command returns successfully and the program starts listening
for the next connection.

On Windows, the request.close() call translates to the C function
closesocket. The closesocket function by default attempts to do
a graceful close in the background. By changing the linger structure,
the closesocket function can be made to do a hard close. That fixes
my problem, but then queued data may not be flushed before the
socket closes. I cannot figure out why the closesocket's graceful
shutdown is waiting for the Popen command to complete.

This problem does not show up with the equivalent os.popen command.

To reproduce:
    run socketTest.py
    use telnet to connect on port 6288
    watch for connection to close or not
msg70572 - (view) Author: Kevin Watters (kevinwatters) Date: 2008-08-01 17:42
I found a workaround for this issue (attached) via the kernel32.dll
function SetHandleInformation. You can patch the socket class to set all
newly created sockets as uninheritable.

It's not perfect--another thread could still spawn a subprocess in
between.  We probably need some kind of API for setting socket inheritance.
msg71167 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-15 10:25
> It's not perfect--another thread could still spawn a subprocess in
> between

Then it could be done in socketmodule.c, when still holding the GIL.
Python code can change back the socket to inheritable afterwards if it
wants to.

> We probably need some kind of API for setting socket inheritance.

Right.
msg126077 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2011-01-12 09:20
> I cannot figure out why the closesocket's graceful
shutdown is waiting for the Popen command to complete.

It doesn't. Your main process closes its socket. You could see it with a netstat/lsof (don't know under Windows).
The problem is that when you call subprocess.Popen(), there's a fork behind, and the child process receives a copy of the FD. That's why your socket is not deallocated until the subprocess completes (and closes the FD on exit).

> This problem does not show up with the equivalent os.popen command.

That's because os.popen() is implemented as subprocess.Popen with close_fds=True, so the socket is closed before execve is called. IMHO, setting close_fds=True by default is the Right Thing to do (and I think it's now done by default in py3k to avoid races between concurrent popen calls, in addition to setting pipe's FD as FD_CLOEXEC).
msg189699 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-05-20 21:41
It's fixed now that subprocess defaults to close_fds=True.
History
Date User Action Args
2013-05-20 21:41:17neologixsetstatus: open -> closed
resolution: out of date
messages: + msg189699

stage: resolved
2011-06-12 21:37:37terry.reedysetversions: + Python 3.3, - Python 2.6, Python 3.1
2011-01-12 09:20:29neologixsetnosy: + neologix
messages: + msg126077
2010-05-11 20:39:10terry.reedysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 3.0
2008-08-26 23:44:23ngrillysetnosy: + ngrilly
2008-08-15 10:25:48pitrousetpriority: high
nosy: + pitrou
messages: + msg71167
versions: + Python 2.6, Python 3.0, - Python 2.5
2008-08-01 17:42:08kevinwatterssetfiles: + socketinherit.py
messages: + msg70572
2008-07-30 18:52:34kevinwatterssetnosy: + kevinwatters
2008-05-29 21:02:37mamulsowcreate