subprocess.Popen.communicate() hangs for daemonized subprocesses that leave a pipe open. The python caller, invoking the daemon subprocess, runs as long as the daemon process. This happens on POSIX platforms with python 2.7 as well as 3.2. 

Please find attached a simple python daemon, which leaves stderr pipe open. It runs for 10 sec and then exits.

To reproduce the problem run the following start script:
##################
# startpydaemon.py
import subprocess

cmd = 'python pydaemon.py'
print("Starting daemon that runs for 10 sec")
daemon = subprocess.Popen(cmd, 
                          stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE,
                          shell=True)
stdout, stderr = daemon.communicate()
print("Stdout:")
print(stdout)
print("Stderr:")
print(stderr)
##################

Expected Behavior:
------------------
Daemon starter should exit immediately after the daemon process is initialized.

Actual Behavior:
----------------
Daemon starter waits for daemon process to be finished due to left open error pipe. 

Workaround:
-----------
One workaround is to not capture output from the left-open pipe. This way.

> subprocess.Popen.communicate() hangs for daemonized subprocesses that 
> leave a pipe open. [...] which leaves stderr pipe open.

Of course: the daemon process (spawned by the second fork()) inherits the subprocess's stderr (since file descriptors are inherited upon fork), and communicate waits until the subprocess's stderr and stdout are closed (EOF returned).
And it's of course documented, see http://docs.python.org/dev/library/subprocess.html#subprocess.Popen.communicate :
"""
Popen.communicate(input=None, timeout=None) 
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
"""

This looks like a bug in your daemon not in subprocess.  Your daemon is intentionally not closing its inherited stderr fd.

The behavior that the daemon is not closing stderr is intentional to reproduce the issue. This problem occured to me when I was invoking the samba init script on a Gentoo system.
Invoked from a bash, the initscript was terminating fine. But when I invoked it using the subprocess module, my python script hang.

The problem I have with the solution that is currently implemented is that subprocess is waiting for the spawned child although the child is not running anymore.
In my case this issue occured when invoking samba or the small sample daemon (see attached file above).
For example, invoking the daemon from a bash, the daemon program exits immediately and its spawned child is running in the background. The point is that the bash is accessible right after starting the daemon. It is not waiting for the daemon process in the background to close the open pipe.

What do you think of using a waitpid for this scenario?
subprocess.Popen could wait for the pid of the daemon starter and gets back control once the pid of the daemon starter is gone. This way, subprocess.Popen does not need to wait for EOF on left-open pipes.

> The problem I have with the solution that is currently implemented is that subprocess is waiting for the spawned child although the child is not running anymore.
> In my case this issue occured when invoking samba or the small sample daemon (see attached file above).
> For example, invoking the daemon from a bash, the daemon program exits immediately and its spawned child is running in the background. The point is that the bash is accessible right after starting the daemon. It is not waiting for the daemon process in the background to close the open pipe.
>
> What do you think of using a waitpid for this scenario?
> subprocess.Popen could wait for the pid of the daemon starter and gets back control once the pid of the daemon starter is gone. This way, subprocess.Popen does not need to wait for EOF on left-open pipes.

That would defeat the semantics of communicate(): it is supposed to
wait until EOF.
If this doesn't work for your use case, simply don't use
communicate(), or close stdout and stderr in your child process.