-
-
Notifications
You must be signed in to change notification settings - Fork 29.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
subprocess is not safe from deadlocks #59077
Comments
There is no way to write a program in Python capable to process large/unlimited output coming from a subprocess stream without deadlocks. http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate http://docs.python.org/library/subprocess.html#subprocess.Popen.stdin So, what should I use? |
I think the note for communicate() just means that you might get MemoryError (or some other exception) if the output is too big. But I agree it is ambiguous. communicate() uses select() on Unix and threads on Windows, so deadlocks should not be possible.
Use communicate() (on a machine with infinite memory;-) |
what sort of machine has infinite memory? |
Well if you're *certain* that the process is only using one stream, then you can just use read/write on that stream. If not, it probably means you have to use either threads or select/poll. This is a known issue with subprocess; there are a few proposals on the tracker about this. See bpo-1191964 for example. |
The problem with memory is more actual for machines with SSD where swap is usually turned off and /tmp files are located on memory disk. Hitting memory limit often means hard reset. My process is pretty generic that uses all streams, and I don't know how to use threads/polls crossplatform way. bpo-1191964 looks interesting. |
See also bpo-1260171. Closing as a duplicate of that. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: