Message329412
When communicate() is called in a loop, it crashes when the child process has already closed any piped standard stream, but still continues to be running.
How this happens:
1) the parent waits for the child events inside communicate() call
2) the child closes its side of any attached pipes long before exiting (in my case there is some complex c++ application which had messed with its termination)
3) communicate() receives an epoll event, tries to read/write, receives SIGPIPE (for stdin) or EOF (for stdout), decides to close corresponding file descriptors from its side
4) communicate() waits for the death of the child, but a timeout is fired
5) parent handles timeout exception and calls communicate() again
6) an exception is raised when communicate() tries to register closed file in epoll
I think there may be a simple solution: before registering file descriptors in epoll, we may check whether any of them is already closed, and don't register it in that case.
Here is a simple reproducible example, ran on Linux 4.15.0-1021-aws x86_64:
import subprocess
child = subprocess.Popen(
['/usr/local/bin/python3.7', '-c', 'import os, time; os.close(1), time.sleep(30)'],
stdout=subprocess.PIPE,
)
while True:
try:
child.communicate(timeout=3)
break
except subprocess.TimeoutExpired:
# do something useful here
pass
Here is a stacktrace:
Traceback (most recent call last):
File "test.py", line 10, in <module>
child.communicate(timeout=3)
File "/usr/local/lib/python3.7/subprocess.py", line 933, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/local/lib/python3.7/subprocess.py", line 1666, in _communicate
selector.register(self.stdout, selectors.EVENT_READ)
File "/usr/local/lib/python3.7/selectors.py", line 352, in register
key = super().register(fileobj, events, data)
File "/usr/local/lib/python3.7/selectors.py", line 238, in register
key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
File "/usr/local/lib/python3.7/selectors.py", line 225, in _fileobj_lookup
return _fileobj_to_fd(fileobj)
File "/usr/local/lib/python3.7/selectors.py", line 40, in _fileobj_to_fd
"{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=3> |
|
Date |
User |
Action |
Args |
2018-11-07 10:23:30 | and800 | set | recipients:
+ and800 |
2018-11-07 10:23:30 | and800 | set | messageid: <1541586210.62.0.788709270274.issue35182@psf.upfronthosting.co.za> |
2018-11-07 10:23:30 | and800 | link | issue35182 messages |
2018-11-07 10:23:30 | and800 | create | |
|