Author izbyshev
Recipients gregory.p.smith, izbyshev, koobs, pablogsal, ronaldoussoren, vstinner
Date 2019-01-26.21:51:17
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1548539478.29.0.204558480235.issue35823@roundup.psfhosted.org>
In-reply-to
Content
I've checked subprocess.Popen() error reporting in QEMU user-mode and WSL and confirm that it works both with my patch (vfork/exec) and the traditional fork/exec, but doesn't work with glibc's posix_spawn.

The first command below uses posix_spawn() internally because close_fds=False. Under QEMU posix_spawn() from glibc doesn't report errors because it relies on address-space sharing of clone(CLONE_VM|CLONE_VFORK), which is not emulated by QEMU. The second command uses vfork()/exec() in _posixsubprocess, but lack of address-space sharing doesn't matter because the error data is transferred via a pipe.

$ qemu-x86_64 --version
qemu-x86_64 version 2.11.1
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
$ ldd --version
ldd (GNU libc) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
$ qemu-x86_64 ./python3 -c 'import subprocess; subprocess.call("/xxx", close_fds=False)'
$ qemu-x86_64 ./python3 -c 'import subprocess; subprocess.call("/xxx")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/izbyshev/install/python-3.8a/lib/python3.8/subprocess.py", line 324, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/home/izbyshev/install/python-3.8a/lib/python3.8/subprocess.py", line 830, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/izbyshev/install/python-3.8a/lib/python3.8/subprocess.py", line 1648, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'xxx'

For WSL, I've tested Ubuntu 18.04 (glibc 2.27) on Windows 10 1709 (Fall Creators Update) and 1803.

In 1709, the result is the same as for QEMU (i.e. the first command silently succeeds). In 1803, both commands raise the exception because address-space sharing was fixed for clone(CLONE_VM|CLONE_VFORK) and vfork() in WSL: https://github.com/Microsoft/WSL/issues/1878

I've also run all subprocess tests under QEMU user-mode (by making sys.executable to point to a wrapper that runs python under QEMU) for the following code bases:
* current master (fork/exec or posix_spawn can be used)
* classic (fork/exec always)
* my patch + master (vfork/exec or posix_spawn)
* vfork/exec always
 All tests pass in all four flavors, which indicates that we don't have a test for error reporting with close_fds=False :)

Out of curiosity I also did the same on WSL in 1803. Only 3 groups of tests fail, all because of WSL bugs:

* It's not possible to send signals to zombies, e.g.:
======================================================================
ERROR: test_terminate_dead (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/test/cpython/Lib/test/test_subprocess.py", line 1972, in test_terminate_dead
    self._kill_dead_process('terminate')
  File "/home/test/cpython/Lib/test/test_subprocess.py", line 1941, in _kill_dead_process
    getattr(p, method)(*args)
  File "/home/test/cpython/Lib/subprocess.py", line 1877, in terminate
    self.send_signal(signal.SIGTERM)
  File "/home/test/cpython/Lib/subprocess.py", line 1872, in send_signal
    os.kill(self.pid, sig)
ProcessLookupError: [Errno 3] No such process
----------------------------------------------------------------------

* getdents64 syscall doesn't work properly (doesn't return the rest of entries on the second call, so close_fds=True doesn't fully work with large number of open descriptors).

======================================================================
FAIL: test_close_fds_when_max_fd_is_lowered (test.test_subprocess.POSIXProcessTestCase)
Confirm that issue21618 is fixed (may fail under valgrind).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/test/cpython/Lib/test/test_subprocess.py", line 2467, in test_close_fds_when_max_fd_is_lowered
    self.assertFalse(remaining_fds & opened_fds,
AssertionError: {34, 35, 36, 37, 38, 39, 40, 41, 42} is not false : Some fds were left open.
----------------------------------------------------------------------

* Signal masks in /proc/self/status are not correct (always zero)

======================================================================
FAIL: test_restore_signals (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/test/cpython/Lib/test/test_subprocess.py", line 1643, in test_restore_signals
    self.assertNotEqual(default_sig_ign_mask, restored_sig_ign_mask,
AssertionError: b'SigIgn:\t0000000000000000' == b'SigIgn:\t0000000000000000' : restore_signals=True should've unblocked SIGPIPE and friends.

----------------------------------------------------------------------

None of the above is related to fork/vfork/posix_spawn -- the set of failing tests is the same in all four flavors.
History
Date User Action Args
2019-01-26 21:51:19izbyshevsetrecipients: + izbyshev, gregory.p.smith, ronaldoussoren, vstinner, koobs, pablogsal
2019-01-26 21:51:18izbyshevsetmessageid: <1548539478.29.0.204558480235.issue35823@roundup.psfhosted.org>
2019-01-26 21:51:18izbyshevlinkissue35823 messages
2019-01-26 21:51:17izbyshevcreate