classification
Title: test_subprocess: test_specific_shell() fails on AMD64 FreeBSD Shared 3.x
Type: Stage: resolved
Components: Tests Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: koobs, vstinner
Priority: normal Keywords:

Created on 2021-01-29 21:49 by vstinner, last changed 2021-03-17 20:57 by vstinner. This issue is now closed.

Messages (9)
msg385945 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-29 21:49
https://buildbot.python.org/all/#/builders/483/builds/758

FAIL: test_specific_shell (test.test_subprocess.POSIXProcessTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_subprocess.py", line 2286, in test_specific_shell
    self.assertEqual(p.stdout.read().strip(), bytes(sh, 'ascii'))
AssertionError: b'' != b'/usr/local/bin/bash'

AMD64 FreeBSD Shared 3.9 is also affected:
https://buildbot.python.org/all/#/builders/151/builds/283

And AMD64 FreeBSD Shared 3.8:
https://buildbot.python.org/all/#/builders/555/builds/197
msg386474 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-02-04 01:12
FreeBSD base was updated on the FreeBSD workers, between versions that changes the way the base curses library was built, and what it built.

The net results was that any software (from ports) linked against base curses needed to be rebuilt, and until then would fail to load with the following error:

  ld-elf.so.1: Shared object "libncursesw.so.6" not found, required by "foo"


The test suite should guard/test for the inability to execute (successfully load/run) the required commands/functions.

The following tests also failed for the same reason:

======================================================================
ERROR: test_deactivate_with_strict_bash_opts (test.test_venv.BasicTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_venv.py", line 391, in test_deactivate_with_strict_bash_opts
    out, err = check_output([bash, test_script])
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_venv.py", line 44, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/usr/local/bin/bash', '/var/tmp/tmprk2_yuoe/test_strict.sh']' returned non-zero exit status 1.

0:14:07 load avg: 1.31 Re-running test_gdb in verbose mode
test test_gdb crashed -- Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest.py", line 270, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/libregrtest/runtest.py", line 221, in _runtest_inner2
    the_module = importlib.import_module(abstest)
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_gdb.py", line 48, in <module>
    gdb_version, gdb_major_version, gdb_minor_version = get_gdb_version()
  File "/usr/home/buildbot/python/3.9.koobs-freebsd-564d/build/Lib/test/test_gdb.py", line 29, in get_gdb_version
    raise Exception(f"Command {' '.join(cmd)!r} failed "
Exception: Command 'gdb -nx --version' failed with exit code 1: stdout='' stderr='ld-elf.so.1: Shared object "libncursesw.so.6" not found, required by "gdb"\n'

Would you like me to rebuild bash/gdb to fix the error, or wait for the tests to be fixed so that they can handle the error condition?
msg386480 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-04 12:56
> Would you like me to rebuild bash/gdb to fix the error, or wait for the tests to be fixed so that they can handle the error condition?

I don't think that Python test suite should handle the case of a broken FreeBSD setup.

Could you either disable the buildbot worker or fix it?
msg386507 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-02-05 01:54
The failure mode is a generic "an external dependency expected by a test suite is not available or doesn't run", applying to many core Python test that invokes an external utilities, and not specific to OS.

As far as I'm aware, the best practice and recommendation for these cases is that 'tests should not test dependencies', and to skip/skipif these (external) dependencies are not available or fail to load/run.

Of course, the call is yours :)

I'll rebuild gdb/bash if you decide you don't want the error to remain to test improving the test suite.
msg386678 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-02-09 01:10
Rebuilding bash & gdb now to pickup new/updated ncurses library.

Still suggest picking up "subprocess.CalledProcessError" generically in test framework for tests that use subprocess/external commands
msg386704 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-09 10:40
> As far as I'm aware, the best practice and recommendation for these cases is that 'tests should not test dependencies', and to skip/skipif these (external) dependencies are not available or fail to load/run.

In theory, we should limit external dependencies. In practice, it's too convenient to make the assumption that some Unix programs are available and work properly.

But PRs are welcome to reduce dependencies to external tools ;-)
msg386706 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-02-09 10:49
Note: Not suggesting dep removals, just a try-except-skip block in the (hopefully) one place the sub-process handler in the test framework that runs then :)
msg386707 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2021-02-09 10:49
P.S gdb and bash have been rebuilt :)
msg388967 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-17 20:57
The two FreeBSD 3.x buildbot are back to green, I close the issue.
History
Date User Action Args
2021-03-17 20:57:11vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg388967

stage: resolved
2021-02-09 10:49:53koobssetmessages: + msg386707
2021-02-09 10:49:32koobssetmessages: + msg386706
2021-02-09 10:40:32vstinnersetmessages: + msg386704
2021-02-09 01:10:37koobssetmessages: + msg386678
2021-02-05 01:54:52koobssetmessages: + msg386507
2021-02-04 12:56:44vstinnersetmessages: + msg386480
2021-02-04 01:12:53koobssetmessages: + msg386474
2021-01-29 21:49:19vstinnersetnosy: + koobs
2021-01-29 21:49:00vstinnercreate