This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_bdb and test_distutils fail on installed Python 3.9, 3.10 and 3.11 (setuptools 60.9.3, pip 22.0.4)
Type: Stage:
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: jaraco, ned.deily, vstinner
Priority: normal Keywords:

Created on 2022-03-14 16:14 by vstinner, last changed 2022-04-11 14:59 by admin.

Messages (8)
msg415157 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-14 16:14
Since build 298, test_bdb test_distutils fail on s390x Fedora Rawhide Clang Installed 3.10:
https://buildbot.python.org/all/#/builders/694/builds/298

2 changes of build 298:

* bpo-46986: Upgrade bundled setuptools to 60.9.3 (GH-31820)(21 hours ago)
* [3.10] bpo-46985: Upgrade bundled pip to 22.0.4 (GH-31819) (GH-31849)(21 hours ago)

Logs:

0:02:03 load avg: 9.95 [213/427/1] test_distutils failed (uncaught exception) -- running: test_concurrent_futures (42.4 sec)
test test_distutils crashed -- Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/libregrtest/runtest.py", line 335, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/libregrtest/runtest.py", line 280, in _runtest_inner2
    the_module = importlib.import_module(abstest)
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_distutils.py", line 15, in <module>
    import distutils.tests
ModuleNotFoundError: No module named 'distutils.tests'

======================================================================
FAIL: test_skip (test.test_bdb.StateTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 730, in test_skip
    with TracerRun(self, skip=skip) as tracer:
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 448, in __exit__
    self.test_case.fail(err_msg)
  File "/home/dje/cpython-buildarea/3.10.edelsohn-fedora-rawhide-z.clang-installed/build/target/lib/python3.10/test/test_bdb.py", line 582, in fail
    raise self.failureException(msg) from None
AssertionError: Wrong event type at expect_set item 2, got 'call'
  Expected: ('line', 3, 'tfunc_import')
  Got:      ('call', 84, 'find_spec'),      ('quit',),
msg415158 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-14 16:15
Reproduce:

$ ./configure --prefix /opt/py310 && make clean && make && make install 
$ cd /somewhere/else/
$ /opt/py310/bin/python3 -m test -v test_bdb
...
FAIL: test_skip (test.test_bdb.StateTestCase)
...
$ /opt/py310/bin/python3 -m test -v test_distutils 
...
ModuleNotFoundError: No module named 'distutils.tests'
...

setuptools installs this file:

$ cat /opt/py310/lib/python3.10/site-packages/distutils-precedence.pth
import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();
msg415160 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-14 16:18
See: [BUG] Having setuptools installed causes cpython stdlib build to fail
https://github.com/pypa/setuptools/issues/3007
msg415161 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-14 16:21
x86 Gentoo Installed with X 3.9:
https://buildbot.python.org/all/#builders/527/builds/474

4 tests failed:
    test_bdb test_distutils test_importlib test_peg_generator

Python 3.9 gets two more issues: test_importlib and test_peg_generator.

======================================================================
FAIL: test_package_discovery (test.test_importlib.test_main.DiscoveryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_importlib/test_main.py", line 159, in test_package_discovery
    assert all(
AssertionError

0:06:27 load avg: 3.96 [ 83/425/3] test_peg_generator failed (uncaught exception) -- running: test_multiprocessing_forkserver (1 min 27 sec)
Warning -- warnings.filters was modified by test_peg_generator
Failed to import test module: test.test_peg_generator.test_c_parser
Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.installed/build/target/lib/python3.9/test/test_peg_generator/test_c_parser.py", line 4, in <module>
    from distutils.tests.support import TempdirManager
ModuleNotFoundError: No module named 'distutils.tests'
msg415162 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-14 16:23
AMD64 Fedora Stable Clang Installed 3.x:
https://buildbot.python.org/all/#builders/350/builds/1621

2 re-run tests:
    test_bdb test_distutils
msg415164 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-14 16:26
> $ ./configure --prefix /opt/py310 && make clean && make && make install 
> $ cd /somewhere/else/
> $ /opt/py310/bin/python3 -m test -v test_bdb

I also reproduce test_bdb and test_distutils failures on the main branch.
msg415180 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2022-03-14 20:05
(The buildbot failures were triggered by yesterday's merges for Issue46986 which updated the bundled setuptools in ensurepip. I'm reverting those merges now to unblock planned releases.)
msg415311 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2022-03-16 01:15
See https://github.com/pypa/setuptools/issues/3007#issuecomment-1068621865 where I did a brief analysis and probable explanation.

What it boils down to: Setuptools needs to supply its own copy of distutils aggressively. It provides an opt out for this behavior and is able to offer built in opt out based on whatever signal is available. The default opt out is the SETUPTOOLS_USE_DISTUTILS=stdlib env var. for CPython, it also disables the behavior when ./pybuilddir.txt is present, but installed Pythons probably don’t have this config.

Can those test runners be updated to set this environment variable?
History
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91169
2022-03-16 01:15:39jaracosetmessages: + msg415311
2022-03-14 20:05:08ned.deilysetnosy: + jaraco, ned.deily

messages: + msg415180
versions: + Python 3.7, Python 3.8
2022-03-14 16:26:11vstinnersetmessages: + msg415164
2022-03-14 16:23:16vstinnersettitle: test_bdb and test_distutils fail on installed Python 3.9 and 3.10 (setuptools 60.9.3, pip 22.0.4) -> test_bdb and test_distutils fail on installed Python 3.9, 3.10 and 3.11 (setuptools 60.9.3, pip 22.0.4)
messages: + msg415162
versions: + Python 3.9, Python 3.11
2022-03-14 16:21:51vstinnersetmessages: + msg415161
title: 3.10: test_bdb test_distutils fail on s390x Fedora Rawhide Clang Installed 3.10 -> test_bdb and test_distutils fail on installed Python 3.9 and 3.10 (setuptools 60.9.3, pip 22.0.4)
2022-03-14 16:18:47vstinnersetmessages: + msg415160
2022-03-14 16:15:17vstinnersetmessages: + msg415158
2022-03-14 16:14:07vstinnercreate