classification
Title: test_lib2to3 fails since Python 3.9
Type: Stage: resolved
Components: Tests Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: federico.granata, felixonmars, miss-islington, pablogsal, xtreak
Priority: normal Keywords: patch

Created on 2020-10-07 23:05 by felixonmars, last changed 2020-10-09 10:09 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22595 merged pablogsal, 2020-10-08 13:01
PR 22609 merged xtreak, 2020-10-09 03:01
Messages (14)
msg378190 - (view) Author: Felix Yan (felixonmars) * Date: 2020-10-07 23:05
I am packaging Python for Arch and the tests suite of Python 3.8.6 pass here without this:

```
0:09:06 load avg: 0.87 [205/424] test_lib2to3                                                                                                                                                                                                                                                                         
test test_lib2to3 crashed -- Traceback (most recent call last):                                                                                                                                                                                                                                                       
  File "/build/python/src/Python-3.9.0/Lib/test/libregrtest/runtest.py", line 270, in _runtest_inner                                                                                                                                                                                                                  
    refleak = _runtest_inner2(ns, test_name)                                                                                                                                                                                                                                                                          
  File "/build/python/src/Python-3.9.0/Lib/test/libregrtest/runtest.py", line 221, in _runtest_inner2                                                                                                                                                                                                                 
    the_module = importlib.import_module(abstest)                                                                                                                                                                                                                                                                     
  File "/build/python/src/Python-3.9.0/Lib/importlib/__init__.py", line 127, in import_module                                                                                                                                                                                                                         
    return _bootstrap._gcd_import(name[level:], package, level)                                                                                                                                                                                                                                                       
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import                                                                                                                                                                                                                                                     
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load                                                                                                                                                                                                                                                  
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked                                                                                                                                                                                                                                          
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked                                                                                                                                                                                                                                                   
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module                                                                                                                                                                                                                                             
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed                                                                                                                                                                                                                                        
  File "/build/python/src/Python-3.9.0/Lib/test/test_lib2to3.py", line 5, in <module>                                                                                                                                                                                                                                 
    from lib2to3.tests import load_tests                                                                                                                                                                                                                                                                              
  File "/build/python/src/Python-3.9.0/Lib/contextlib.py", line 124, in __exit__                                                                                                                                                                                                                                      
    next(self.gen)                                                                                                                                                                                                                                                                                                    
  File "/build/python/src/Python-3.9.0/Lib/test/support/__init__.py", line 1171, in _filterwarnings                                                                                                                                                                                                                   
    raise AssertionError("filter (%r, %s) did not catch any warning" %                                                                                                                                                                                                                                                
AssertionError: filter ('', PendingDeprecationWarning) did not catch any warning                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                      
0:09:06 load avg: 0.87 [206/424/1] test_linecache -- test_lib2to3 failed    
```
msg378196 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-07 23:31
Same thing as in https://bugs.python.org/issue41971. I cannot reproduce this with a fresh install of 3.9 and our buildbots are not complaining about this.

ownloads/Python-3.9.0
❯ ./python -m test test_lib2to3
0:00:00 load avg: 3.64 Run tests sequentially
0:00:00 load avg: 3.64 [1/1] test_lib2to3

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.5 sec
Tests result: SUCCESS

-------

❯ ./lel/bin/python3 -m test test_lib2to3
0:00:00 load avg: 3.87 Run tests sequentially
0:00:00 load avg: 3.87 [1/1] test_lib2to3

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.4 sec
Tests result: SUCCESS
msg378205 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-10-08 00:15
I have also seen this and linked issue on running whole test suite while building python myself but running individual tests show no error and buildbots also never complained so I thought it's something wrong with my setup. I introduced the commit to silence deprecation warning due to lib2to3 and cannot reproduce the same running it individually.
msg378208 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-08 00:24
> I have also seen this and linked issue on running whole test suite while building python myself but running individual tests show no error and buildbots also never complained so I thought it's something wrong with my setup. I introduced the commit to silence deprecation warning due to lib2to3 and cannot reproduce the same running it individually.

What I don't understand then is how the buildbots or the CI in Github that run all tests sequentially with the extra flags (-uall and friends) have not complained yet about this.

@Felix, how consistent is this failure you are experiencing?
msg378232 - (view) Author: Federico Granata (federico.granata) Date: 2020-10-08 10:57
I just experienced the very same issue.
I use pyenv and doing all tests gives me a failure on test_lib2to3 while running it individually end succesfully.

Tell me if I can provide any useful information
msg378234 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-10-08 11:17
Pablo, I can see the error in Travis. The difference I can see is that the tests are executed sequentially in coverage run but normal run has tests executing in parallel. Felix and Federico, can you please add the command you use to run the tests?

Sample error build : https://travis-ci.com/github/python/cpython/jobs/396688564#L2022

coverage run but has true in the end thus being always successful

xvfb-run ./venv/bin/python -m coverage run --branch --pylib -m test --fail-env-changed -uall,-cpu -x test_multiprocessing_fork -x test_multiprocessing_forkserver -x test_multiprocessing_spawn -x test_concurrent_futures || true

Normal run

/home/travis/build/python/cpython/python -u -W default -bb -E -m test -r -w -j 1 -u all -W --slowest --fail-env-changed --timeout=1200 -j4 -uall,-cpu
msg378238 - (view) Author: Felix Yan (felixonmars) * Date: 2020-10-08 11:36
I use this in Arch's packaging:

LC_CTYPE=en_US.UTF-8 xvfb-run -s "-screen 0 1920x1080x16 -ac +extension GLX" -a -n "$servernum" \
    "${srcdir}/Python-${pkgver}/python" -m test.regrtest -v -uall -x test_tk

(test_tk is currently skipped due to https://bugs.python.org/issue41306)
msg378240 - (view) Author: Federico Granata (federico.granata) Date: 2020-10-08 12:02
I just run "python -m test" for it to fail and "python -m test test_lib2to3" for it to succeded.
msg378241 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-10-08 12:13
I can reproduce this by running test_lib2to3 twice or running test___all__ with test_lib2to3

./python -m test test_lib2to3 test_lib2to3
0:00:00 load avg: 0.03 Run tests sequentially
0:00:00 load avg: 0.03 [1/2] test_lib2to3
0:00:13 load avg: 0.25 [2/2] test_lib2to3
test test_lib2to3 crashed -- Traceback (most recent call last):
  File "/root/cpython/Lib/test/libregrtest/runtest.py", line 272, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
  File "/root/cpython/Lib/test/libregrtest/runtest.py", line 223, in _runtest_inner2
    the_module = importlib.import_module(abstest)
  File "/root/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/cpython/Lib/test/test_lib2to3.py", line 5, in <module>
    from lib2to3.tests import load_tests
  File "/root/cpython/Lib/contextlib.py", line 124, in __exit__
    next(self.gen)
  File "/root/cpython/Lib/test/support/warnings_helper.py", line 179, in _filterwarnings
    raise AssertionError("filter (%r, %s) did not catch any warning" %
AssertionError: filter ('', PendingDeprecationWarning) did not catch any warning

test_lib2to3 failed

== Tests result: FAILURE ==

1 test OK.

1 test failed:
    test_lib2to3

Total duration: 13.5 sec
Tests result: FAILURE

./python -m test test___all__ test_lib2to3     
0:00:00 load avg: 0.05 Run tests sequentially
0:00:00 load avg: 0.05 [1/2] test___all__
0:00:00 load avg: 0.05 [2/2] test_lib2to3
test test_lib2to3 crashed -- Traceback (most recent call last):
  File "/root/cpython/Lib/test/libregrtest/runtest.py", line 272, in _runtest_inner
    refleak = _runtest_inner2(ns, test_name)
  File "/root/cpython/Lib/test/libregrtest/runtest.py", line 223, in _runtest_inner2
    the_module = importlib.import_module(abstest)
  File "/root/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/root/cpython/Lib/test/test_lib2to3.py", line 5, in <module>
    from lib2to3.tests import load_tests
  File "/root/cpython/Lib/contextlib.py", line 124, in __exit__
    next(self.gen)
  File "/root/cpython/Lib/test/support/warnings_helper.py", line 179, in _filterwarnings
    raise AssertionError("filter (%r, %s) did not catch any warning" %
AssertionError: filter ('', PendingDeprecationWarning) did not catch any warning

test_lib2to3 failed

== Tests result: FAILURE ==

1 test OK.

1 test failed:
    test_lib2to3

Total duration: 1.1 sec
Tests result: FAILURE
msg378242 - (view) Author: Federico Granata (federico.granata) Date: 2020-10-08 12:17
I can have the test to fails if I run "python -m test test_lib2to3 test_lib2to3" as well
msg378245 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-08 12:41
Ok, thanks everyone for the help reproducing this. I think I can work on a fix today
msg378246 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-10-08 12:55
I guess this could be due to loading the module twice where the warning is not emitted again on reimport which test__all__ seems to do so by importing lib2to3. One obvious fix would be to pass quiet=True. Else we need to find a way to import the module fresh like using import_fresh_module. Two possible patches. I assumed running tests sequentially will provide support for isolation too.

    Optional argument:
     - if 'quiet' is True, it does not fail if a filter catches nothing
        (default True without argument,
         default False if some filters are defined)

    Without argument, it defaults to:
        check_warnings(("", Warning), quiet=True)

# Using import fresh module

diff --git a/Lib/test/test_lib2to3.py b/Lib/test/test_lib2to3.py
index 159a8387e4..861ae5ad53 100644
--- a/Lib/test/test_lib2to3.py
+++ b/Lib/test/test_lib2to3.py
@@ -1,7 +1,9 @@
 import unittest
+from test.support.import_helper import import_fresh_module
 from test.support.warnings_helper import check_warnings
 
 with check_warnings(("", PendingDeprecationWarning)):
+    lib2to3 = import_fresh_module("lib2to3")
     from lib2to3.tests import load_tests
 
 if __name__ == '__main__':

# Passing quiet=True

diff --git a/Lib/test/test_lib2to3.py b/Lib/test/test_lib2to3.py
index 159a8387e4..e4c5cade8b 100644
--- a/Lib/test/test_lib2to3.py
+++ b/Lib/test/test_lib2to3.py
@@ -1,7 +1,7 @@
 import unittest
 from test.support.warnings_helper import check_warnings
 
-with check_warnings(("", PendingDeprecationWarning)):
+with check_warnings(("", PendingDeprecationWarning), quiet=True):
     from lib2to3.tests import load_tests
 
 if __name__ == '__main__':

➜  cpython git:(master) ✗ ./python -m test test_lib2to3 test_lib2to3     
0:00:00 load avg: 0.01 Run tests sequentially
0:00:00 load avg: 0.01 [1/2] test_lib2to3
0:00:13 load avg: 0.23 [2/2] test_lib2to3

== Tests result: SUCCESS ==

All 2 tests OK.

Total duration: 27.3 sec
Tests result: SUCCESS
msg378249 - (view) Author: miss-islington (miss-islington) Date: 2020-10-08 13:24
New changeset 4a9f82f50d957b6cf3fd207de8b583d9137316b8 by Pablo Galindo in branch 'master':
bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (GH-22595)
https://github.com/python/cpython/commit/4a9f82f50d957b6cf3fd207de8b583d9137316b8
msg378310 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-09 10:08
New changeset c6f41e62f55933f9735e9b9c103c6860685b028a by Karthikeyan Singaravelan in branch '3.9':
[3.9] bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (GH-22595) (GH-22609)
https://github.com/python/cpython/commit/c6f41e62f55933f9735e9b9c103c6860685b028a
History
Date User Action Args
2020-10-09 10:09:01pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-10-09 10:08:49pablogsalsetmessages: + msg378310
2020-10-09 03:01:41xtreaksetpull_requests: + pull_request21592
2020-10-08 13:24:37miss-islingtonsetnosy: + miss-islington
messages: + msg378249
2020-10-08 13:01:15pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request21583
2020-10-08 12:55:15xtreaksetmessages: + msg378246
versions: + Python 3.10
2020-10-08 12:41:06pablogsalsetmessages: + msg378245
2020-10-08 12:17:11federico.granatasetmessages: + msg378242
2020-10-08 12:13:56xtreaksetmessages: + msg378241
2020-10-08 12:02:13federico.granatasetmessages: + msg378240
2020-10-08 11:36:07felixonmarssetmessages: + msg378238
2020-10-08 11:17:39xtreaksetmessages: + msg378234
2020-10-08 10:57:29federico.granatasetnosy: + federico.granata
messages: + msg378232
2020-10-08 00:24:41pablogsalsetmessages: + msg378208
2020-10-08 00:15:15xtreaksetnosy: + xtreak
messages: + msg378205
2020-10-07 23:31:12pablogsalsetnosy: + pablogsal
messages: + msg378196
2020-10-07 23:05:57felixonmarscreate