classification
Title: multiple tests in test_tools fail since Python 3.9
Type: Stage: resolved
Components: Tests Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: corona10, eric.snow, felixonmars, pablogsal, xtreak, yan12125
Priority: normal Keywords: patch

Created on 2020-10-07 23:07 by felixonmars, last changed 2020-10-11 20:54 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22652 merged pablogsal, 2020-10-11 19:20
PR 22653 merged miss-islington, 2020-10-11 20:35
Messages (7)
msg378191 - (view) Author: Felix Yan (felixonmars) * Date: 2020-10-07 23:07
I am packaging Python for Arch and the test suite of Python 3.8.6 passes here without these:

======================================================================
FAIL: test_multiple_roots (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 118, in test_multiple_roots
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_w[26 chars]tion walk at 0x7ff2eb562f70>)), ('_walk', ('eg[42 chars]0>))] != [('_w[26 chars]tion _walk_tree at 0x7ff2e5babdc0>)), ('_walk'[54 chars]0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>)),
?                                                     ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>)),
?                                     +    +++++           + ^^^^

-  ('_walk', ('eggs', '.c', <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+  ('_walk', ('eggs', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_multiple_suffixes (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 189, in test_multiple_suffixes
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_no_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests) [None]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 209, in test_no_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_no_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests) []
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 209, in test_no_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '', <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', '', <function walk at 0x7ff2eb562f70>))]
?                                                   ^^^^^

+ [('_walk', ('spam', '', <function _walk_tree at 0x7ff2e5babdc0>))]
?                                   +    +++++           + ^^^^


======================================================================
FAIL: test_no_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests) [()]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 209, in test_no_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', (), <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', (), <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', (), <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', (), <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', (), <function walk at 0x7ff2eb562f70>))]
?                                                   ^^^^^

+ [('_walk', ('spam', (), <function _walk_tree at 0x7ff2e5babdc0>))]
?                                   +    +++++           + ^^^^


======================================================================
FAIL: test_one_root (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 99, in test_one_root
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_one_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 167, in test_one_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_relparent (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 228, in test_relparent
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_w[33 chars]tion walk at 0x7ff2eb562f70>)), ('_walk', ('/x[49 chars]0>))] != [('_w[33 chars]tion _walk_tree at 0x7ff2e5babdc0>)), ('_walk'[61 chars]0>))]

First differing element 0:
('_walk', ('/x/y/z/spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('/x/y/z/spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('/x/y/z/spam', '.c', <function walk at 0x7ff2eb562f70>)),
?                                                            ^^^^^

+ [('_walk', ('/x/y/z/spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>)),
?                                            +    +++++           + ^^^^

-  ('_walk', ('/x/y/z/eggs', '.c', <function walk at 0x7ff2eb562f70>))]
?                                                            ^^^^^

+  ('_walk', ('/x/y/z/eggs', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
?                                            +    +++++           + ^^^^


test test_tools failed
======================================================================
FAIL: test_single_root (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 82, in test_single_root
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_single_suffix (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 146, in test_single_suffix
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))] != [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]

First differing element 0:
('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', '.c', <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+ [('_walk', ('spam', '.c', <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


======================================================================
FAIL: test_typical (test.test_tools.test_c_analyzer.test_common.test_files.IterFilesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python/src/Python-3.9.0/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py", line 64, in test_typical
    self.assertEqual(self.calls, [
AssertionError: Lists differ: [('_w[26 chars]tion walk at 0x7ff2eb562f70>)), ('_walk', ('eg[42 chars]0>))] != [('_w[26 chars]tion _walk_tree at 0x7ff2e5babdc0>)), ('_walk'[54 chars]0>))]

First differing element 0:
('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>))
('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>))

- [('_walk', ('spam', None, <function walk at 0x7ff2eb562f70>)),
?                                                     ^^^^^

+ [('_walk', ('spam', None, <function _walk_tree at 0x7ff2e5babdc0>)),
?                                     +    +++++           + ^^^^

-  ('_walk', ('eggs', None, <function walk at 0x7ff2eb562f70>))]
?                                                     ^^^^^

+  ('_walk', ('eggs', None, <function _walk_tree at 0x7ff2e5babdc0>))]
?                                     +    +++++           + ^^^^


----------------------------------------------------------------------

Ran 184 tests in 3.357s

FAILED (failures=11, skipped=2, expected failures=14)
0:25:39 load avg: 0.95 [362/424/3] test_trace -- test_tools failed
msg378194 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-07 23:29
All our buildbots are running these test without any problem and I cannot reproduce this locally after a fresh install from the binaries in 3.9, so my guess is that there must be something going on on your end.

❯ ./installed_dir/bin/python3 -m test test_tools
0:00:00 load avg: 6.90 Run tests sequentially
0:00:00 load avg: 6.90 [1/1] test_tools

== Tests result: SUCCESS ==

1 test OK.

Total duration: 40 ms
Tests result: SUCCESS

~/Downloads/Python-3.9.0
❯ ./python -m test test_tools
0:00:00 load avg: 6.43 Run tests sequentially
0:00:00 load avg: 6.43 [1/1] test_tools

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.5 sec
Tests result: SUCCESS
msg378438 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2020-10-11 14:37
Looks like those failures in test_tools happen only when test_check_c_globals is imported and a test in test_imp is run.

```
$ cat matchfile
test.test_tools.*
test.test_imp.ImportTests.test_load_from_source

$ ./python -m test --matchfile matchfile test_check_c_globals test_imp test_tools
0:00:00 load avg: 0.90 Run tests sequentially
0:00:00 load avg: 0.90 [1/3] test_check_c_globals
0:00:00 load avg: 0.90 [2/3] test_imp -- test_check_c_globals run no tests
0:00:00 load avg: 0.90 [3/3] test_tools
test test_tools failed -- multiple errors occurred; run in verbose mode for details
test_tools failed

== Tests result: FAILURE ==

1 test OK.

1 test failed:
    test_tools

1 test run no tests:
    test_check_c_globals

Total duration: 1.9 sec
Tests result: FAILURE
```

If I remove test_check_c_globals from the second command or test.test_imp.ImportTests.test_load_from_source from matchfile, test_tools does not fail.
msg378446 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-11 18:52
Adding Eric Snow as this seems to be something going on on the c_analyzer
msg378450 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-11 19:19
Seems that the problem is that this check:

https://github.com/python/cpython/blob/master/Tools/c-analyzer/c_analyzer/common/files.py#L90

is done using a default value for "get_files":

https://github.com/python/cpython/blob/master/Tools/c-analyzer/c_analyzer/common/files.py#L63

But the import from source creates a new function for os.walk and therefore the comparison fails.
msg378453 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-11 20:34
New changeset 47ecfd8030c391779dd9d3b4b07e6b01a8392481 by Pablo Galindo in branch 'master':
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652)
https://github.com/python/cpython/commit/47ecfd8030c391779dd9d3b4b07e6b01a8392481
msg378454 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-11 20:54
New changeset 4af672921949b8a10340d5e29b456091060cedeb by Miss Skeleton (bot) in branch '3.9':
bpo-41971: Fix test failure in test.test_tools.test_c_analyzer when mutating global state (GH-22652) (GH-22653)
https://github.com/python/cpython/commit/4af672921949b8a10340d5e29b456091060cedeb
History
Date User Action Args
2020-10-11 20:54:42pablogsalsetmessages: + msg378454
2020-10-11 20:35:33pablogsalsetstatus: open -> closed
nosy: - miss-islington

resolution: fixed
stage: patch review -> resolved
2020-10-11 20:35:10miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request21626
2020-10-11 20:34:59pablogsalsetmessages: + msg378453
2020-10-11 19:20:45pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request21625
2020-10-11 19:19:44pablogsalsetmessages: + msg378450
2020-10-11 18:52:15pablogsalsetnosy: + eric.snow
messages: + msg378446
2020-10-11 14:37:35yan12125setmessages: + msg378438
2020-10-11 07:29:14yan12125setnosy: + yan12125
2020-10-08 13:55:45corona10setnosy: + corona10
2020-10-08 00:16:00xtreaksetnosy: + xtreak
2020-10-07 23:29:17pablogsalsetnosy: + pablogsal
messages: + msg378194
2020-10-07 23:07:56felixonmarscreate