This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Multiple test modules fail to run if invoked directly
Type: behavior Stage: patch review
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, corona10, ethan.furman, jaraco, miss-islington, serhiy.storchaka, sobolevn, terry.reedy, zach.ware
Priority: normal Keywords: patch

Created on 2022-01-18 18:41 by sobolevn, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 30666 merged sobolevn, 2022-01-18 18:43
PR 30676 merged sobolevn, 2022-01-18 22:11
PR 30681 merged sobolevn, 2022-01-19 07:18
PR 30682 merged sobolevn, 2022-01-19 10:30
PR 30699 merged sobolevn, 2022-01-19 17:08
PR 30700 merged sobolevn, 2022-01-19 17:13
PR 30725 merged sobolevn, 2022-01-20 18:24
PR 30746 merged sobolevn, 2022-01-21 13:15
PR 30782 merged miss-islington, 2022-01-22 11:06
PR 30783 merged miss-islington, 2022-01-22 11:06
PR 30792 merged sobolevn, 2022-01-22 14:30
PR 30799 merged jaraco, 2022-01-22 19:06
PR 30800 closed sobolevn, 2022-01-22 19:17
PR 30802 open jaraco, 2022-01-22 19:46
PR 30803 open jaraco, 2022-01-22 19:49
Messages (18)
msg410879 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-01-18 18:41
While working on a similar issue for `test_typing.py` (https://bugs.python.org/issue46416) I wondered: how many other modules also have this problem?

So, after several hours of local testing, I got several problems:
- Lib/test/test_compileall.py
  Uses relative `.` import that cannot be resolved

- Lib/test/test_distutils.py
  Is missing `import unittest` to run `unittest.main()`

- Lib/test/test_dtrace.py
  Calls undefined `test_main()` instead of  `unittest.main()`

- Lib/test/test_tools/test_freeze.py
  Uses relative `.` import that cannot be resolved

- Lib/test/test_zipfile64.py
  Imports undefined `from test.support import TESTFN`

- Lib/unittest/test/test_program.py
  Uses relative `.` import that cannot be resolved

Probably there are other problems, because I haven't checked:
- Windows tests, because I don't have a Win machine
- `test_importlib`, because it has a lot of failure (I don't know how to fix them yet)
- Multiple other modules with slow tests
- Modules with doctests which assert full `__qualname__` with module names

To keep PRs reviews sane, I will include changes for several modules and split this big task of checking all test modules into several consecutive and rather simple pull requests. 

The first one with the problems described above is on its way!
msg410880 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2022-01-18 18:59
I suggest your grouping of PRs be by error type -- so have batches of "relative import" fixes and batches of "TESTFN" fixes, etc.
msg410881 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-01-18 19:28
New changeset 1292aa6db5bed889a3c87df443754fcae0177801 by Nikita Sobolev in branch 'main':
bpo-46425: Fix direct invocation of multiple test modules (GH-30666)
https://github.com/python/cpython/commit/1292aa6db5bed889a3c87df443754fcae0177801
msg410886 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-01-18 20:20
Thank you, Ethan. Good idea! Will do from the second PR.
Thanks for merging, Serhiy.
msg410887 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2022-01-18 20:31
Nikita, thanks for doing this.

Ethan's suggestion has two advantages. 1. It would fix most backport issues.  By containing a mix of changes, PR-30666 cannot be backported as is.  'unittest.main' should always backport unless there is a context conflict.  But 3.9 does not have os.helper and TESTFN is still in test.support in 3.9 and that change cannot backport.  (I did not check 3.10.)

2. It restricts the knowledge needed to review (and be willing to merge).  I could and would review and merge a patch with only unittest.main changes, but not, without checking the docs to learn about new changes, the mixture you posted.

I suggest you write a script to directly run each test.test_xyz file via subprocess and report to stdout.  The test for this issue is a clean report.  If you do so, I will run it on Windows in a fresh build of each of 3.9, 3.10, and 3.11.

Note that there are two possible commands: 'python -m test test_xyz' and 'python -m test.test_xyz'.  They are subtly different.  So I would add an option to do it each way.

With current 3.11, test_importlib runs fine on my Windows, with 1436 tests.
msg410911 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-01-18 22:59
Terry, yes I am using this script to run tests locally:

```bash
#!/usr/bin/env bash

set -e

while read line; do 
  ./python.exe "$line"
done < files.txt
```

And this one to populate `files.txt`:

```bash
find ./Lib -iname 'test*.py' -o -iname '*tests.py' -exec echo {} >> files.txt \;
```

---

There are several other problems that I have to fix manually. For example, module-level `raise unittest.case.SkipTest()`:

```
» ./python.exe Lib/test/test_winreg.py 
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/import_helper.py", line 77, in import_module
    return importlib.import_module(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1142, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'winreg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_winreg.py", line 11, in <module>
    import_helper.import_module('winreg', required_on=['win'])
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/import_helper.py", line 26, in _ignore_deprecated_imports
    yield
    ^^^^^
  File "/Users/sobolev/Desktop/cpython/Lib/test/support/import_helper.py", line 81, in import_module
    raise unittest.SkipTest(str(msg))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
unittest.case.SkipTest: No module named 'winreg'
```

I am not sure what is the proper solution to this one :thinking:

Probably, I will update my script for more advanced Python-based one and ignore this exception.

In the end, I expect to run this script without any failures. This is going to be a great result! 

---

I will also check that all possible modes `python -m test test_xyz` / `python -m test.test_xyz` / `python Lib/test/test_xyz.py` are supported.

---

Also, thank you for listing concrete problems with my first submission. The second one is bound to a single problem and a single submodule.
msg410924 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-01-19 07:24
Important note, in practice there's also a difference in how one calls some test module:

- python Lib/test/test_contextlib.py
- python ./Lib/test/test_contextlib.py

Example of a test failure because of that: https://github.com/python/cpython/pull/30681
msg410999 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2022-01-20 01:43
New changeset 8105dd24112509fab2eabfce5352afc41e3a34b6 by Nikita Sobolev in branch '3.9':
[3.9] bpo-46425: Fix direct invocation of multiple test modules (GH-30666) (GH-30700)
https://github.com/python/cpython/commit/8105dd24112509fab2eabfce5352afc41e3a34b6
msg411000 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2022-01-20 01:44
New changeset a6a088548063226233c08b8d35dde130746fdd10 by Nikita Sobolev in branch '3.10':
[3.10] bpo-46425: Fix direct invocation of multiple test modules (GH-30666) (GH-30699)
https://github.com/python/cpython/commit/a6a088548063226233c08b8d35dde130746fdd10
msg411088 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-01-21 07:36
New changeset 22f73bd9f1fc573d5c998f345b66c29f7ca6614d by Nikita Sobolev in branch 'main':
bpo-46425: Fix direct invocation of `test_contextlib` (GH-30681)
https://github.com/python/cpython/commit/22f73bd9f1fc573d5c998f345b66c29f7ca6614d
msg411247 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-01-22 11:06
New changeset 5a5340044ca98cbe6297668d91bccba04b102923 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of `asyncio` tests (#30725)
https://github.com/python/cpython/commit/5a5340044ca98cbe6297668d91bccba04b102923
msg411250 - (view) Author: miss-islington (miss-islington) Date: 2022-01-22 11:54
New changeset 3c4a3745b900e748f99e80fc3728b534e857d1ff by Miss Islington (bot) in branch '3.9':
bpo-46425: fix direct invocation of `asyncio` tests (GH-30725)
https://github.com/python/cpython/commit/3c4a3745b900e748f99e80fc3728b534e857d1ff
msg411252 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-01-22 12:29
New changeset 6111d5dee2b24916ff95dba56efc569396a31851 by Miss Islington (bot) in branch '3.10':
bpo-46425: fix direct invocation of `asyncio` tests (GH-30725) (#30782)
https://github.com/python/cpython/commit/6111d5dee2b24916ff95dba56efc569396a31851
msg411264 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-01-22 16:03
New changeset 101a184d49756043a0c39dde6eca08b1891137a2 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of `test_traceback` (GH-30746)
https://github.com/python/cpython/commit/101a184d49756043a0c39dde6eca08b1891137a2
msg411265 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-01-22 16:04
New changeset 55f4ec460ee6dcffc26180fd982ad89083c9acb1 by Nikita Sobolev in branch 'main':
bpo-46425: use absolute imports in `test_sqlite3` (GH-30676)
https://github.com/python/cpython/commit/55f4ec460ee6dcffc26180fd982ad89083c9acb1
msg411266 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-01-22 16:05
New changeset 57316c52bae5d6420f5067f3891ec328deb97305 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of `test_importlib` (GH-30682)
https://github.com/python/cpython/commit/57316c52bae5d6420f5067f3891ec328deb97305
msg411267 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2022-01-22 16:05
New changeset 1f8014c5b4ea7acee069ca453f6fbcad5990ebf0 by Nikita Sobolev in branch 'main':
bpo-46425: fix direct invocation of `test_fileutils` and `test_zoneinfo` (GH-30792)
https://github.com/python/cpython/commit/1f8014c5b4ea7acee069ca453f6fbcad5990ebf0
msg411334 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2022-01-23 02:38
New changeset d888ff5381594641126065e78dc9210dae4436a4 by Jason R. Coombs in branch 'main':
bpo-46425: Partially revert "bpo-46425: fix direct invocation of `test_importlib` (GH-30682)" (GH-30799)
https://github.com/python/cpython/commit/d888ff5381594641126065e78dc9210dae4436a4
History
Date User Action Args
2022-04-11 14:59:54adminsetgithub: 90583
2022-01-23 02:38:46jaracosetmessages: + msg411334
2022-01-22 19:49:52jaracosetpull_requests: + pull_request28990
2022-01-22 19:46:19jaracosetpull_requests: + pull_request28988
2022-01-22 19:17:23sobolevnsetpull_requests: + pull_request28985
2022-01-22 19:06:06jaracosetpull_requests: + pull_request28984
2022-01-22 18:36:18jaracosetnosy: + jaraco
2022-01-22 16:05:47asvetlovsetmessages: + msg411267
2022-01-22 16:05:08asvetlovsetmessages: + msg411266
2022-01-22 16:04:00asvetlovsetmessages: + msg411265
2022-01-22 16:03:17asvetlovsetmessages: + msg411264
2022-01-22 14:30:53sobolevnsetpull_requests: + pull_request28977
2022-01-22 12:29:54asvetlovsetmessages: + msg411252
2022-01-22 11:54:10miss-islingtonsetmessages: + msg411250
2022-01-22 11:06:48miss-islingtonsetpull_requests: + pull_request28968
2022-01-22 11:06:43miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request28967
2022-01-22 11:06:31asvetlovsetnosy: + asvetlov
messages: + msg411247
2022-01-21 13:15:12sobolevnsetpull_requests: + pull_request28933
2022-01-21 07:36:37serhiy.storchakasetmessages: + msg411088
2022-01-20 18:24:16sobolevnsetpull_requests: + pull_request28915
2022-01-20 01:44:25corona10setmessages: + msg411000
2022-01-20 01:43:57corona10setmessages: + msg410999
2022-01-19 17:13:55sobolevnsetpull_requests: + pull_request28898
2022-01-19 17:08:32sobolevnsetpull_requests: + pull_request28897
2022-01-19 10:30:28sobolevnsetpull_requests: + pull_request28881
2022-01-19 07:24:10sobolevnsetmessages: + msg410924
2022-01-19 07:18:19sobolevnsetpull_requests: + pull_request28880
2022-01-18 22:59:52sobolevnsetmessages: + msg410911
2022-01-18 22:11:52sobolevnsetpull_requests: + pull_request28876
2022-01-18 20:31:30terry.reedysetnosy: + terry.reedy
messages: + msg410887
2022-01-18 20:20:00sobolevnsetmessages: + msg410886
2022-01-18 19:28:35serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg410881
2022-01-18 18:59:15ethan.furmansetnosy: + ethan.furman
messages: + msg410880
2022-01-18 18:43:32sobolevnsetkeywords: + patch
stage: patch review
pull_requests: + pull_request28866
2022-01-18 18:42:58zach.waresetnosy: + zach.ware
2022-01-18 18:41:39sobolevncreate