classification
Title: test_pkgutil:test_name_resolution fails on macOS HFS+ file systems
Type: Stage:
Components: Build, Distutils, Tests Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: RJ722, The Compiler, brett.cannon, dstufft, eric.araujo, ned.deily, vinay.sajip
Priority: critical Keywords:

Created on 2020-06-29 06:56 by RJ722, last changed 2020-08-31 19:57 by brett.cannon.

Messages (7)
msg372549 - (view) Author: Rahul Jha (RJ722) * Date: 2020-06-29 06:56
After configuring and building using the command:

    ./configure --with-pydebug ** make -j

I ran the test suite (without changing anything) and saw that test_pkg has failed.  Here is the output of `./python.exe -m test -v test_pkgutil`:

== CPython 3.10.0a0 (heads/master:cd3c2bdd5d, Jun 28 2020, 13:29:09) [Clang 9.0.0 (clang-900.0.39.2)]
== macOS-10.12.6-x86_64-i386-64bit little-endian
== cwd: /Users/rahuljha/Documents/cpython/build/test_python_10678æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.01 Run tests sequentially
0:00:00 load avg: 2.01 [1/1] test_pkgutil
test_getdata_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_getdata_zipfile (test.test_pkgutil.PkgutilTests) ... ok
test_name_resolution (test.test_pkgutil.PkgutilTests) ... ERROR
test_unreadable_dir_on_syspath (test.test_pkgutil.PkgutilTests) ... ok
test_walk_packages_raises_on_string_or_bytes_input (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_zipfile (test.test_pkgutil.PkgutilTests)
Tests the same as test_walkpackages_filesys, only with a zip file. ... ok
test_alreadyloaded (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_getdata_pep302 (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_iter_importers (test.test_pkgutil.ExtendPathTests) ... ok
test_mixed_namespace (test.test_pkgutil.ExtendPathTests) ... ok
test_simple (test.test_pkgutil.ExtendPathTests) ... ok
test_nested (test.test_pkgutil.NestedNamespacePackageTest) ... ok
test_find_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_find_loader_missing_module (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_None_in_sys_modules (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_loader_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_spec_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_spec_attribute_none (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_importer_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_iter_importers_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_loader_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rahuljha/Documents/cpython/Lib/test/test_pkgutil.py", line 262, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/Users/rahuljha/Documents/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

----------------------------------------------------------------------

Ran 24 tests in 0.186s

FAILED (errors=1)
test test_pkgutil failed
test_pkgutil failed

== Tests result: FAILURE ==

1 test failed:
    test_pkgutil

Total duration: 482 ms
Tests result: FAILURE
Py vulture ~/Documents/cpython master !1 ?1 ❯ ./python.exe -m test -v test_pkgutil
== CPython 3.10.0a0 (heads/master:cd3c2bdd5d, Jun 28 2020, 13:29:09) [Clang 9.0.0 (clang-900.0.39.2)]
== macOS-10.12.6-x86_64-i386-64bit little-endian
== cwd: /Users/rahuljha/Documents/cpython/build/test_python_21819æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.69 Run tests sequentially
0:00:00 load avg: 2.69 [1/1] test_pkgutil
test_getdata_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_getdata_zipfile (test.test_pkgutil.PkgutilTests) ... ok
test_name_resolution (test.test_pkgutil.PkgutilTests) ... ERROR
test_unreadable_dir_on_syspath (test.test_pkgutil.PkgutilTests) ... ok
test_walk_packages_raises_on_string_or_bytes_input (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_zipfile (test.test_pkgutil.PkgutilTests)
Tests the same as test_walkpackages_filesys, only with a zip file. ... ok
test_alreadyloaded (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_getdata_pep302 (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_iter_importers (test.test_pkgutil.ExtendPathTests) ... ok
test_mixed_namespace (test.test_pkgutil.ExtendPathTests) ... ok
test_simple (test.test_pkgutil.ExtendPathTests) ... ok
test_nested (test.test_pkgutil.NestedNamespacePackageTest) ... ok
test_find_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_find_loader_missing_module (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_None_in_sys_modules (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_loader_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_spec_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_spec_attribute_none (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_importer_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_iter_importers_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_loader_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rahuljha/Documents/cpython/Lib/test/test_pkgutil.py", line 262, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/Users/rahuljha/Documents/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

----------------------------------------------------------------------

Ran 24 tests in 0.184s

FAILED (errors=1)
test test_pkgutil failed
test_pkgutil failed

== Tests result: FAILURE ==

1 test failed:
    test_pkgutil

Total duration: 635 ms
Tests result: FAILURE
msg372550 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 07:24
This same failure is observed with the most recent 3.9beta (3.9.0b3) when running on macOS 10.9 with the binaries from the python.org macOS installer.  However, no failure is seen when running the same Python on macOS 10.15. I have not investigated further with other releases.  This test failure should be addressed prior t0 3.9.0rc1.
msg372551 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-29 07:25
PS, the problem may have been introduced by the changes for bpo-12915.
msg372603 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2020-06-29 17:06
The problem appears to be in importlib. I put in a breakpoint in the test just before it tries to import the module é. (At this point it has successfully imported a module called वमस in an earlier loop iteration.)

Then, in a different terminal console, I cd'd to the test directory and tried the import directly:
$ ls -l
total 0
drwxr-xr-x  3 xxxxxxxx  xxxxx  102 29 Jun 17:55 é
drwxr-xr-x  4 xxxxxxxx  xxxxx  136 29 Jun 17:55 वमस
$ ~/projects/cpython/python.exe
Python 3.10.0a0 (heads/master:a3ad95d, Jun 29 2020, 17:50:09)
[Clang 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import वमस
>>> import é
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'é'

So for some reason, importlib is failing to import the é module. (The é/__init__.py is a zero-byte file.)

Adding Brett to nosy.
msg372624 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-06-29 19:46
I wonder if it's a problem from Unicode representation since I don't think importlib tries to have a standard normalization form of Unicode. I could see that leading to how the file system encoded something differing from how it was encoded in the test file, leading to a claim the file doesn't exist.

And thanks to binary file paths that isn't necessarily a wrong result. :P
msg376141 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-08-31 07:52
> I wonder if it's a problem from Unicode representation

That does seem to be the issue. In particular, there are differences in Unicode representation between file names on traditional HFS+ versus the newer APFS. APFS is the default for system file systems as of macOS 10.13; unlike HFS+, APFS is by default file name normalization insensitive. By creating two disk image files systems on a current macOS system, it's easy to demonstrate that test_name_resolution passes on APFS but fails on HFS+.  So the question becomes what to do about the test.  Without digging into it further, I suppose we could add something to test whether the file system where test temp files are created is normalization insensitive and, if so, skip. Or use brute force and always skip on macOS. In any case, while annoying, it probably doesn't need to be a "deferred blocker".
msg376165 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-08-31 19:57
I think if we can craft a skip check that is targeted enough then that's our best solution, but barring that then skipping for all of macOS wouldn't be the end of the world (maybe a slightly less brute force is a macOS version check?).
History
Date User Action Args
2020-08-31 19:57:06brett.cannonsetmessages: + msg376165
2020-08-31 07:52:58ned.deilysetpriority: deferred blocker -> critical

messages: + msg376141
title: test_pkgutil:test_name_resolution fails on some platforms -> test_pkgutil:test_name_resolution fails on macOS HFS+ file systems
2020-06-29 19:46:05brett.cannonsetmessages: + msg372624
2020-06-29 17:06:04vinay.sajipsetnosy: + brett.cannon
messages: + msg372603
2020-06-29 10:38:57vstinnersetnosy: - vstinner
2020-06-29 07:34:25The Compilersetnosy: + The Compiler
2020-06-29 07:25:37ned.deilysetmessages: + msg372551
2020-06-29 07:24:18ned.deilysetpriority: normal -> deferred blocker
title: test_pkgutil:test_name_resolution fails on master -> test_pkgutil:test_name_resolution fails on some platforms
nosy: + ned.deily, vstinner

messages: + msg372550

versions: + Python 3.9
2020-06-29 06:56:41RJ722create