Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

Closed
RJ722 mannequin opened this issue Jun 29, 2020 · 8 comments
Closed

test_pkgutil:test_name_resolution fails on macOS HFS+ file systems #85326

RJ722 mannequin opened this issue Jun 29, 2020 · 8 comments
Labels
3.9 only security fixes 3.10 only security fixes build The build process and cross-build stdlib Python modules in the Lib dir tests Tests in the Lib/test dir

Comments

@RJ722
Copy link
Mannequin

RJ722 mannequin commented Jun 29, 2020

BPO 41154
Nosy @brettcannon, @vsajip, @ned-deily, @merwok, @zooba, @dstufft, @The-Compiler, @RJ722

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-03.18:05:45.937>
created_at = <Date 2020-06-29.06:56:41.028>
labels = ['tests', 'build', 'library', '3.9', '3.10']
title = 'test_pkgutil:test_name_resolution fails on macOS HFS+ file systems'
updated_at = <Date 2021-02-03.18:05:45.936>
user = 'https://github.com/RJ722'

bugs.python.org fields:

activity = <Date 2021-02-03.18:05:45.936>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-03.18:05:45.937>
closer = 'steve.dower'
components = ['Build', 'Distutils', 'Tests']
creation = <Date 2020-06-29.06:56:41.028>
creator = 'RJ722'
dependencies = []
files = []
hgrepos = []
issue_num = 41154
keywords = []
message_count = 8.0
messages = ['372549', '372550', '372551', '372603', '372624', '376141', '376165', '386244']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'vinay.sajip', 'ned.deily', 'eric.araujo', 'steve.dower', 'dstufft', 'The Compiler', 'RJ722']
pr_nums = []
priority = 'critical'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue41154'
versions = ['Python 3.9', 'Python 3.10']
@RJ722
Copy link
Mannequin Author

RJ722 mannequin commented Jun 29, 2020

After configuring and building using the command:

./configure --with-pydebug ** make -j

I ran the test suite (without changing anything) and saw that test_pkg has failed. Here is the output of ./python.exe -m test -v test_pkgutil:

== CPython 3.10.0a0 (heads/master:cd3c2bdd5d, Jun 28 2020, 13:29:09) [Clang 9.0.0 (clang-900.0.39.2)]
== macOS-10.12.6-x86_64-i386-64bit little-endian
== cwd: /Users/rahuljha/Documents/cpython/build/test_python_10678æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.01 Run tests sequentially
0:00:00 load avg: 2.01 [1/1] test_pkgutil
test_getdata_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_getdata_zipfile (test.test_pkgutil.PkgutilTests) ... ok
test_name_resolution (test.test_pkgutil.PkgutilTests) ... ERROR
test_unreadable_dir_on_syspath (test.test_pkgutil.PkgutilTests) ... ok
test_walk_packages_raises_on_string_or_bytes_input (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_zipfile (test.test_pkgutil.PkgutilTests)
Tests the same as test_walkpackages_filesys, only with a zip file. ... ok
test_alreadyloaded (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_getdata_pep302 (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_iter_importers (test.test_pkgutil.ExtendPathTests) ... ok
test_mixed_namespace (test.test_pkgutil.ExtendPathTests) ... ok
test_simple (test.test_pkgutil.ExtendPathTests) ... ok
test_nested (test.test_pkgutil.NestedNamespacePackageTest) ... ok
test_find_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_find_loader_missing_module (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_None_in_sys_modules (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_loader_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_spec_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_spec_attribute_none (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_importer_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_iter_importers_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_loader_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/rahuljha/Documents/cpython/Lib/test/test_pkgutil.py", line 262, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/Users/rahuljha/Documents/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

Ran 24 tests in 0.186s

FAILED (errors=1)
test test_pkgutil failed
test_pkgutil failed

== Tests result: FAILURE ==

1 test failed:
test_pkgutil

Total duration: 482 ms
Tests result: FAILURE
Py vulture ~/Documents/cpython master !1 ?1 ❯ ./python.exe -m test -v test_pkgutil
== CPython 3.10.0a0 (heads/master:cd3c2bdd5d, Jun 28 2020, 13:29:09) [Clang 9.0.0 (clang-900.0.39.2)]
== macOS-10.12.6-x86_64-i386-64bit little-endian
== cwd: /Users/rahuljha/Documents/cpython/build/test_python_21819æ
== CPU count: 4
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 2.69 Run tests sequentially
0:00:00 load avg: 2.69 [1/1] test_pkgutil
test_getdata_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_getdata_zipfile (test.test_pkgutil.PkgutilTests) ... ok
test_name_resolution (test.test_pkgutil.PkgutilTests) ... ERROR
test_unreadable_dir_on_syspath (test.test_pkgutil.PkgutilTests) ... ok
test_walk_packages_raises_on_string_or_bytes_input (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_filesys (test.test_pkgutil.PkgutilTests) ... ok
test_walkpackages_zipfile (test.test_pkgutil.PkgutilTests)
Tests the same as test_walkpackages_filesys, only with a zip file. ... ok
test_alreadyloaded (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_getdata_pep302 (test.test_pkgutil.PkgutilPEP302Tests) ... ok
test_iter_importers (test.test_pkgutil.ExtendPathTests) ... ok
test_mixed_namespace (test.test_pkgutil.ExtendPathTests) ... ok
test_simple (test.test_pkgutil.ExtendPathTests) ... ok
test_nested (test.test_pkgutil.NestedNamespacePackageTest) ... ok
test_find_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_find_loader_missing_module (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_importer_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_None_in_sys_modules (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_loader_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_missing_spec_attribute (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_get_loader_handles_spec_attribute_none (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_importer_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_iter_importers_avoids_emulation (test.test_pkgutil.ImportlibMigrationTests) ... ok
test_loader_deprecated (test.test_pkgutil.ImportlibMigrationTests) ... ok

======================================================================
ERROR: test_name_resolution (test.test_pkgutil.PkgutilTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/rahuljha/Documents/cpython/Lib/test/test_pkgutil.py", line 262, in test_name_resolution
    mod = importlib.import_module(uw)
  File "/Users/rahuljha/Documents/cpython/Lib/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'é'

Ran 24 tests in 0.184s

FAILED (errors=1)
test test_pkgutil failed
test_pkgutil failed

== Tests result: FAILURE ==

1 test failed:
test_pkgutil

Total duration: 635 ms
Tests result: FAILURE

@RJ722 RJ722 mannequin added 3.10 only security fixes build The build process and cross-build stdlib Python modules in the Lib dir tests Tests in the Lib/test dir labels Jun 29, 2020
@ned-deily
Copy link
Member

This same failure is observed with the most recent 3.9beta (3.9.0b3) when running on macOS 10.9 with the binaries from the python.org macOS installer. However, no failure is seen when running the same Python on macOS 10.15. I have not investigated further with other releases. This test failure should be addressed prior t0 3.9.0rc1.

@ned-deily ned-deily added 3.9 only security fixes deferred-blocker labels Jun 29, 2020
@ned-deily ned-deily changed the title test_pkgutil:test_name_resolution fails on master test_pkgutil:test_name_resolution fails on some platforms Jun 29, 2020
@ned-deily ned-deily added 3.9 only security fixes deferred-blocker labels Jun 29, 2020
@ned-deily ned-deily changed the title test_pkgutil:test_name_resolution fails on master test_pkgutil:test_name_resolution fails on some platforms Jun 29, 2020
@ned-deily
Copy link
Member

PS, the problem may have been introduced by the changes for bpo-12915.

@vsajip
Copy link
Member

vsajip commented Jun 29, 2020

The problem appears to be in importlib. I put in a breakpoint in the test just before it tries to import the module é. (At this point it has successfully imported a module called वमस in an earlier loop iteration.)

Then, in a different terminal console, I cd'd to the test directory and tried the import directly:
$ ls -l
total 0
drwxr-xr-x  3 xxxxxxxx  xxxxx  102 29 Jun 17:55 é
drwxr-xr-x  4 xxxxxxxx  xxxxx  136 29 Jun 17:55 वमस
$ ~/projects/cpython/python.exe
Python 3.10.0a0 (heads/master:a3ad95d, Jun 29 2020, 17:50:09)
[Clang 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import वमस
>>> import é
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'é'

So for some reason, importlib is failing to import the é module. (The é/init.py is a zero-byte file.)

Adding Brett to nosy.

@brettcannon
Copy link
Member

I wonder if it's a problem from Unicode representation since I don't think importlib tries to have a standard normalization form of Unicode. I could see that leading to how the file system encoded something differing from how it was encoded in the test file, leading to a claim the file doesn't exist.

And thanks to binary file paths that isn't necessarily a wrong result. :P

@ned-deily
Copy link
Member

I wonder if it's a problem from Unicode representation

That does seem to be the issue. In particular, there are differences in Unicode representation between file names on traditional HFS+ versus the newer APFS. APFS is the default for system file systems as of macOS 10.13; unlike HFS+, APFS is by default file name normalization insensitive. By creating two disk image files systems on a current macOS system, it's easy to demonstrate that test_name_resolution passes on APFS but fails on HFS+. So the question becomes what to do about the test. Without digging into it further, I suppose we could add something to test whether the file system where test temp files are created is normalization insensitive and, if so, skip. Or use brute force and always skip on macOS. In any case, while annoying, it probably doesn't need to be a "deferred blocker".

@ned-deily ned-deily changed the title test_pkgutil:test_name_resolution fails on some platforms test_pkgutil:test_name_resolution fails on macOS HFS+ file systems Aug 31, 2020
@ned-deily ned-deily changed the title test_pkgutil:test_name_resolution fails on some platforms test_pkgutil:test_name_resolution fails on macOS HFS+ file systems Aug 31, 2020
@brettcannon
Copy link
Member

I think if we can craft a skip check that is targeted enough then that's our best solution, but barring that then skipping for all of macOS wouldn't be the end of the world (maybe a slightly less brute force is a macOS version check?).

@zooba
Copy link
Member

zooba commented Feb 3, 2021

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools

@zooba zooba closed this as completed Feb 3, 2021
@zooba zooba closed this as completed Feb 3, 2021
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@dmacks dmacks mentioned this issue Feb 18, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.9 only security fixes 3.10 only security fixes build The build process and cross-build stdlib Python modules in the Lib dir tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@brettcannon @vsajip @zooba @ned-deily