classification
Title: New `files()` api from importlib_resources.
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: jaraco Nosy List: Michael.Felt, barry, benthayer, jaraco, lukasz.langa, miss-islington, ned.deily, vstinner
Priority: normal Keywords: patch

Created on 2020-02-29 04:11 by jaraco, last changed 2020-06-17 14:39 by vstinner.

Pull Requests
URL Status Linked Edit
PR 19722 merged jaraco, 2020-04-26 22:01
PR 20576 merged jaraco, 2020-06-01 16:33
PR 20656 closed jaraco, 2020-06-05 19:08
PR 20659 merged jaraco, 2020-06-05 19:28
PR 20661 merged miss-islington, 2020-06-05 20:34
PR 20662 closed miss-islington, 2020-06-05 20:34
PR 20681 merged jaraco, 2020-06-06 17:10
PR 20695 merged miss-islington, 2020-06-07 14:58
PR 20703 merged miss-islington, 2020-06-08 01:01
PR 20760 merged lukasz.langa, 2020-06-09 12:53
PR 20938 vstinner, 2020-06-17 14:39
Messages (16)
msg362962 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-02-29 04:11
In the [importlib_resources backport](https://gitlab.com/python-devs/importlib_resources/)... in particular in [issue 58](https://gitlab.com/python-devs/importlib_resources/issues/58) and [merge request 76](https://gitlab.com/python-devs/importlib_resources/-/merge_requests/76), the backport now has a new feature, a "files()" function.

Let's incorporate that functionality into importlib.resources.
msg363556 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-03-06 21:27
The latest release, 1.3.0, includes extensibility support and has been merged with the cpython branch of the importlib_resources project. I believe that code is now synced with this project and ready to be applied here. I'm hoping benthayer can apply the changes and submit the pr.
msg368484 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-05-08 23:20
New changeset 7f7e706d78ab968a1221c6179dfdba714860bd12 by Jason R. Coombs in branch 'master':
bpo-39791: Add files() to importlib.resources (GH-19722)
https://github.com/python/cpython/commit/7f7e706d78ab968a1221c6179dfdba714860bd12
msg368490 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-05-08 23:59
I've merged PR 19722. Some follow up actions I'd like to do:

- Add hooks for `.files()` on built-in loaders.
- Replace `loader.get_resource_reader()` with adapters around `.files()` for built-in loaders.
msg370782 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-05 20:34
New changeset 161541ab45278df6603dd870113b10f13e4d9e16 by Jason R. Coombs in branch 'master':
bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659)
https://github.com/python/cpython/commit/161541ab45278df6603dd870113b10f13e4d9e16
msg370790 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-05 21:46
New changeset a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb by Miss Islington (bot) in branch '3.9':
bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) (GH-20661)
https://github.com/python/cpython/commit/a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb
msg370825 - (view) Author: Michael Felt (Michael.Felt) * Date: 2020-06-06 15:12
The 'fancy' file name breaks on latin-1 character set.

aixtools@gcc119:[/home/aixtools/python/py39-3.9] a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb Lib/test/test_importlib/test_main.py                   <
diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py
index 42a7999..7b18c3d 100644
--- a/Lib/test/test_importlib/test_main.py
+++ b/Lib/test/test_importlib/test_main.py
@@ -246,3 +246,19 @@ class TestEntryPoints(unittest.TestCase):
         """
         with self.assertRaises(Exception):
             json.dumps(self.ep)
+
+    def test_module(self):
+        assert self.ep.module == 'value'
+
+    def test_attr(self):
+        assert self.ep.attr is None
+
+
+class FileSystem(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
+    def test_unicode_dir_on_sys_path(self):
+        """
+        Ensure a Unicode subdirectory of a directory on sys.path
+        does not crash.
+        """
+        fixtures.build_files({'☃': {}}, prefix=self.site_dir)
+        list(distributions())

AIX bots fail with:

Captured traceback
==================

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/test_main.py", line 263, in test_unicode_dir_on_sys_path
    fixtures.build_files({'\u2603': {}}, prefix=self.site_dir)
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/fixtures.py", line 202, in build_files
    full_name.mkdir()
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/pathlib.py", line 1309, in mkdir
    self._accessor.mkdir(self, mode)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2603' in position 17: ordinal not in range(256)



Test report
===========

Failed tests:

- test_importlib
msg370834 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-06 16:38
Thanks for the report Michael. I'm trying to figure out the best way to address the issue. That test is shared with importlib_metadata, so needs to run without CPython's test suite fixtures, such as the ones that generate non-ascii filenames. I'm tempted to just attempt to create the fixture and skip if the fixture fails. An alternate approach might be to attempt to load CPython's fixture, skip if that value is None, and fallback to the snowman otherwise.
msg370840 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-06 17:12
See GH-20681 for a proposed fix. I've scheduled the build bots to run the patch. Will build bots prove the fix? If not, can you test the patch in the same environment where it was discovered?
msg370903 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-07 14:57
New changeset 2efe18bf277dd0f38a1d248ae6bdd30947c26880 by Jason R. Coombs in branch 'master':
bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (#20681)
https://github.com/python/cpython/commit/2efe18bf277dd0f38a1d248ae6bdd30947c26880
msg370904 - (view) Author: miss-islington (miss-islington) Date: 2020-06-07 15:17
New changeset 71f501698d64af6463246bc6efdbbf1b85616102 by Miss Islington (bot) in branch '3.9':
bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (GH-20681)
https://github.com/python/cpython/commit/71f501698d64af6463246bc6efdbbf1b85616102
msg370945 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2020-06-08 01:00
New changeset 843c27765652e2322011fb3e5d88f4837de38c06 by Jason R. Coombs in branch 'master':
bpo-39791 native hooks for importlib.resources.files (GH-20576)
https://github.com/python/cpython/commit/843c27765652e2322011fb3e5d88f4837de38c06
msg370947 - (view) Author: miss-islington (miss-islington) Date: 2020-06-08 01:30
New changeset 9cf1be46e3692d565461afd3afa326d124d743dd by Miss Islington (bot) in branch '3.9':
bpo-39791 native hooks for importlib.resources.files (GH-20576)
https://github.com/python/cpython/commit/9cf1be46e3692d565461afd3afa326d124d743dd
msg371074 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-06-09 06:06
Note that the most recent commits have introduced a critical regression in importlib.resources.path() that breaks the certifi package from PyPI and presumably other users of path().  See Issue40924.
msg371131 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-06-09 17:50
New changeset ce5e6f098f8a270e50b989baa75765584573706b by Łukasz Langa in branch '3.9':
[3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" (#20760)
https://github.com/python/cpython/commit/ce5e6f098f8a270e50b989baa75765584573706b
msg371744 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-17 14:34
> bpo-39791 native hooks for importlib.resources.files (GH-20576)
> https://github.com/python/cpython/commit/843c27765652e2322011fb3e5d88f4837de38c06

This change introduced bpo-41007 "test_importlib logs ResourceWarning: test_path.CommonTests.test_importing_module_as_side_effect()" regression.
History
Date User Action Args
2020-06-17 14:39:06vstinnersetpull_requests: + pull_request20118
2020-06-17 14:34:19vstinnersetnosy: + vstinner

messages: + msg371744
versions: + Python 3.10, - Python 3.9
2020-06-09 17:50:09lukasz.langasetmessages: + msg371131
2020-06-09 12:53:47lukasz.langasetnosy: + lukasz.langa
pull_requests: + pull_request19960
2020-06-09 06:06:42ned.deilysetnosy: + ned.deily
messages: + msg371074
2020-06-08 01:30:15miss-islingtonsetmessages: + msg370947
2020-06-08 01:01:07miss-islingtonsetpull_requests: + pull_request19918
2020-06-08 01:00:55jaracosetmessages: + msg370945
2020-06-07 15:17:54miss-islingtonsetmessages: + msg370904
2020-06-07 14:58:02miss-islingtonsetpull_requests: + pull_request19910
2020-06-07 14:57:49jaracosetmessages: + msg370903
2020-06-06 17:12:33jaracosetmessages: + msg370840
2020-06-06 17:10:05jaracosetpull_requests: + pull_request19894
2020-06-06 16:38:25jaracosetmessages: + msg370834
2020-06-06 15:12:49Michael.Feltsetnosy: + Michael.Felt
messages: + msg370825
2020-06-05 21:46:31jaracosetmessages: + msg370790
2020-06-05 20:34:44miss-islingtonsetpull_requests: + pull_request19880
2020-06-05 20:34:38miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19879
2020-06-05 20:34:30jaracosetmessages: + msg370782
2020-06-05 19:28:23jaracosetpull_requests: + pull_request19878
2020-06-05 19:08:05jaracosetpull_requests: + pull_request19875
2020-06-01 16:33:05jaracosetpull_requests: + pull_request19816
2020-05-08 23:59:25jaracosetmessages: + msg368490
2020-05-08 23:20:29jaracosetmessages: + msg368484
2020-04-26 22:01:22jaracosetkeywords: + patch
stage: patch review
pull_requests: + pull_request19044
2020-03-08 03:48:01benthayersetnosy: + benthayer
2020-03-06 21:27:28jaracosetmessages: + msg363556
2020-02-29 06:22:44barrysetnosy: + barry
2020-02-29 04:11:49jaracosetassignee: jaraco
type: behavior
components: + Library (Lib)
2020-02-29 04:11:30jaracocreate