Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New files() api from importlib_resources. #83972

Closed
jaraco opened this issue Feb 29, 2020 · 17 comments
Closed

New files() api from importlib_resources. #83972

jaraco opened this issue Feb 29, 2020 · 17 comments
Assignees
Labels
3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@jaraco
Copy link
Member

jaraco commented Feb 29, 2020

BPO 39791
Nosy @warsaw, @nascheme, @nascheme, @jaraco, @ned-deily, @ambv, @aixtools, @miss-islington, @hauntsaninja, @benthayer
PRs
  • bpo-39791: Add files() to importlib.resources #19722
  • bpo-39791 native hooks for importlib.resources.files #20576
  • bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. #20656
  • bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. #20659
  • [3.9] bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) #20661
  • [3.8] bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) #20662
  • bpo-39791: Support file systems that cannot support non-ascii filenames. #20681
  • [3.9] bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (GH-20681) #20695
  • [3.9] bpo-39791 native hooks for importlib.resources.files (GH-20576) #20703
  • [3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" #20760
  • bpo-41006: importlib.util no longer imports typing #20938
  • bpo-39791: importlib.abc: avoid use of deprecated abstractproperty #23715
  • bpo-43288: Fix test_importlib test to correctly skip #24612
  • bpo-43288: Fix test_importlib test to correctly skip #24612
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/jaraco'
    closed_at = <Date 2020-12-09.23:47:15.564>
    created_at = <Date 2020-02-29.04:11:30.709>
    labels = ['type-bug', 'library', '3.9']
    title = 'New `files()` api from importlib_resources.'
    updated_at = <Date 2021-02-22.00:21:55.488>
    user = 'https://github.com/jaraco'

    bugs.python.org fields:

    activity = <Date 2021-02-22.00:21:55.488>
    actor = 'vstinner'
    assignee = 'jaraco'
    closed = True
    closed_date = <Date 2020-12-09.23:47:15.564>
    closer = 'jaraco'
    components = ['Library (Lib)']
    creation = <Date 2020-02-29.04:11:30.709>
    creator = 'jaraco'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 39791
    keywords = ['patch']
    message_count = 17.0
    messages = ['362962', '363556', '368484', '368490', '370782', '370790', '370825', '370834', '370840', '370903', '370904', '370945', '370947', '371074', '371131', '371744', '382812']
    nosy_count = 9.0
    nosy_names = ['barry', 'nascheme', 'nascheme', 'jaraco', 'ned.deily', 'lukasz.langa', 'Michael.Felt', 'miss-islington', 'hauntsaninja', 'benthayer']
    pr_nums = ['19722', '20576', '20656', '20659', '20661', '20662', '20681', '20695', '20703', '20760', '20938', '23715', '24612', '24612']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue39791'
    versions = ['Python 3.9']

    @jaraco
    Copy link
    Member Author

    jaraco commented Feb 29, 2020

    In the importlib_resources backport... in particular in issue 58 and merge request 76, the backport now has a new feature, a "files()" function.

    Let's incorporate that functionality into importlib.resources.

    @jaraco jaraco added 3.9 only security fixes stdlib Python modules in the Lib dir labels Feb 29, 2020
    @jaraco jaraco self-assigned this Feb 29, 2020
    @jaraco jaraco added the type-bug An unexpected behavior, bug, or error label Feb 29, 2020
    @jaraco
    Copy link
    Member Author

    jaraco commented Mar 6, 2020

    The latest release, 1.3.0, includes extensibility support and has been merged with the cpython branch of the importlib_resources project. I believe that code is now synced with this project and ready to be applied here. I'm hoping benthayer can apply the changes and submit the pr.

    @jaraco
    Copy link
    Member Author

    jaraco commented May 8, 2020

    New changeset 7f7e706 by Jason R. Coombs in branch 'master':
    bpo-39791: Add files() to importlib.resources (GH-19722)
    7f7e706

    @jaraco
    Copy link
    Member Author

    jaraco commented May 8, 2020

    I've merged PR 19722. Some follow up actions I'd like to do:

    • Add hooks for .files() on built-in loaders.
    • Replace loader.get_resource_reader() with adapters around .files() for built-in loaders.

    @jaraco
    Copy link
    Member Author

    jaraco commented Jun 5, 2020

    New changeset 161541a by Jason R. Coombs in branch 'master':
    bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659)
    161541a

    @jaraco
    Copy link
    Member Author

    jaraco commented Jun 5, 2020

    New changeset a4fa9a9 by Miss Islington (bot) in branch '3.9':
    bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) (GH-20661)
    a4fa9a9

    @aixtools
    Copy link
    Contributor

    aixtools commented Jun 6, 2020

    The 'fancy' file name breaks on latin-1 character set.

    aixtools@gcc119:[/home/aixtools/python/py39-3.9] a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb Lib/test/test_importlib/test_main.py                   <
    diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py
    index 42a7999..7b18c3d 100644
    --- a/Lib/test/test_importlib/test_main.py
    +++ b/Lib/test/test_importlib/test_main.py
    @@ -246,3 +246,19 @@ class TestEntryPoints(unittest.TestCase):
             """
             with self.assertRaises(Exception):
                 json.dumps(self.ep)
    +
    +    def test_module(self):
    +        assert self.ep.module == 'value'
    +
    +    def test_attr(self):
    +        assert self.ep.attr is None
    +
    +
    +class FileSystem(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
    +    def test_unicode_dir_on_sys_path(self):
    +        """
    +        Ensure a Unicode subdirectory of a directory on sys.path
    +        does not crash.
    +        """
    +        fixtures.build_files({'☃': {}}, prefix=self.site_dir)
    +        list(distributions())

    AIX bots fail with:

    Captured traceback
    ==================

    Traceback (most recent call last):
      File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/test_main.py", line 263, in test_unicode_dir_on_sys_path
        fixtures.build_files({'\u2603': {}}, prefix=self.site_dir)
      File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/fixtures.py", line 202, in build_files
        full_name.mkdir()
      File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/pathlib.py", line 1309, in mkdir
        self._accessor.mkdir(self, mode)
    UnicodeEncodeError: 'latin-1' codec can't encode character '\u2603' in position 17: ordinal not in range(256)

    Test report
    ===========

    Failed tests:

    • test_importlib

    @jaraco
    Copy link
    Member Author

    jaraco commented Jun 6, 2020

    Thanks for the report Michael. I'm trying to figure out the best way to address the issue. That test is shared with importlib_metadata, so needs to run without CPython's test suite fixtures, such as the ones that generate non-ascii filenames. I'm tempted to just attempt to create the fixture and skip if the fixture fails. An alternate approach might be to attempt to load CPython's fixture, skip if that value is None, and fallback to the snowman otherwise.

    @jaraco
    Copy link
    Member Author

    jaraco commented Jun 6, 2020

    See #64880 for a proposed fix. I've scheduled the build bots to run the patch. Will build bots prove the fix? If not, can you test the patch in the same environment where it was discovered?

    @jaraco
    Copy link
    Member Author

    jaraco commented Jun 7, 2020

    New changeset 2efe18b by Jason R. Coombs in branch 'master':
    bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (bpo-20681)
    2efe18b

    @miss-islington
    Copy link
    Contributor

    New changeset 71f5016 by Miss Islington (bot) in branch '3.9':
    bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (GH-20681)
    71f5016

    @jaraco
    Copy link
    Member Author

    jaraco commented Jun 8, 2020

    New changeset 843c277 by Jason R. Coombs in branch 'master':
    bpo-39791 native hooks for importlib.resources.files (GH-20576)
    843c277

    @miss-islington
    Copy link
    Contributor

    New changeset 9cf1be4 by Miss Islington (bot) in branch '3.9':
    bpo-39791 native hooks for importlib.resources.files (GH-20576)
    9cf1be4

    @ned-deily
    Copy link
    Member

    Note that the most recent commits have introduced a critical regression in importlib.resources.path() that breaks the certifi package from PyPI and presumably other users of path(). See bpo-40924.

    @ambv
    Copy link
    Contributor

    ambv commented Jun 9, 2020

    New changeset ce5e6f0 by Łukasz Langa in branch '3.9':
    [3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" (bpo-20760)
    ce5e6f0

    @vstinner
    Copy link
    Member

    bpo-39791 native hooks for importlib.resources.files (GH-20576)
    843c277

    This change introduced bpo-41007 "test_importlib logs ResourceWarning: test_path.CommonTests.test_importing_module_as_side_effect()" regression.

    @vstinner vstinner added 3.10 only security fixes and removed 3.9 only security fixes labels Jun 17, 2020
    @jaraco
    Copy link
    Member Author

    jaraco commented Dec 9, 2020

    This issue was implemented, just not as fully as I'd have hoped. Still lacking is native support for .files on the built-in package providers and removing the legacy APIs or at least configuring them to rely on the files API, but as far as supplying the files API, this work is done.

    @jaraco jaraco added 3.9 only security fixes and removed 3.10 only security fixes labels Dec 9, 2020
    @jaraco jaraco closed this as completed Dec 9, 2020
    @jaraco jaraco closed this as completed Dec 9, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants