Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zipfile does not support pathlib #72418

Closed
ethanfurman opened this issue Sep 21, 2016 · 19 comments
Closed

zipfile does not support pathlib #72418

ethanfurman opened this issue Sep 21, 2016 · 19 comments
Labels
3.7 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@ethanfurman
Copy link
Member

BPO 28231
Nosy @brettcannon, @ned-deily, @bitdancer, @ethanfurman, @berkerpeksag, @serhiy-storchaka, @zooba
PRs
  • bpo-28231: Make file parameter of ZipFile accept PathLike objects #322
  • bpo-28231: The zipfile module now accepts path-like objects for external paths. #511
  • [3.6] bpo-28231: The zipfile module now accepts path-like objects for external paths. #561
  • [Do Not Merge] Sample of CPython life with blurb. #703
  • Files
  • open-zipfile.stoneleaf.patch
  • zipfile-pathlib.patch
  • zipfile-pathlib-3.6.1.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-03-08.14:06:01.084>
    created_at = <Date 2016-09-21.07:32:16.977>
    labels = ['3.7', 'type-feature', 'library']
    title = 'zipfile does not support pathlib'
    updated_at = <Date 2017-03-24.22:41:06.911>
    user = 'https://github.com/ethanfurman'

    bugs.python.org fields:

    activity = <Date 2017-03-24.22:41:06.911>
    actor = 'serhiy.storchaka'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-03-08.14:06:01.084>
    closer = 'serhiy.storchaka'
    components = ['Library (Lib)']
    creation = <Date 2016-09-21.07:32:16.977>
    creator = 'ethan.furman'
    dependencies = []
    files = ['44773', '46672', '46677']
    hgrepos = []
    issue_num = 28231
    keywords = ['patch']
    message_count = 19.0
    messages = ['277109', '277391', '284754', '284758', '284791', '284792', '285184', '285194', '285234', '286497', '288602', '288620', '288650', '288653', '288656', '288689', '288693', '290261', '290262']
    nosy_count = 8.0
    nosy_names = ['brett.cannon', 'ned.deily', 'r.david.murray', 'ethan.furman', 'berker.peksag', 'serhiy.storchaka', 'steve.dower', 'jtf621']
    pr_nums = ['322', '511', '561', '703']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue28231'
    versions = ['Python 3.6', 'Python 3.7']

    @ethanfurman ethanfurman added the type-bug An unexpected behavior, bug, or error label Sep 21, 2016
    @serhiy-storchaka
    Copy link
    Member

    Shouldn't the ZipFile.filename attribute be converted to str?

    If add support of pathlib, maybe add support of bytes?

    The file name of ZipFile is only a part of the issue. There are other uses of file paths: paths for added files, path for extracted directory.

    And there are internal paths in zip archive. Maybe it is worth to introduce new path-like type for them and provide pathlib-like interface to ZipFile. But this is separate not easy issue.

    @brettcannon
    Copy link
    Member

    the patch LGTM, but Serhiy has a point that maybe we should add tests for dealing with other paths such as those contained within the zipfile.

    @berkerpeksag berkerpeksag added stdlib Python modules in the Lib dir 3.7 (EOL) end of life labels Oct 5, 2016
    @jtf621
    Copy link
    Mannequin

    jtf621 mannequin commented Jan 5, 2017

    This also affects python 3.5 on Windows and OSX.

    Python 3.5.2 (default, Sep 21 2016, 15:07:18)
    [GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from pathlib import Path
    >>> import zipfile
    >>> f = Path('zipfile.zip')
    >>> with zipfile.ZipFile(f) as zf:
    ... 	pass
    ...
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/jeremy/.pyenv/versions/3.5.2/lib/python3.5/zipfile.py", line 1026, in __init__
        self._RealGetContents()
      File "/Users/jeremy/.pyenv/versions/3.5.2/lib/python3.5/zipfile.py", line 1089, in _RealGetContents
        endrec = _EndRecData(fp)
      File "/Users/jeremy/.pyenv/versions/3.5.2/lib/python3.5/zipfile.py", line 241, in _EndRecData
        fpin.seek(0, 2)
    AttributeError: 'PosixPath' object has no attribute 'seek'

    @bitdancer
    Copy link
    Member

    In 3.5 the stdlib is not supporting PathLib. So this issue only affects 3.6 and 3.7.

    @jtf621
    Copy link
    Mannequin

    jtf621 mannequin commented Jan 6, 2017

    OK, I understand. How can I help get this issue fixed?

    1. review the patch?
    2. update the docs to reflect the patch?
    3. find the other uses of pathlib in the zipfile module?
    4. something else ...

    I am a longtime user of python but a first time contributor but happy to help.

    @berkerpeksag
    Copy link
    Member

    I think the next steps are:

    1. Add tests for the cases Serhiy has mentioned in msg277109:

      > Shouldn't the ZipFile.filename attribute be converted to str?

      and

      > The file name of ZipFile is only a part of the issue. There are other uses of file paths: paths for added files, path for extracted directory.

    2. Update the docs to reflect the pathlib support.

    @jtf621
    Copy link
    Mannequin

    jtf621 mannequin commented Jan 11, 2017

    I have reviewed the code and docs for the public API that should take a pathlib.Path object:

    • zipfile.is_zipfile(filename)
      • filename
    • zipfile.ZipFile(file)
      • file
    • ZipFile.extract(member, path=None)
      • path
    • ZipFile.extractall(path=None)
      • path
    • ZipFile.write(filename)
      • filename
    • zipfile.PyZipFile(file)
      • file
    • PyZipFile.writepy(pathname)
      • pathname
    • ZipInfo.from_file(filename, arcname=None)
      • filename

    Does this appear complete?

    Working on tests that probe each of these API points with pathlib.Path objects.

    I am not sure what "Shouldn't the ZipFile.filename attribute be converted to str?" means, can you elaborate?

    @serhiy-storchaka
    Copy link
    Member

    I am not sure what "Shouldn't the ZipFile.filename attribute be converted to str?" means, can you elaborate?

    If you pass a Path object to ZipFile constructor, should the filename attribute be a Path object or a str?

    @ethanfurman
    Copy link
    Member Author

    Any path/file attributes, etc, inside a ZipFile should be str. ZipFile should also function properly if path/file requests are given as os.PathLike objects.

    @zooba
    Copy link
    Member

    zooba commented Jan 30, 2017

    Speaking as a "regular user" who just ran into this, my main concern is that PathLike paths get used properly. For filenames being passed back out, if I really want them to be Path objects, I'll wrap them in Path() anyway.

    Please don't let a full conversion to pathlib hold up fixing passing a PathLike into the constructor. That's the main use case, and given how nicely most of the rest of the stdlib handles Path objects now, it's an annoying wart.

    @berkerpeksag
    Copy link
    Member

    PR 322 should make the example in msg284754 work:

    >>> import pathlib, zipfile
    >>> f = pathlib.Path('spam.zip')
    >>> with zipfile.ZipFile(f) as zf:
    ...   zf.namelist()
    ... 
    ['LICENSE']

    It doesn't implement full PathLike support, but it at least covers the use cases of Jeremy and Steve.

    @serhiy-storchaka
    Copy link
    Member

    I have different path. It adds the support of path-like objects for all external paths.

    @serhiy-storchaka serhiy-storchaka added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Feb 27, 2017
    @zooba
    Copy link
    Member

    zooba commented Feb 27, 2017

    Why can't we fix this in 3.6? We were meant to support pathlike in that version, and this is an oversight, not a new feature.

    @serhiy-storchaka
    Copy link
    Member

    I consider this a new feature.

    Some modules got the support of path-like objects for free after implementing the support of path-like objects in os, os.path and io modules. But others need additional work for implementing it, writing tests and documentation. In case of zipfile this work is significant.

    @berkerpeksag
    Copy link
    Member

    Note that Ned gave us a permission to get this into 3.6.1.

    @ned-deily
    Copy link
    Member

    Note that Ned gave us a permission to get this into 3.6.1.

    I may have although I don't remember specifically discussing zipfile. In any case, I'm willing to consider it. I think you can make good arguments for and against. Yes, it could smell like adding a feature but, on the other add, one of the implicit goals of 3.6.0 was to make Path objects supported across the standard library as much as possible, so the lack of support in zipfile (and a few other similar modules) could be viewed as bug. Also, as far as I can tell, this should be a totally upwards-compatible change except in the presumably unlikely case something is counting on getting an exception when passing a Path object to zipfile. I say we invoke "practicality beats purity" for this as long as Serhiy is OK with having it cherry-picked to 3.6 and as long as no other core developer here has a strong objection.

    @serhiy-storchaka
    Copy link
    Member

    The patch is backported to 3.6.1.

    @serhiy-storchaka
    Copy link
    Member

    New changeset eb65edd by Serhiy Storchaka in branch '3.6':
    [3.6] bpo-28231: The zipfile module now accepts path-like objects for external paths. (#561)
    eb65edd

    @serhiy-storchaka
    Copy link
    Member

    New changeset 8606e95 by Serhiy Storchaka in branch 'master':
    bpo-28231: The zipfile module now accepts path-like objects for external paths. (#511)
    8606e95

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants