Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PEP 529 for io.FileIO #73595

Closed
eryksun opened this issue Feb 1, 2017 · 6 comments
Closed

Implement PEP 529 for io.FileIO #73595

eryksun opened this issue Feb 1, 2017 · 6 comments
Assignees
Labels
3.7 (EOL) end of life OS-windows topic-IO type-bug An unexpected behavior, bug, or error

Comments

@eryksun
Copy link
Contributor

eryksun commented Feb 1, 2017

BPO 29409
Nosy @pfmoore, @tjguk, @zware, @eryksun, @zooba
Files
  • issue_29409_01.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zooba'
    closed_at = <Date 2018-01-07.01:37:04.102>
    created_at = <Date 2017-02-01.03:06:20.594>
    labels = ['type-bug', '3.7', 'expert-IO', 'OS-windows']
    title = 'Implement PEP 529 for io.FileIO'
    updated_at = <Date 2018-01-07.01:37:04.102>
    user = 'https://github.com/eryksun'

    bugs.python.org fields:

    activity = <Date 2018-01-07.01:37:04.102>
    actor = 'steve.dower'
    assignee = 'steve.dower'
    closed = True
    closed_date = <Date 2018-01-07.01:37:04.102>
    closer = 'steve.dower'
    components = ['Windows', 'IO']
    creation = <Date 2017-02-01.03:06:20.594>
    creator = 'eryksun'
    dependencies = []
    files = ['46473']
    hgrepos = []
    issue_num = 29409
    keywords = ['patch']
    message_count = 6.0
    messages = ['286585', '286998', '287004', '287006', '287015', '287020']
    nosy_count = 6.0
    nosy_names = ['paul.moore', 'tim.golden', 'python-dev', 'zach.ware', 'eryksun', 'steve.dower']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue29409'
    versions = ['Python 3.6', 'Python 3.7']

    @eryksun
    Copy link
    Contributor Author

    eryksun commented Feb 1, 2017

    PEP-529 isn't implemented for io.FileIO, and I think it should be. If a UTF-8 path is passed to open(), it ends up calling C _open instead of decoding the path and calling C _wopen. Also, if a pathlike object is passed to io.FileIO, it calls PyUnicode_FSConverter on it, converting it to UTF-8 and then passing it to _open. For example:

        >>> p = r'C:\Temp\αβψδ'
        >>> os.path.exists(p)
        True
    
        >>> open(p.encode())
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        FileNotFoundError: [Errno 2] No such file or directory: b'C:\\Temp\\\xce\xb1\xce\xb2\xcf\x88\xce\xb4'
    
        >>> io.FileIO(pathlib.Path(p))
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
        FileNotFoundError: [Errno 2] No such file or directory: WindowsPath('C:/Temp/αβψδ')

    The Windows implementation should mirror the POSIX implementation via PyUnicode_FSDecoder.

    @eryksun eryksun added 3.7 (EOL) end of life OS-windows topic-IO type-bug An unexpected behavior, bug, or error labels Feb 1, 2017
    @zooba
    Copy link
    Member

    zooba commented Feb 4, 2017

    Merging this in now, along with a few other patches. This is the test I'm adding, in case anyone spots any problems with it before I push (copied from the test for pure-ASCII bytes):

    @unittest.skipIf(sys.getfilesystemencoding() != 'utf-8',
                     "test only works for utf-8 filesystems")
    def testUtf8BytesOpen(self):
        # Opening a UTF-8 bytes filename
        try:
            fn = TESTFN_UNICODE.encode("utf-8")
        except UnicodeEncodeError:
            self.skipTest('could not encode %r to utf-8' % TESTFN_UNICODE)
        f = self.FileIO(fn, "w")
        try:
            f.write(b"abc")
            f.close()
            with open(TESTFN_UNICODE, "rb") as f:
                self.assertEqual(f.read(), b"abc")
        finally:
            os.unlink(TESTFN_UNICODE)
    

    @zooba zooba self-assigned this Feb 4, 2017
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 4, 2017

    New changeset 0bf72810f8ea by Steve Dower in branch '3.6':
    Issue bpo-29409: Implement PEP-529 for io.FileIO (Patch by Eryk Sun)
    https://hg.python.org/cpython/rev/0bf72810f8ea

    New changeset a5538734cc87 by Steve Dower in branch 'default':
    Merge issue bpo-28164 and issue bpo-29409
    https://hg.python.org/cpython/rev/a5538734cc87

    @zooba
    Copy link
    Member

    zooba commented Feb 4, 2017

    Committed, but I'll leave this open for a bit in case anyone wants to comment on the commit.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 5, 2017

    New changeset fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a by Steve Dower in branch 'master':
    Issue bpo-29409: Implement PEP-529 for io.FileIO (Patch by Eryk Sun)
    fe7e868

    New changeset 965f8d68a8dcc2ebb2480fe7e9121ac7efdee54e by Steve Dower in branch 'master':
    Merge issue bpo-28164 and issue bpo-29409
    965f8d6

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Feb 5, 2017

    New changeset fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a by Steve Dower in branch '3.6':
    Issue bpo-29409: Implement PEP-529 for io.FileIO (Patch by Eryk Sun)
    fe7e868

    @zooba zooba closed this as completed Jan 7, 2018
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life OS-windows topic-IO type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants