classification
Title: Implement PEP 529 for io.FileIO
Type: behavior Stage: resolved
Components: IO, Windows Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: steve.dower Nosy List: eryksun, paul.moore, python-dev, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2017-02-01 03:06 by eryksun, last changed 2018-01-07 01:37 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
issue_29409_01.patch eryksun, 2017-02-01 03:08 review
Messages (6)
msg286585 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-02-01 03:06
PEP 529 isn't implemented for io.FileIO, and I think it should be. If a UTF-8 path is passed to open(), it ends up calling C _open instead of decoding the path and calling C _wopen. Also, if a pathlike object is passed to io.FileIO, it calls PyUnicode_FSConverter on it, converting it to UTF-8 and then passing it to _open. For example:

    >>> p = r'C:\Temp\αβψδ'
    >>> os.path.exists(p)
    True

    >>> open(p.encode())
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: b'C:\\Temp\\\xce\xb1\xce\xb2\xcf\x88\xce\xb4'

    >>> io.FileIO(pathlib.Path(p))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: WindowsPath('C:/Temp/αβψδ')

The Windows implementation should mirror the POSIX implementation via PyUnicode_FSDecoder.
msg286998 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-02-04 22:39
Merging this in now, along with a few other patches. This is the test I'm adding, in case anyone spots any problems with it before I push (copied from the test for pure-ASCII bytes):

    @unittest.skipIf(sys.getfilesystemencoding() != 'utf-8',
                     "test only works for utf-8 filesystems")
    def testUtf8BytesOpen(self):
        # Opening a UTF-8 bytes filename
        try:
            fn = TESTFN_UNICODE.encode("utf-8")
        except UnicodeEncodeError:
            self.skipTest('could not encode %r to utf-8' % TESTFN_UNICODE)
        f = self.FileIO(fn, "w")
        try:
            f.write(b"abc")
            f.close()
            with open(TESTFN_UNICODE, "rb") as f:
                self.assertEqual(f.read(), b"abc")
        finally:
            os.unlink(TESTFN_UNICODE)
msg287004 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-04 23:14
New changeset 0bf72810f8ea by Steve Dower in branch '3.6':
Issue #29409: Implement PEP 529 for io.FileIO (Patch by Eryk Sun)
https://hg.python.org/cpython/rev/0bf72810f8ea

New changeset a5538734cc87 by Steve Dower in branch 'default':
Merge issue #28164 and issue #29409
https://hg.python.org/cpython/rev/a5538734cc87
msg287006 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-02-04 23:15
Committed, but I'll leave this open for a bit in case anyone wants to comment on the commit.
msg287015 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-05 00:00
New changeset fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a by Steve Dower in branch 'master':
Issue #29409: Implement PEP 529 for io.FileIO (Patch by Eryk Sun)
https://github.com/python/cpython/commit/fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a

New changeset 965f8d68a8dcc2ebb2480fe7e9121ac7efdee54e by Steve Dower in branch 'master':
Merge issue #28164 and issue #29409
https://github.com/python/cpython/commit/965f8d68a8dcc2ebb2480fe7e9121ac7efdee54e
msg287020 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-05 00:00
New changeset fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a by Steve Dower in branch '3.6':
Issue #29409: Implement PEP 529 for io.FileIO (Patch by Eryk Sun)
https://github.com/python/cpython/commit/fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a
History
Date User Action Args
2018-01-07 01:37:04steve.dowersetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2017-02-05 00:00:34python-devsetmessages: + msg287020
2017-02-05 00:00:31python-devsetmessages: + msg287015
2017-02-04 23:15:27steve.dowersetmessages: + msg287006
stage: patch review -> commit review
2017-02-04 23:14:35python-devsetnosy: + python-dev
messages: + msg287004
2017-02-04 22:39:42steve.dowersetassignee: steve.dower
messages: + msg286998
stage: test needed -> patch review
2017-02-01 03:08:32eryksunsetstage: needs patch -> test needed
2017-02-01 03:08:09eryksunsetfiles: + issue_29409_01.patch
keywords: + patch
2017-02-01 03:06:20eryksuncreate