classification
Title: PEP 519 support in the stdlib
Type: enhancement Stage:
Components: Versions: Python 3.6
process
Status: open Resolution:
Dependencies: 26027 26667 27184 27186 Superseder:
Assigned To: Nosy List: Jelle Zijlstra, brett.cannon, ethan.furman, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-06-02 17:16 by ethan.furman, last changed 2016-09-07 02:36 by python-dev.

Files
File name Uploaded Description Edit
issue27182-open.patch Jelle Zijlstra, 2016-06-03 18:45 patch adding fspath support to builtins.open review
issue27182-path_type.patch Jelle Zijlstra, 2016-06-04 17:36 review
test_fspath.py ethan.furman, 2016-06-04 22:31
os.diff brett.cannon, 2016-08-12 19:27 Support for os.walk() and os.fwalk() (depends on path_converter) review
Messages (21)
msg266891 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-02 17:16
Meta issue to track adding PEP 519 support in the various stdlib modules.
msg266892 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-02 17:18
posix module: issue26027
msg266893 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-02 17:21
importlib: issue26667
msg266897 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-02 17:30
nt module: issue27184
msg266901 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-02 18:23
os.fspath(): issue27186
msg266908 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-06-02 18:36
Isn't the nt module just an alias of the posix module?
msg266953 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-02 21:11
Nope.

There is a posixpath.py and an ntpath.py, and they are not the same.
msg267123 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2016-06-03 18:01
FYI, I'm working on a patch for builtins.open to call PyOS_FSPath.
msg267131 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2016-06-03 18:45
This patch makes the Python and C versions of open()/io.open() support the fspath protocol.
msg267132 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-03 18:45
Sorry, Serhiy, I had my module names mixed up.

`nt` and `posix` are the same, but `ntpath` and `posixpath` are not (I may have those first two names wrong again, but hopefully you get the idea).
msg267280 - (view) Author: Roundup Robot (python-dev) Date: 2016-06-04 17:19
New changeset 00991aa5fdb5 by Ethan Furman in branch 'default':
issue27182: update fsencode and fsdecode for os.path(); patch by Dusty Phillips
https://hg.python.org/cpython/rev/00991aa5fdb5
msg267282 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2016-06-04 17:24
In the patch that was just committed, the path_type variables are undefined (os.py lines 892 and 908). There is also no test for these error cases—we should test the cases where the path has no __fspath__ or does not return a bytes or str.
msg267287 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2016-06-04 17:36
Attached patch fixes the undefined variable and adds additional tests.
msg267335 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2016-06-04 22:31
Currently, os.fspath will raise an exception if the thing passed in is not str/bytes/PathLike, and that error message will proclaim that str or bytes or PathLike is required; however, this is not true in cases such as Path (which doesn't allow bytes), and incomplete in cases such as os.open (which also allows ints).

On the other hand, if the thing has a functional __fspath__ (meaning calling it doesn't raise an exception) then os.fspath will return whatever that method returns, which could be complete garbage.

So os.fspath is being too strict, too open, and too lax all at the same time.

Given Guido's reluctance to check the output of __fspath__(), plus the current difficulty of painless integration with existing functions, I think we should have os.fspath() only raise an exception if obj.__fspath__ exists and calling it raises an exception, otherwise we return the result of calling obj.__fspath__(), or obj if it doesn't have __fspath__.

In case that wasn't clear, attached is a unit test that passes when the above changes are implemented.
msg267340 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * Date: 2016-06-04 22:59
If we do that, then os.* functions that accept fds would also work on objects whose __fspath__ method returns an integer. I don't think that is desirable (I was just writing a test to ensure that fspath returning an integer throws an error).
msg267457 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-06-05 18:45
Functions that only accept file descriptors should not be updated to work with __fspath__() as it will never return an int/fd.

As for Ethan's suggestion, are you saying you want to toss the str/bytes check from os.fspath()? If so then you will need to go to python-dev and bring that up as the PEP clearly specifies that str/bytes is checked for and specifically in the order of the Python code. The thinking behind the current design is that since __fspath__() has to be explicitly implemented that people will do so properly, versus accidentally passing in some type that isn't str/bytes like the pre-PEP 519 world (i.e. trust the __fspath__() implementors to do the right thing and only protect against someone passing in something wrong from complicated code flow).

There has been discussion about using the ``path.__fspath__() if hasattr(path, '__fspath__') else path`` idiom in os.path so that the pre-existing type-checks can do their thing instead of checking twice, although that's different from how os.fspath() works (then again, since this is all new code we could argue that going our own route in os.path is acceptable in the name of performance).
msg268065 - (view) Author: Roundup Robot (python-dev) Date: 2016-06-09 22:58
New changeset 6239673d5e1d by Brett Cannon in branch 'default':
Issue #27182: Document os.PathLike.
https://hg.python.org/cpython/rev/6239673d5e1d
msg272063 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-08-05 22:00
Just a quick update: between the patches for issue #26027 and issue #26667, the necessary code to make os.path work with path-like objects is done. At this point I'm just waiting for code reviews on those patches.
msg272559 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-08-12 19:27
Here is an odd patch because I don't know where else to put it ATM that adds the remaining support in the os module/package not covered by other issues w/ patches (specifically os.walk() and os.fwalk()). I think everything else simply falls through thanks to os.path and path_converter.
msg273737 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-08-26 21:49
The os and os.path modules are now done! The means PEP 519 is finished. At this point individual modules will need to be checked to see if they do (not) support os.PathLike.
msg274712 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-07 02:36
New changeset 3417d324cbf9 by Brett Cannon in branch 'default':
Issue #27182: Add support for path-like objects to PyUnicode_FSDecoder().
https://hg.python.org/cpython/rev/3417d324cbf9
History
Date User Action Args
2016-09-07 02:36:31python-devsetmessages: + msg274712
2016-08-26 21:49:44brett.cannonsetassignee: brett.cannon ->
2016-08-26 21:49:37brett.cannonsetmessages: + msg273737
2016-08-12 19:27:20brett.cannonsetfiles: + os.diff

messages: + msg272559
2016-08-05 22:00:35brett.cannonsetmessages: + msg272063
2016-06-24 20:43:30brett.cannonsetdependencies: - Add a "What's New" entry for PEP 519
2016-06-09 23:02:31brett.cannonsetassignee: brett.cannon
dependencies: + Add a "What's New" entry for PEP 519
2016-06-09 22:58:15python-devsetmessages: + msg268065
2016-06-05 18:45:39brett.cannonsetmessages: + msg267457
2016-06-04 22:59:34Jelle Zijlstrasetmessages: + msg267340
2016-06-04 22:31:40ethan.furmansetfiles: + test_fspath.py

messages: + msg267335
2016-06-04 20:30:27ethan.furmansetmessages: + msg267280
2016-06-04 19:10:17ethan.furmansetmessages: - msg267280
2016-06-04 17:36:12Jelle Zijlstrasetfiles: + issue27182-path_type.patch

messages: + msg267287
2016-06-04 17:24:54Jelle Zijlstrasetmessages: + msg267282
2016-06-04 17:19:55python-devsetnosy: + python-dev
messages: + msg267280
2016-06-03 18:45:52ethan.furmansetmessages: + msg267132
2016-06-03 18:45:49Jelle Zijlstrasetfiles: + issue27182-open.patch
keywords: + patch
messages: + msg267131
2016-06-03 18:01:55Jelle Zijlstrasetnosy: + Jelle Zijlstra
messages: + msg267123
2016-06-02 21:11:39ethan.furmansetmessages: + msg266953
2016-06-02 18:36:55serhiy.storchakasetmessages: + msg266908
2016-06-02 18:23:01ethan.furmansetdependencies: + add os.fspath()
messages: + msg266901
2016-06-02 17:39:35ethan.furmansettype: behavior -> enhancement
2016-06-02 17:38:54ethan.furmansetdependencies: + Support Path objects in the posix module, Update importlib to accept pathlib.Path objects, Support path objects in the ntpath module
type: behavior
versions: + Python 3.6
2016-06-02 17:30:33ethan.furmansetmessages: + msg266897
2016-06-02 17:21:29ethan.furmansetmessages: + msg266893
2016-06-02 17:18:33ethan.furmansetmessages: + msg266892
2016-06-02 17:16:29ethan.furmancreate