classification
Title: Better stdlib support for Path objects using .path attribute
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, barry, brett.cannon, bronger, cool-RR, daniel.ugra, ethan.furman, ezio.melotti, georg.brandl, gvanrossum, pitrou, python-dev, serhiy.storchaka, tshepang
Priority: normal Keywords:

Created on 2014-10-06 15:33 by barry, last changed 2016-06-02 17:13 by ethan.furman. This issue is now closed.

Messages (44)
msg228704 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-10-06 15:33
pathlib is really nice, but currently it's rather inconvenient to use due to the lack of support in other parts of the stdlib for Path objects.  For historical reasons, everything accepts string paths, but few places accept Paths.  As an example: configparser.ConfigParser.read() but there are lots of others.

I'm opening this bug to start a conversation about better support for Path objects in the stdlib.  Against all hope, I wish there was a simple way to extend the compatibility, but I don't like having to sprinkle `str(some_path)` calls everywhere (kind of defeats the purpose of having the nicer pathlib API IMHO).  I suspect instead that it will be a matter of adding type tests or str() conversions to the relevant methods, but there may be other issues to discuss, like is it even a good idea to do this? ;)
msg228707 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-06 15:43
Since we're unlikely to ever change all the places, I'd say it's better to be consistent.  I'd rather write str(path) all over the place than having to look up in the docs each time if that specific API happens to support passing Paths directly.

However, Antoine has been positive towards more utility methods on Path objects.  (Not that ConfigParser read() would fall in that category :)
msg228713 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-10-06 16:01
On Oct 06, 2014, at 03:43 PM, Georg Brandl wrote:

>I'd rather write str(path) all over the place than having to look up in the
>docs each time if that specific API happens to support passing Paths
>directly.

Have you tried to write a large-ish application using path objects?
str-infection is definitely a disincentive to using pathlib. :/
msg228715 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-06 16:12
I was about to suggest deriving your own Path class from Path and str, but got a base class layout conflict because Path objects define lots of __slots__ :(
msg228716 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-06 16:17
> I was about to suggest deriving your own Path class from Path and str

That would be a rather horrible solution.
It has already been suggested to create a "path protocol". I would suggest Barry tries to float and investigate the idea on python-ideas:
https://mail.python.org/pipermail/python-ideas/2014-May/027869.html
msg228717 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-06 16:18
> That would be a rather horrible solution.

I know :)
msg228718 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-06 16:19
As for adding convenience methods to Path objects, yes, I'm quite open to that. Best is to open an issue when you have a specific idea (and a patch :-)).
msg228767 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2014-10-07 13:43
I think I'm missing something here because the idea of doing `path = str(path)` at the API boundary for an old function to support both Path and str objects for paths seems fairly minimal. Only when manipulating a path is wanting a Path object going to come up, and in that case can't you just do `path = pathlib.Path(path)` instead?
msg229544 - (view) Author: Ram Rachum (cool-RR) * Date: 2014-10-16 18:11
I agree with Brett. Making old functions support `Path` sounds trivial to me, and I don't think there are any backward compatibility issues. Having to do str(path) every time you call a function is ugly. So I think that all stdlib functions that currently take a string path should be able to take a `Path` object.

I'll add to that the functions in the `zipfile` module.
msg229546 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-16 18:42
`path = str(path)` is minimal, but has the side effect of accepting almost any object, which is definitely not what you'd like ("where did that file named '<type object at ...>' come from?!")
msg229547 - (view) Author: Ram Rachum (cool-RR) * Date: 2014-10-16 18:44
Fine, so you add an `if isinstance...` in front of it and you're done.
msg229548 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-10-16 18:53
> Making old functions support `Path` sounds trivial to me

We're looking forward to trivial patches that enable Path handling consistently throughout the stdlib.

> Fine, so you add an `if isinstance...` in front of it and you're done.

Easier said than done in C modules.
msg229549 - (view) Author: Ram Rachum (cool-RR) * Date: 2014-10-16 18:54
Georg: You're right, I forgot about C modules which make this not as trivial as I thought.

As for Python modules: If there's general agreement that this is a feature we want, I'll be happy to make a patch for the `zipfile` module to accept `Path` arguments.
msg229550 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-16 19:14
Well, if you use an isinstance check you privilege the stdlib Path over any other pathlike implementation.  Since it *is* in the stdlib, this isn't an automatic reason for rejection, but it does have a bit of a code smell to it.  Why should everything that deals with path strings have to have intimate knowledge of Path objects?

I originally wrote here "Maybe we need a __path__ magic method" as a half-joke, but rereading the issue I see that this has in fact been proposed seriously, and referenced by Antoine (the pathlib author).

I'm -1 on just sprinkling support for Path throughout the stdlib.  Do it in a universally applicable fashion or don't do it at all, IMO.
msg229552 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-16 20:34
> I'm -1 on just sprinkling support for Path throughout the stdlib.
> Do it in a universally applicable fashion or don't do it at all, IMO.

How about doing it per module?
msg253124 - (view) Author: Torsten Bronger (bronger) Date: 2015-10-17 10:13
Please be conservative with adding methods to Path.

FWIW, my own experiences with pathlib have shown that utility methods have the disadvantage of duplicating well-established Python idioms.  This reduces code readability in my opinion.  While it is desirable that the rather inconvenient os.path may be superseded by pathlib in the long run,  utility methods like Path.glob, Path.open, Path.mkdir, and Path.readtext have convenient stdlib counterparts with long tradition, and using the methods seems disruptive and confusing to me.

This is a further argument for having a path protocol instead IMO.
msg257570 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 01:02
Random idea: what if pathlib.Path defined a .path attribute that was a plain string? Then you could write p.path instead of str(p), and "if hasattr(p, 'path'): p = p.path". This would be the new protocol. Advantage is also that DirEntry (returned by the new os.scandir()) already supports it. :-)
msg257593 - (view) Author: Ram Rachum (cool-RR) * Date: 2016-01-06 09:38
I thought about it some more, and personally I'd prefer each function to do `str(path)` internally rather than `if hasattr(p, 'path'): p = p.path`. Even if it means we'll have to deal with "where did that file named '<type object at ...>' come from?!" errors. I think it's clumsy that the path protocol is to access `path.path`. We already have a protocol for getting a string out of an object and it's `str(object)`. I think we should use it even if it makes debugging harder in the case where someone mistakenly passes a non-path object to a function that wants a path. (And without using `isinstance` either.)
msg257597 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-06 11:05
`str(object)` is not a protocol for getting a string out of an object. It's a protocol for getting a string for print(). __str__ is defined for every object and therefore is useless for getting a string out of "string-like" object (as __float__ for floats and __bytes__ for bytes). Perhaps we need a new special method __string__ that relates to __str__ as __index__ to __int__.
msg257599 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-01-06 11:08
Here the aim is really to distinguish path-like objects from other objects, not to accept arbitrary strings.

A ".path" attribute sounds like the wrong name, though: a Path has a path? And the Path's path is not a Path? Ouch :-)
msg257615 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 17:30
I think it's actually very reasonable for a Path to have a path attribute
that's a string. The DirEntry has two string attributes: name (the last
component) and path (the full path). The Path object already has the
former. Adding the latter makes sense to me. After all you've gotta give it
*some* name, and 'path' is used (unsurprisingly) in this meaning already in
many places.

The shortest idiom in libraries wanting to support this would be

    path = gettattr(arg, 'path', arg)

This extracts the path attribute from a DirEntry or Path object, and
assumes the argument is a string otherwise. I think this is relatively
reasonable to encode in C as well.
msg257616 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-06 17:35
Would `location` work as an attribute name?
msg257621 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 17:51
Only if we changed DirEntry to support that too. But it's a kind of
high-falootin' word that also has some other connotations (e.g.
geographical location, and the HTTP Location header). I've never heard it
use in relation to filenames -- those are invariably called some variant of
file, file name, path, full path. Really, the argument Antoine brings up
doesn't hold much weight.
msg257622 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-06 17:53
Personally I thought the name `path` fit; just trying to see if some other option might work that Antoine would also like.
msg257624 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-01-06 17:56
In any case I don't think "location" is any better ;-) If "path" fits other people then good.
msg257630 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 18:38
OK, I'll add 'path' to unblock changes to the stdlib (but I won't close
this issue).
msg257632 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-06 19:04
New changeset 7e9605697dfc by Guido van Rossum in branch '3.4':
Issue #22570: Add 'path' attribute to pathlib.Path objects.
https://hg.python.org/cpython/rev/7e9605697dfc

New changeset 9c49c417a68a by Guido van Rossum in branch '3.5':
Issue #22570: Add 'path' attribute to pathlib.Path objects. (Merge 3.4->3.5)
https://hg.python.org/cpython/rev/9c49c417a68a

New changeset d5f96a5da219 by Guido van Rossum in branch 'default':
Issue #22570: Add 'path' attribute to pathlib.Path objects. (Merge 3.5->3.6)
https://hg.python.org/cpython/rev/d5f96a5da219
msg257633 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2016-01-06 19:24
No docs? ;)
msg257634 - (view) Author: Roundup Robot (python-dev) Date: 2016-01-06 19:38
New changeset 2e3c31ab586a by Guido van Rossum in branch '3.4':
Docs for issue #22570.
https://hg.python.org/cpython/rev/2e3c31ab586a

New changeset 408f8b255b56 by Guido van Rossum in branch '3.5':
Docs for issue #22570. (Merge 3.4->3.5)
https://hg.python.org/cpython/rev/408f8b255b56

New changeset 759b2cecc289 by Guido van Rossum in branch '3.4':
Add versionadded (3.4.5) to docs for issue #22570.
https://hg.python.org/cpython/rev/759b2cecc289

New changeset 1a6b485e717f by Guido van Rossum in branch '3.5':
Add versionadded (3.4.5) to docs for issue #22570. (Merge 3.4->3.5)
https://hg.python.org/cpython/rev/1a6b485e717f

New changeset eab349b5c6d7 by Guido van Rossum in branch '3.5':
Cross-reference os.DirEntry and pathlib.Path for issue #22570.
https://hg.python.org/cpython/rev/eab349b5c6d7

New changeset 97ab0ccac893 by Guido van Rossum in branch 'default':
Docs for issue #22570. (Merge 3.5->3.6)
https://hg.python.org/cpython/rev/97ab0ccac893
msg257635 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 19:50
So, I added docs, mentioning the getattr(arg, 'path', arg) idiom, and (for 3.5 and 3.6) also cross-referencing with DirEntry.

I'm not sure whether to now close this issue or whether to leave it open to remind people of adding patches using the new idiom to various stdlib modules. Opinions?

Also, since pathlib is provisional, I felt okay with adding this to 3.4.5 and 3.5.2.
msg257638 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-06 20:44
We could leave this open as a meta issue and spin off individual issues for any specific module people choose to update to support Path objects, setting those new issues as dependencies here.
msg257643 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-06 21:08
Opened issue26027 for adding support in the posix module.

Should the path attribute be only string, or other types are acceptable (bytes, file descriptor, and even None if the function accepts None)?
msg257647 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-06 21:59
Let's say that the path attribute should be str or bytes -- this matches
the behavior of DirEntry. (But for pathlib.Path it is always a str.) It
cannot be None or FD. But note that the getattr(x, 'path', x) idiom returns
x unchanged if x is None or an FD (or a stream, for that matter).
msg257913 - (view) Author: Ram Rachum (cool-RR) * Date: 2016-01-10 15:09
Here's an alternate idea I thought of now. Maybe `path.path` should be a pointer to the same `Path` object, so every function will do this: 

    str(arg.path) if hasattr(arg, 'path') else arg

What I like about this is that it avoids having the path argument be a string. I don't like misnomers :)

I'd understand though if people won't like this suggestion since it introduces another step, of turning the path back into a string.
msg257920 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2016-01-10 17:21
I appreciate the thought, Ram, but I'm not a fan.
msg257924 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-10 18:23
Using 'path' for this field is not a misnomer (example: DictEntry uses 'path' for the same field).

See issue #26027 for a follow-up idea.
msg257926 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-10 18:31
> str(arg.path) if hasattr(arg, 'path') else arg

DirEntry.path can be bytes.
msg257928 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-10 19:18
You can't win 'em all... :-)
msg262935 - (view) Author: Roundup Robot (python-dev) Date: 2016-04-06 06:52
New changeset d0c8b2c1544e by Serhiy Storchaka in branch '3.5':
Issue #22570: Renamed Py_SETREF to Py_XSETREF.
https://hg.python.org/cpython/rev/d0c8b2c1544e

New changeset 719c11b6b6ff by Serhiy Storchaka in branch 'default':
Issue #22570: Renamed Py_SETREF to Py_XSETREF.
https://hg.python.org/cpython/rev/719c11b6b6ff

New changeset 7197809a7428 by Serhiy Storchaka in branch '2.7':
Issue #22570: Renamed Py_SETREF to Py_XSETREF.
https://hg.python.org/cpython/rev/7197809a7428
msg262938 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-06 07:45
Sorry, these changesets were related to issue26200.
msg265414 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-12 16:04
Reopening as we need to rename the path attribute to __fspath__ once Brett's PEP is accepted (which will be soon). https://github.com/brettcannon/path-pep/blob/master/pep-0NNN.rst

The 3.4 and 3.5 versions of this should probably just be reversed before their releases (early June).
msg265843 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-19 04:26
PEP 519 is accepted now. We need to revert the commits from http://bugs.python.org/issue22570#msg257634
msg265844 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-19 04:27
And those from http://bugs.python.org/issue22570#msg257632 (these are the actual code -- the others were the docs).
msg265892 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-05-19 20:18
Done. The revs are 90e58a77d386, 97198545e6c3, ade839421b8f.
History
Date User Action Args
2016-06-02 17:13:36ethan.furmansetnosy: + ethan.furman

title: Better stdlib support for Path objects -> Better stdlib support for Path objects using .path attribute
2016-05-19 20:18:47gvanrossumsetstatus: open -> closed
resolution: out of date -> fixed
messages: + msg265892
2016-05-19 04:27:51gvanrossumsetmessages: + msg265844
2016-05-19 04:26:42gvanrossumsetmessages: + msg265843
2016-05-12 16:04:20gvanrossumsetstatus: closed -> open
resolution: fixed -> out of date
messages: + msg265414
2016-04-06 07:45:34serhiy.storchakasetmessages: + msg262938
2016-04-06 06:52:01python-devsetmessages: + msg262935
2016-01-10 19:18:08gvanrossumsetmessages: + msg257928
2016-01-10 18:31:27serhiy.storchakasetmessages: + msg257926
2016-01-10 18:23:01gvanrossumsetstatus: open -> closed
messages: + msg257924

dependencies: - Support Path objects in the posix module
resolution: fixed
stage: resolved
2016-01-10 17:21:42brett.cannonsetmessages: + msg257920
2016-01-10 15:09:35cool-RRsetmessages: + msg257913
2016-01-07 04:09:21r.david.murraysetnosy: - r.david.murray
2016-01-06 21:59:22gvanrossumsetmessages: + msg257647
2016-01-06 21:08:42serhiy.storchakasetdependencies: + Support Path objects in the posix module
messages: + msg257643
2016-01-06 20:44:27brett.cannonsetmessages: + msg257638
2016-01-06 19:50:24gvanrossumsetmessages: + msg257635
versions: + Python 3.4, Python 3.6
2016-01-06 19:38:39python-devsetmessages: + msg257634
2016-01-06 19:24:25georg.brandlsetmessages: + msg257633
2016-01-06 19:04:31python-devsetnosy: + python-dev
messages: + msg257632
2016-01-06 18:38:47gvanrossumsetmessages: + msg257630
2016-01-06 17:56:08pitrousetmessages: + msg257624
2016-01-06 17:53:48brett.cannonsetmessages: + msg257622
2016-01-06 17:51:06gvanrossumsetmessages: + msg257621
2016-01-06 17:35:26brett.cannonsetmessages: + msg257616
2016-01-06 17:30:22gvanrossumsetmessages: + msg257615
2016-01-06 11:08:21pitrousetmessages: + msg257599
2016-01-06 11:05:16serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg257597
2016-01-06 09:38:47cool-RRsetmessages: + msg257593
2016-01-06 01:02:15gvanrossumsetnosy: + gvanrossum
messages: + msg257570
2015-10-17 10:13:34brongersetmessages: + msg253124
2015-10-17 08:58:07brongersetnosy: + bronger
2014-10-29 15:39:53daniel.ugrasetnosy: + daniel.ugra
2014-10-16 21:14:25Arfreversetnosy: + Arfrever
2014-10-16 20:34:46pitrousetmessages: + msg229552
2014-10-16 19:14:13r.david.murraysetmessages: + msg229550
2014-10-16 18:54:57cool-RRsetmessages: + msg229549
2014-10-16 18:53:16georg.brandlsetmessages: + msg229548
2014-10-16 18:44:44cool-RRsetmessages: + msg229547
2014-10-16 18:42:47georg.brandlsetmessages: + msg229546
2014-10-16 18:11:39cool-RRsetnosy: + cool-RR
messages: + msg229544
2014-10-12 05:22:51tshepangsetnosy: + tshepang
2014-10-07 13:51:48ezio.melottisetnosy: + ezio.melotti
type: enhancement
2014-10-07 13:43:57brett.cannonsetnosy: + brett.cannon
messages: + msg228767
2014-10-06 17:59:27r.david.murraysetnosy: + r.david.murray
2014-10-06 16:19:19pitrousetmessages: + msg228718
2014-10-06 16:18:00georg.brandlsetmessages: + msg228717
2014-10-06 16:17:08pitrousetmessages: + msg228716
2014-10-06 16:12:18georg.brandlsetmessages: + msg228715
2014-10-06 16:01:25barrysetmessages: + msg228713
2014-10-06 15:43:46georg.brandlsetnosy: + georg.brandl
messages: + msg228707
2014-10-06 15:35:03barrysetnosy: + pitrou
2014-10-06 15:34:03barrysetcomponents: + Library (Lib)
2014-10-06 15:33:51barrycreate