classification
Title: Odd behavior of ~ in os.path.abspath and os.path.realpath
Type: behavior Stage: resolved
Components: None Versions: Python 3.2
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: georg.brandl, ibshields, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2013-01-06 04:25 by ibshields, last changed 2013-01-07 01:04 by ibshields. This issue is now closed.

Files
File name Uploaded Description Edit
doc_no_default_expand.patch r.david.murray, 2013-01-06 20:51 review
Messages (12)
msg179170 - (view) Author: Ian Shields (ibshields) Date: 2013-01-06 04:25
Filespecs that start with ~ are not properly handled by os.path.realpath or os.path.abspath (and maybe other functions). Following console output from Fedora 17 using Puthon 3.2 illustrates the issue. Similar issue in 2.7
[ian@attic4 developerworks]$ cd ..
[ian@attic4 ~]$ mkdir testpath
[ian@attic4 ~]$ cd testpath
[ian@attic4 testpath]$ pwd
/home/ian/testpath
[ian@attic4 testpath]$ python3
Python 3.2.3 (default, Jun  8 2012, 05:36:09) 
[GCC 4.7.0 20120507 (Red Hat 4.7.0-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.abspath("~")
'/home/ian/testpath/~'
>>> os.path.realpath("~/xxx/zzz")
'/home/ian/testpath/~/xxx/zzz'
>>> os.path.abspath("~/..")
'/home/ian/testpath'

Function should probably use expanduser to determine if path is already absolute. Documentation at http://docs.python.org/3/library/os.path.html is also misleading as this is not how these functions work if given an absolute path to start with.
msg179171 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-06 04:54
This is intentional. You have to call expanduser yourself if you want ~ to be expanded.  Otherwise it is treated as any other path component would be.

I don't understand your final comment, can you clarify?
msg179197 - (view) Author: Ian Shields (ibshields) Date: 2013-01-06 17:40
Regarding last comment. I had missed the comment in documentation fo os.path.join "Join one or more path components intelligently. If any component is an absolute path, all previous components (on Windows, including the previous drive letter, if there was one) are thrown away, and joining continues". So the issue is really the behavior of os.path.join where the intelligence in the joining does not recognize that "~" is usually expanded to an absolute path. Consider the following Bash commands:
[ian@attic4 testpath]$ pwd
/home/ian/testpath
[ian@attic4 testpath]$ echo $(cd ~/testpath/..;pwd)
/home/ian
[ian@attic4 testpath]$ cd /home/ian/~
bash: cd: /home/ian/~: No such file or directory

Now consider some Python
>>> os.getcwd()
'/home/ian/testpath'
>>> os.path.join(os.getcwd(), "/home/ian")
'/home/ian'
>>> os.path.expanduser("~")
'/home/ian'
>>> os.path.join(os.getcwd(), "~")
'/home/ian/testpath/~'
>>> os.path.expanduser(os.path.abspath("~"))
'/home/ian/testpath/~'
>>> os.path.abspath(os.path.expanduser("~"))
'/home/ian'

I find the Python behavior rather odd. I cna live with it now I know about it, but if it is really intentional it would help to document this rather odd behavior somewhat better.
msg179198 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-06 17:49
Can you point exactly where the "odd behavior" is?  Note that "~" is a normal component for UNIX file/path names, and in such it has no special meaning (as opposed to "/").

This is why it gets no special handling by any Python path functions, except for expanduser().  That it is handled specially by the shell is no reason for Python to do the same.
msg179205 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-06 18:50
In addition, as far as I know, ba/sh has no equivalent to os.path.join (see, eg: http://unix.stackexchange.com/questions/23208/building-paths-robustly).  So even if we *did* want to "act like the shell" there's no analog to point to.  The shell only expands ~ when it is the first component of a path in a globbing context, and Python does not do globbing automatically anywhere.  So calling expanduser on a path before doing the join allows you to recognize '~'s if you want to support them in your input.  But, as I said, the shell has no real analog for join, so talking about the behavior of join in analogy to the behavior of the shell is pretty meaningless.

The key thing to realize here is that Python does not do globbing automatically (there is a separate glob module for that).  I don't think even a doc update is warranted, since it would be a bit weird to document things we *don't* do :)  However, if you have a specific proposal for what such a note would look like and where it would be placed, we would consider it, as we are always looking to make the docs better.
msg179206 - (view) Author: Ian Shields (ibshields) Date: 2013-01-06 19:14
Oddity may be in the eye of the beholder. I've been programming and scripting for about 40 years, including several *IX shells and many other systems. I'm relatively new to Python. Mostly the results of doing things in Python are what I expect. Not doing expansion of a leading tilde when I ask for an absolute path is not what I expect. So to me it's odd. Or different. Or just not what I expect. Substitute "unexpected" for "odd" if you like. Sure, tilde expansion wasn't part of the Bourne shell, but it's been in POSIX shells for about the same amount of time that Python has been around, so it's odd to me that Python differs in this way. It's not hard to work around now that I know about it.
msg179214 - (view) Author: Ian Shields (ibshields) Date: 2013-01-06 19:59
David, Tilde expansion is different to globbing. Globbing in Python doesn't automatically do tilde expansion either.
>>> glob.glob("~")
[]

Looking at the documentation, I don't think it would be practical to add documentation to each affected function - there are too many. Apart from abspath and realpath (which started this conversation), there are join, isdir, isfile, isabs, and many others in os.path. posixpath and the other path handling modules.

Maybe a note at the top of the module documentation (e.g. http://docs.python.org/3/library/os.path.html) which has two notes already. Something along the line of:

Unlike many shells, Python does not automatically do tilde expansion or variable substitution in paths. Use expanduser for tilde expansion or expandvars for variable substitution.

Or just simply
Use expanduser for tilde expansion or expandvars for variable substitution in paths as the other path functions do not do this automatically.
msg179216 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-06 20:18
Both are a kind of command-line expansion, see for example bash(1).

Listed there are "brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion".  Python does none of them.
msg179224 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-01-06 20:51
Here is a proposed doc patch.  While not a necessary doc addition, I do think it provides some useful guidance, and is short enough that it doesn't weigh down the docs.
msg179227 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-01-06 21:07
LGTM.
msg179228 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-01-06 21:15
New changeset 02e2fa22f8b7 by R David Murray in branch '3.2':
#16877: Add mention that shell-style path expansions are not automatic.
http://hg.python.org/cpython/rev/02e2fa22f8b7

New changeset 2ff547c165fd by R David Murray in branch '3.3':
merge #16877: Add mention that shell-style path expansions are not automatic.
http://hg.python.org/cpython/rev/2ff547c165fd

New changeset b6284d2aaada by R David Murray in branch 'default':
merge #16877: Add mention that shell-style path expansions are not automatic.
http://hg.python.org/cpython/rev/b6284d2aaada

New changeset 58ac62bc3cd5 by R David Murray in branch '2.7':
#16877: Add mention that shell-style path expansions are not automatic.
http://hg.python.org/cpython/rev/58ac62bc3cd5
msg179238 - (view) Author: Ian Shields (ibshields) Date: 2013-01-07 01:04
I think that's an excellent resolution to the problem. Thank you.
History
Date User Action Args
2013-01-07 01:04:31ibshieldssetmessages: + msg179238
2013-01-06 21:15:21python-devsetnosy: + python-dev
messages: + msg179228
2013-01-06 21:07:01georg.brandlsetmessages: + msg179227
2013-01-06 20:51:01r.david.murraysetfiles: + doc_no_default_expand.patch
keywords: + patch
messages: + msg179224
2013-01-06 20:18:02georg.brandlsetmessages: + msg179216
2013-01-06 19:59:24ibshieldssetmessages: + msg179214
2013-01-06 19:14:30ibshieldssetmessages: + msg179206
2013-01-06 18:50:01r.david.murraysetmessages: + msg179205
2013-01-06 17:49:44georg.brandlsetstatus: open -> closed
nosy: + georg.brandl
messages: + msg179198

2013-01-06 17:40:02ibshieldssetstatus: pending -> open

messages: + msg179197
2013-01-06 04:54:37r.david.murraysetstatus: open -> pending

nosy: + r.david.murray
messages: + msg179171

resolution: not a bug
stage: resolved
2013-01-06 04:25:15ibshieldscreate