classification
Title: os.listdir and os.path.join inconsistent on empty path
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Thomas Fenzl, babou, ncoghlan, r.david.murray, woparry
Priority: normal Keywords: easy, patch

Created on 2013-03-25 16:26 by babou, last changed 2013-07-25 16:56 by r.david.murray.

Files
File name Uploaded Description Edit
issue17545.patch woparry, 2013-05-25 18:11
Messages (12)
msg185207 - (view) Author: Bernard Lang (babou) Date: 2013-03-25 16:26
The empty path '' is considered as an acceptable path in os.path.join, and works as a neutral prefix:
print os.path.join('','aaa')  ===>  aaa
which seems rather natural.
But it raises an exception when used as a parameter to os.listdir.
Logically, it should then list the current directory.
(the alternative would be to raise an exception when os.path.join gets an empty argument).

The inconsistency became clear to me when I had to write the following function :

def listdirs(path,flist): # Appends to "flist" all paths to files that
            # start with "path". Argument "path" can be empty string ''.
    if path=='' : localist=os.listdir('.')
    else : localist=os.listdir(path)
    for f in localist :
        p=os.path.join(path,f)
        flist.append(p)
        if os.path.isdir(p) :
            listdirs(p,flist)
    return flist

The conditionnal is needed only to avoid the exception, while the code is unique afterwards.

This is related to Issue818059, but I did not see how that issue was resolved.

Furthermore, the case of the empty path as argument to os.listdir should be documented in http://docs.python.org/2/library/os.html
msg185209 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-25 16:48
See also issue 6095.

However:

rdmurray@hey:~>ls ''
ls: cannot access : No such file or directory

So, the behavior is consistent with the shell.  The shell has no equivalent to os.path.join.
msg185212 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-25 16:51
By the way, you could write this as:

  localist = os.listdir(path if path else '.')
msg185297 - (view) Author: Bernard Lang (babou) Date: 2013-03-26 20:15
Reply to R.David.Murray

> See also issue 6095.

You are right. I goofed, this is the issue I meant to point to.

> $ ls ''
> ls: cannot access : No such file or directory
> So, the behavior is consistent with the shell.

This is a fair remark.
But still, giving a meaning to something that has none in the shell would not be an inconsistency either. There are lots of other differences in the Unix shell design, and strings are often used as a syntactic device. You do not have to quote file names or paths, unless they raise syntactic problems.

Now if you look at path manipulation commands in the shell, you have :

$ basename aaa
aaa
$ dirname aaa
.

This is a fair choice for the shell since an empty path would print as nothing. Furthermore, string manipulation is not as convenient with the shell as it is with Python. So the shell is altogether ignoring the empty path, and string manipulations (possibly using the empty string) to build a path representation are not part of the path system.

Python has already made a different choice.

In [4]: os.path.basename('aaa')
Out[4]: 'aaa'

In [5]: os.path.dirname('aaa')
Out[5]: ''

These are the two results of os.path.split('aaa'), which is somewhat the inverse of os.path.join(...) which I initially considered.

So os.path.dirname in Python does return the empty string '' where the shell dirname returns a dot. This could also be seen as an inconsistency between Unix shell and Python.

However, the Unix shell is internally consistent, while taking into account its own specific constraints.

It seems that it is more important for Python to similarly have its own internal consistency, especially when considering that Python is already departing from Unix shell in some minor ways, which are related to the internal consistency issue that was raised.
msg185299 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-03-26 20:34
I'm inclined to agree with you.  I believe '' represents the current directory in sys.path, for example.  It ought to be easy to add this since issue 6095 is already fixed.
msg187783 - (view) Author: Bernard Lang (babou) Date: 2013-04-25 13:32
Thank you, David.

BTW, I sent a message on april 20 to docs@python.org about a bug in
the documentation regarding     os.readlink(path)
on page  http://docs.python.org/2/library/os.html

and proposing an alternative text.

I got no reply.

This was not long ago  ... should I just wait ?

Sorry for asking you ... I know no one else.

Regards

Bernard

* R. David Murray <report@bugs.python.org>, le 20-04-13, a écrit:
> 
> Changes by R. David Murray <rdmurray@bitdance.com>:
> 
> 
> ----------
> keywords: +easy
> stage:  -> needs patch
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17545>
> _______________________________________

-- 
SVP  Ne plus m'écrire à Bernard.Lang@inria.fr mais à l'adresse ci-dessous

Please  No longer write to Bernard.Lang@inria.fr but to the address below

  Bernard.Lang@datcha.net       ,_  /\o    \o/    gsm  +33 6 6206 1693
  http://www.datcha.net/       ^^^^^^^^^^^^^^^^^  tel  +33 1 3056 1693
        Je n'exprime que mon opinion - I express only my opinion
                 CAGED BEHIND WINDOWS or FREE WITH LINUX
msg187788 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-25 14:56
I don't know if the docs team replies to messages or not.  I know that issues sometimes get opened up here that refer to emails to that alias.  You can open an issue here yourself for the doc bug, if you want to.
msg189401 - (view) Author: W. Owen Parry (woparry) Date: 2013-05-16 18:43
I started working on a patch for this, but the more I think about it the less I am convinced it is wanted.

The issue requests that os.listdir('') be equal to os.listdir('.')

The given example of os.path.join doesn't follow this:

>>> os.path.join('','aaa')
'aaa'
>>> os.path.join('.','aaa')
'.\\aaa'

This makes sense: prepending an empty path should be a no-op, while prepending the current directory has a different meaning.

It seems consistent in this case that listing an empty path and listing the current directory should have different meanings.

Is there any other traction/use case for this change?
msg189689 - (view) Author: Thomas Fenzl (Thomas Fenzl) * Date: 2013-05-20 20:28
I also looked into creating a patch and now I'm not convinced about the solution. 

While os.path.join accepts an empty string, the functions is os (e.g. stat and friends, utime, chown don't. os.walk doesn't throw an Exception, but generates an empty iterator on ''.
So from a module perspective, things wouldn't get more consistent, but less. 

Any thoughts about how to proceed?
msg189773 - (view) Author: W. Owen Parry (woparry) Date: 2013-05-21 17:13
I suggest that this is a documentation issue.

I have seen three classes of functions is os and os.path
i - those which operate on path names only (os.path.join, os.path.dirname, etc) and do not depend on the state of the file system. Since these are string manipulation functions, they treat '' as the empty string.
ii - those which turn path names into valid paths via knowledge of the state of the file system (os.path.abspath, os.path.normpath, etc). These treat '' as an empty relative path and thus equivalent to the current directory.
iii - those which operate on valid paths only and thus depend on the state of the file system (os.listdir, os.stat, os.path.relpath, os.path.exists, etc). '' is not a valid path so these functions throw an exception.

The documentation should clearly state which class a function belongs to. I will provide a patch sometime this week.

os.walk is an odd duck - it has class iii behaviour except that it returns an empty iterator instead of raising an exception. Note that it still distinguishes between '' (invalid path) and '.' (current directory) and so is not a good reason to change os.listdir:

>>> [x for x in os.walk('')]
[]
>>> [x for x in os.walk('.')]
[('.', [], [])]

in an empty directory.
msg189988 - (view) Author: W. Owen Parry (woparry) Date: 2013-05-25 18:11
Patch as described above. Comments appreciated.
msg193704 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-07-25 16:56
Is your patch backward?

"Treat the path naturally" assume everyone agrees with you as to what is natural.  It would be better, I think, to be explicit about it.
History
Date User Action Args
2013-07-25 16:56:15r.david.murraysetmessages: + msg193704
2013-07-08 01:17:08ncoghlansetassignee: ncoghlan ->
2013-07-08 01:13:02ncoghlansetassignee: ncoghlan

nosy: + ncoghlan
2013-05-25 18:11:25woparrysetfiles: + issue17545.patch
keywords: + patch
messages: + msg189988
2013-05-21 17:13:33woparrysetmessages: + msg189773
2013-05-20 20:28:02Thomas Fenzlsetnosy: + Thomas Fenzl
messages: + msg189689
2013-05-16 18:43:27woparrysetnosy: + woparry
messages: + msg189401
2013-04-25 14:56:01r.david.murraysetmessages: + msg187788
2013-04-25 13:32:09babousetmessages: + msg187783
2013-04-20 11:32:30r.david.murraysetkeywords: + easy
stage: needs patch
2013-03-26 20:34:55r.david.murraysetmessages: + msg185299
versions: + Python 3.4, - Python 2.7
2013-03-26 20:15:59babousetmessages: + msg185297
2013-03-25 16:51:19r.david.murraysetmessages: + msg185212
2013-03-25 16:48:25r.david.murraysetnosy: + r.david.murray
messages: + msg185209
2013-03-25 16:26:53baboucreate