This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: os.path.join does not use os.sep as documentation claims
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eryksun, fdrake, jpsutton, miss-islington
Priority: normal Keywords: easy, patch

Created on 2021-03-24 20:35 by jpsutton, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25025 merged jpsutton, 2021-03-26 14:03
PR 25027 merged miss-islington, 2021-03-26 16:03
PR 25030 merged miss-islington, 2021-03-26 19:52
Messages (13)
msg389489 - (view) Author: Jared Sutton (jpsutton) * Date: 2021-03-24 20:35
The behavior of os.path.join() does not match the documentation, in regards to the use of os.sep.  From the docs:

"""
The return value is the concatenation of path and any members of *paths with exactly one directory separator (os.sep) following each non-empty part except the last, meaning that the result will only end in a separator if the last part is empty.
"""

The documentation clearly states that the function uses the value of os.sep (which differs based on platform).  However, if you review the 2 implementations (ntpath.py and posixpath.py), the separator character used is clearly hard-coded and doesn't reference os.sep at all.

One could say that this is either a doc bug or an implementation bug, depending on what the intended behavior is. I submit that this is an implementation bug, as one might want to use os.path.join() to construct a path to be used on a platform other than the one currently running the application. For example, a person might be running Python on Windows, but calling a web API and constructing a path for use on a remote posix system.
msg389493 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-25 00:32
The documentation of os.path.join() uses os.sep because it's written from the perspective of the platform's os.path, which is posixpath on a POSIX system and ntpath on a Windows system.

If you need to work with POSIX paths in Windows, then use posixpath. It will of course use '/' as the path separator, since using backslash (os.sep) in a POSIX path is wrong. Vice versa, if you need to work with Windows paths in POSIX, use ntpath.
msg389500 - (view) Author: Jared Sutton (jpsutton) * Date: 2021-03-25 13:48
I can understand your suggestion to just utilize the posixpath library on Windows if needed. That's a reasonable work-around. But certainly you can see this is a doc bug, since the doc clearly states that os.sep is utilized to join the elements of the path, when it clearly isn't; right?
msg389505 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-25 15:07
> certainly you can see this is a doc bug, since the doc clearly 
> states that os.sep is utilized to join the elements of the path,
> when it clearly isn't; right?

os.path is ntpath in Windows, which uses backslash as the path separator, which is the same as os.sep in Windows. os.path is posixpath in POSIX, which uses slash as the path separator, which is the same as os.sep in POSIX.
msg389514 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2021-03-25 18:17
Perhaps Jared was expecting that modifying os.sep would affect the functions in os.path?

os.sep was never intended to be updated.

Using the specific *path modules to work with "foreign" paths has long been advocated as the way to do this.  It isn't a work-around.
msg389521 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-25 19:20
Fred, do you think this needs to be reopened to clarify that os.sep and os.altsep are not parameters for the ntpath and posixpath modules? 

I would have thought that the note at the top (path name conventions, etc) would be sufficient to let someone know that os.path is meant for working with paths on the current platform. Working with Windows paths in POSIX, and vice versa, is not simply a matter of replacing os.sep. For example:

    >>> posixpath.join('C:/spam', '/eggs')
    '/eggs'
    >>> ntpath.join('C:/spam', '/eggs')
    'C:/eggs'
msg389532 - (view) Author: Jared Sutton (jpsutton) * Date: 2021-03-26 01:03
> Perhaps Jared was expecting that modifying os.sep would affect the functions in os.path?

This is precisely what I thought, because the documentation makes it sound like that variable named os.sep is read and used as the path delimiter when constructing something with join(). In fact, that variable isn't read *at all* because the path separator is hard-coded in both posixpath.py and ntpath.py. Since os.sep isn't used, I see no reason why it should be referenced in the documentation at all.

I'm not trying to be pedantic here (though we nerds are famous for that :) ), but what I see here is a disagreement between what is documented and what actually exists in the implementation. Yes the *value* of os.sep happens to be the same as the one hard-coded into either ntpath or posixpath, but since that variable is not referenced in the implementation, its presence in the doc only serves to confuse people who take the documentation at face value and then get an unexpected result.
msg389539 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2021-03-26 04:51
Just reviewed the documentation for both os.sep and os.path.join().

The os.sep docs do not suggest it can be set, and the reference in the os.path.join() description is silent regarding that, so can be read as Jared did.  I don't recall this coming up before, but... there's a lot I don't remember at this point.  --sigh--

Removing the reference to os.sep from the os.path.join() description would not be confusing, IMO.  Clearly, the ambiguity can bite.

I'd be open to a patch to remove the reference to os.sep from the os.path.join() docs.  Reopening, but removing 3.6 and 3.7 since they're in security-fix-only mode.
msg389551 - (view) Author: Jared Sutton (jpsutton) * Date: 2021-03-26 14:13
Thank you for understanding my position, Fred. I submitted a draft PR (25025) for this bug.
msg389554 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2021-03-26 16:02
New changeset 21a2cabb3795f5170c746ab8f29e9d25c7442550 by Jared Sutton in branch 'master':
bpo-43620: Remove reference to os.sep from os.path.join() doc (#25025)
https://github.com/python/cpython/commit/21a2cabb3795f5170c746ab8f29e9d25c7442550
msg389555 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2021-03-26 17:24
New changeset f311290f091957653bba5ebfda28ad981bb78363 by Miss Islington (bot) in branch '3.9':
bpo-43620: Remove reference to os.sep from os.path.join() doc (GH-25025) (#25027)
https://github.com/python/cpython/commit/f311290f091957653bba5ebfda28ad981bb78363
msg389566 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2021-03-26 20:07
New changeset 455583b54aaec9a4266ff37dd438cbbd8ec6068a by Miss Islington (bot) in branch '3.8':
bpo-43620: Remove reference to os.sep from os.path.join() doc (GH-25025, GH-5030)
https://github.com/python/cpython/commit/455583b54aaec9a4266ff37dd438cbbd8ec6068a
msg389567 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2021-03-26 20:08
PR applied and backported; closing this.

Thanks, Jared!
History
Date User Action Args
2022-04-11 14:59:43adminsetgithub: 87786
2021-03-26 20:08:38fdrakesetstatus: open -> closed
resolution: fixed
messages: + msg389567

stage: patch review -> resolved
2021-03-26 20:07:07fdrakesetmessages: + msg389566
2021-03-26 19:52:30miss-islingtonsetpull_requests: + pull_request23779
2021-03-26 17:24:41fdrakesetmessages: + msg389555
2021-03-26 16:03:30miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request23777
2021-03-26 16:02:42fdrakesetmessages: + msg389554
2021-03-26 14:13:47jpsuttonsetmessages: + msg389551
2021-03-26 14:03:55jpsuttonsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request23775
2021-03-26 04:51:41fdrakesetstatus: closed -> open
versions: - Python 3.6, Python 3.7
messages: + msg389539

keywords: + easy
resolution: not a bug -> (no value)
stage: resolved -> needs patch
2021-03-26 01:03:32jpsuttonsetmessages: + msg389532
2021-03-25 19:20:59eryksunsetmessages: + msg389521
2021-03-25 18:17:21fdrakesetnosy: + fdrake
messages: + msg389514
2021-03-25 15:07:11eryksunsetmessages: + msg389505
2021-03-25 13:48:37jpsuttonsetmessages: + msg389500
components: - Library (Lib)
2021-03-25 00:32:15eryksunsetstatus: open -> closed

nosy: + eryksun
messages: + msg389493

resolution: not a bug
stage: resolved
2021-03-24 20:35:52jpsuttoncreate