This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: normpath does not work with local literal paths
Type: behavior Stage: needs patch
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brian.curtin, eryksun, mandel, paul.moore, pitrou, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: needs review, patch

Created on 2012-07-07 16:47 by mandel, last changed 2022-04-11 14:57 by admin.

Files
File name Uploaded Description Edit
literal-normpath.patch mandel, 2012-07-07 16:47 Patch that adds support for normpath on literal local paths. review
Repositories containing patches
https://bitbucket.org/mandel/normpath-literal-paths
Messages (8)
msg164909 - (view) Author: Manuel de la Pena (mandel) Date: 2012-07-07 16:47
Local literal paths are those paths that do use the \\?\ that allows to have paths longer that the MAX_PATH set by Windows (http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247%28v=vs.85%29.aspx#short_vs._long_names).

While UNC (http://en.wikipedia.org/wiki/Path_%28computing%29) paths should not be normalized, local paths that do use the \\?\ prefix should so that developers that use such a trink to allow longer paths on windows do not have to wrapp the call in the following way:

LONG_PATH_PREFIX = '\\?\'
path = path.replace(LONG_PATH_PREFIX, '')
result = LONG_PATH_PREFIX + os.path.normpath(path)

The possible solution would be for the normalization to work and return the path normalized with the prefix added.
msg164921 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2012-07-07 19:10
Hi mandel :)

With the exception of the "import string" inside of _get_letters (policy is to do all imports at the top), it looks ok just by reading.

Assigning to myself to complete it after I return from holiday in a few days (unless someone beats me).
msg164925 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-07 19:36
I think this is wrong. The MSDN doc says:

“Because it turns off automatic expansion of the path string, the "\\?\" prefix also allows the use of ".." and "." in the path names, which can be useful if you are attempting to perform operations on a file with these otherwise reserved relative path specifiers as part of the fully qualified path.”

(from http://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx#namespaces)

So by "normalizing" the extended path, you are actually changing its meaning.

Furthermore, normpath() is generally wrong in the face of symlinks, meaning it shouldn't be used at all.
msg164985 - (view) Author: Manuel de la Pena (mandel) Date: 2012-07-08 09:58
Antoine,

What the MSDN is stating is that the Windows functions from COM will not normalize the path if it is prefixed by \\?\. That is, if a user wanted to do:

path = r'\\?\C:\Users\mandel\..\Desktop\test'
with open(path, 'w') as fd:
    fd.write('hello!')

he will get the following:

[Errorno 22] Invalid argument. r'\\?\C:\Users\mandel\..\Desktop\test'

The same think would happen if a C function is used, that is, open is doing the right thing. On the other hand, the same code without the \\?\ works.

This makes it even more important to allow the normpath users to normalize such paths, that is, a developer knows that the path has more than 260 chars and wants to make sure that the path can be written in the system:

May I ask you why you mention the symbolic links? I know that if one of the segments of the path is a symbolic link there are problems but this is not related to \\?\ or am I confused? Just curious :)

Brian,

The ntpath module is a little mess (look at my other patch http://bugs.python.org/issue15275) and I think there are more performance problems hidden there somewhere...

I imported string within the function because the same is done in expandvars (around line 430) and wanted to follow the style that was already in use in the file. I do agree that imports at the top are the way to go :)
msg164990 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-08 10:11
> May I ask you why you mention the symbolic links? I know that if one
> of the segments of the path is a symbolic link there are problems but
> this is not related to \\?\ or am I confused? Just curious :)

No, it is not related with "\\?\" but I'm pointing out that normpath()
isn't very useful because of that. And Windows has symlink support
nowadays :-)

For the record, I'm trying to build a saner path-handling library at
http://pypi.python.org/pypi/pathlib/ . I hope to propose it for
inclusion in 3.4.
msg220031 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-08 12:32
As Antoine's pathlib made it into 3.4 is the patch here now obsolete or what?  Also note the reference to issue15275.
msg236992 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-03-01 22:22
As ntpath was cleaned up on #15275 do we need this patch or not, especially given that pathlib made it into 3.4?
msg389263 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-21 23:12
This old issue still needs to be fixed. The check for special_prefixes in ntpath.normpath() must be removed in order to be consistent with WinAPI GetFullPathNameW(). In Windows, one can just call ntpath.abspath() to ensure that nt._getfullpathname() is called. But "Lib/ntpath.py" is cross-platform.
History
Date User Action Args
2022-04-11 14:57:32adminsetgithub: 59491
2021-03-21 23:12:21eryksunsetassignee: brian.curtin ->
components: + Library (Lib)
versions: + Python 3.9, Python 3.10, - Python 3.7
nosy: + paul.moore, zach.ware, steve.dower

messages: + msg389263
stage: patch review -> needs patch
2021-03-21 22:59:12eryksunsetmessages: - msg338058
2019-03-16 07:20:37eryksunsetnosy: + eryksun

messages: + msg338058
versions: + Python 3.7, Python 3.8, - Python 3.3
2019-03-15 22:38:29BreamoreBoysetnosy: - BreamoreBoy
2015-03-01 22:22:22BreamoreBoysetmessages: + msg236992
2014-06-08 12:32:07BreamoreBoysetnosy: + BreamoreBoy
messages: + msg220031
2012-07-08 10:11:18pitrousetmessages: + msg164990
2012-07-08 09:58:57mandelsetmessages: + msg164985
2012-07-07 19:36:18pitrousetnosy: + pitrou
messages: + msg164925
2012-07-07 19:10:32brian.curtinsetkeywords: + needs review
assignee: brian.curtin
type: behavior
messages: + msg164921
2012-07-07 17:38:38pitrousetnosy: + tim.golden, brian.curtin

stage: patch review
2012-07-07 16:47:21mandelcreate