classification
Title: test_tarfile: test_extractall_symlinks() fails on Windows with: [WinError 206] The filename or extension is too long
Type: Stage:
Components: Tests, Windows Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, pablogsal, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2019-07-09 11:34 by vstinner, last changed 2019-08-22 09:39 by vstinner.

Messages (5)
msg347549 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-09 11:34
AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/2862

FAIL: test_extractall_symlinks (test.test_tarfile.Bz2WriteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_tarfile.py", line 1312, in test_extractall_symlinks
    tar.extractall(path=tempdir)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\buildarea\\3.x.ware-win81-release.nondebug\\build\\build\\test_python_3164\\test_python_worker_5512\\@test_5512_tmp-tardir\\testsymlinks\\buildarea\\3.x.ware-win81-release.nondebug\\build\\build\\test_python_3164\\test_python_worker_5512\\@test_5512_tmp-tardir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_tarfile.py", line 1314, in test_extractall_symlinks
    self.fail("extractall failed with symlinked files")
AssertionError: extractall failed with symlinked files


Extract of pythoninfo:

windows.RtlAreLongPathsEnabled: <function not available>

RtlAreLongPathsEnabled() is not available on Windows 8.1. I don't know if long paths support is enabled or not on this buildbot (I guess that no, it isn't).

I'm not sure why the test started to fail.
msg347550 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-09 11:35
On this buildbot build, test_tarfile succeeded when run again. Likely because the test is run in the main regrtest process which doesn't have "test_python_worker_xxx" in its current working directory.
msg347597 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-07-10 04:30
If you want to harden the test for Windows, you could transform TEMPDIR
into an extended path. os.path._getfinalpathname always returns an extended path. For example:

    >>> os.path._getfinalpathname('C:/Temp')
    '\\\\?\\C:\\Temp'

    >>> os.path._getfinalpathname('//localhost/C$/Temp')
    '\\\\?\\UNC\\localhost\\C$\\Temp'

> windows.RtlAreLongPathsEnabled: <function not available>
> 
> RtlAreLongPathsEnabled() is not available on Windows 8.1. I don't 
> know if long paths support is enabled or not on this buildbot (I 
> guess that no, it isn't).

Normalized long paths are implemented in Windows 10 1607 and later. Previous versions do not have RtlAreLongPathsEnabled and cannot support normalized long paths. RtlAreLongPathsEnabled() will be true if, at process startup, the "LongPathsEnabled" policy is enabled in the registry and the application manifest claims to be "longPathAware". 

In all supported versions of Windows, long paths are supported by most filesystem functions if we use an extended path, i.e. a fully-qualified Unicode path that starts with the \\?\ prefix. This path type is not normalized in a create or open context, so it can only use backslash as the path separator, not forward slash. Also, the process working directory does not allow extended paths, so when using extended paths we have to manage our own working directory and manually resolve relative paths, including "." and ".." components (e.g. by calling GetFullPathNameW).
msg350143 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-08-22 00:34
I disabled the long path support on my own build and can reproduce this by passing a long path to --tempdir

Unfortunately, I'm not familiar enough with what tarfile is trying to do here - why is it recreating the entire directory structure within itself?

Browsing through lib/tarfile.py, I suspect it's the mix of "/" literals and os.path functions that's at fault here. It probably requires a much bigger job to go through and fix it up than simply fixing tests.
msg350177 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-08-22 09:39
Hum, tests are still failing, example on AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/2992
History
Date User Action Args
2019-08-22 09:39:08vstinnersetmessages: + msg350177
2019-08-22 00:34:07steve.dowersetnosy: + paul.moore, tim.golden, steve.dower
messages: + msg350143
components: + Windows
2019-07-10 04:30:47eryksunsetnosy: + eryksun
messages: + msg347597
2019-07-09 11:35:29vstinnersetmessages: + msg347550
2019-07-09 11:34:25vstinnersetnosy: + pablogsal
2019-07-09 11:34:17vstinnercreate