classification
Title: os.listdir() crashes on some long and deep paths in Windows 7
Type: crash Stage: resolved
Components: Library (Lib), Windows Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: tim.golden Nosy List: aheejin, tim.golden
Priority: normal Keywords:

Created on 2010-08-12 11:02 by aheejin, last changed 2010-08-13 07:41 by tim.golden. This issue is now closed.

Files
File name Uploaded Description Edit
temp.py aheejin, 2010-08-12 11:02 test script resulting a crash
Messages (6)
msg113672 - (view) Author: Heejin (aheejin) Date: 2010-08-12 11:02
As far as I have seen, this bug only appears in Windows 7. I tested with Python 2.5, 2.6, and 2.7 and they all seem to have this bug.
I tested with Windows 7 Korean version. I'm not sure if other language versions have the same problem.

Reproduction steps:

1. Run a command shell 'cmd'.

2. Run following commands in order:
cd c:\
mkdir rpcc\build
cd rpcc\build
mkdir customs\llvm2dre\llvm-2.6\tools\clang\test\CXX\over\over.match\over.match.best\over.best.ics\over.ics.ellipsis\.svn\tmp\prop-base

3. Create a python script file temp.py with the contents below
import os
path = 'customs\\llvm2dre\\llvm-2.6\\tools\\clang\\test\\CXX\\over\\over.match\\over.match.best\\over.best.ics\\over.ics.ellipsis\\.svn\\tmp\\prop-base'
os.listdir(path)
(I attached this file in this post)

4. Run the script.
python temp.py

5. Then you can see it crashes.
msg113677 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2010-08-12 13:01
I can't get it crash on a path that short. I can produce an error message if I push it beyond the 254 limit, but you can work around that by applying the filesystem namespace prefix:

<code>
import os

path = "c:\\" + "\\".join (130 * ['xx'])
os.makedirs (path)
#
# will fail more or less violently
#
path = "\\\\?\\" + path
os.makedirs (path)
os.listdir (path)

</code>

Probably related issues:

http://bugs.python.org/issue9246
http://bugs.python.org/issue4071
msg113678 - (view) Author: Heejin (aheejin) Date: 2010-08-12 13:57
Thank you for your answer.

Actually I tried that in two Windows7-installed computers
and failed in both. But I don't think it is the problem of
path lengths; I created other paths that long and that deep
but os.listdir() works on them. That's why I wrote the path
verbatim on which the function crashes.
I couldn't find the exact conditions for it to crash.

I haven't heard of that "filesystem namespace prefix"
'\\\\?\\' before. I searched it in the manual and googled it
but couldn't find an explanation about it. But when I modified
the path in my example temp.py it worked! I don't know why though.
Could you point out some informative pages about this prefix?

Unfortunately, the example temp.py is characterizing only the
key reason of the crash. My program uses os.walk() and
os.listdir() was actually called within os.walk(),
so I cannot prepend the path with '\\\\?\\'.

Fortunately I found another workaround for my program,
but this still seems a bug..

Thank you.
msg113679 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2010-08-12 14:04
See: http://msdn.microsoft.com/en-us/library/aa365247%28VS.85%29.aspx

I tried first with your exact path and it caused no issues on
my Win7 box. FWIW you could easily roll your own os.walk
(starting by copying the code that's there) if you needed
to roll in special-cases.

Could you provide a batch file / Python script which exactly
reproduces the issue and a cut-and-paste of the resulting
traceback. (Unless "crash" really does mean "crash").

Would you be able to narrow the issue down any further: see
if any other paths cause the same issue, or if a longer or
a shorter version of the path is problematic.

TJG
msg113746 - (view) Author: Heejin (aheejin) Date: 2010-08-13 06:45
Thank you for your answer.

I cannot provide any error messages or stack traces
because it really *crashed*. python.exe suddenly stops
working and a windows pops up saying something
like "python.exe had a problem and needed to be closed"
I cannot say what the pop-up message exactly was because
I'm using Korean version windows and the message was
Korean. There were no python error messages or stack traces.

And the file temp.py I attached first was the file 
exactly reproducing the situation, at least in my
computer.
And I think it's perhaps a Korean windows problem or
related to some other programs in my computer.
I also tried to narrow down the cause but I couldn't find
any rules.
Maybe it is better to open a thread again when I get to
find some exact conditions on which the problem appears.

Thank you.
msg113748 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2010-08-13 07:41
Thanks for the feedback. I'll close this for now as "works for me". Feel free to reopen if you can come up with anything fresh.
History
Date User Action Args
2010-08-13 07:41:02tim.goldensetstatus: open -> closed
resolution: works for me
messages: + msg113748

stage: resolved
2010-08-13 06:45:11aheejinsetmessages: + msg113746
2010-08-12 14:04:33tim.goldensetmessages: + msg113679
title: os.listdir() crashes on some long and deep paths in Windows 7 -> os.listdir() crashes on some long and deep paths in Windows 7
2010-08-12 13:57:33aheejinsetmessages: + msg113678
2010-08-12 13:01:33tim.goldensetmessages: + msg113677
2010-08-12 11:04:34tim.goldensetassignee: tim.golden

nosy: + tim.golden
versions: - Python 2.5
2010-08-12 11:02:36aheejincreate