classification
Title: Windows test_import failure thanks to ImportError.path
Type: behavior Stage: needs patch
Components: Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: 15169 Superseder:
Assigned To: brett.cannon Nosy List: brett.cannon, brian.curtin, pitrou, python-dev, r.david.murray, rpetrov, skrah
Priority: normal Keywords: buildbot

Created on 2012-04-16 17:52 by r.david.murray, last changed 2012-07-13 17:58 by brett.cannon. This issue is now closed.

Messages (18)
msg158489 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-04-16 17:52
Not sure which revision triggered this, so opening a new bug:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/6381/steps/test/logs/stdio

There's also a test_reprlib failure, no idea if it is related.
msg158501 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-16 21:17
Issue #14581 covers the .PY failure (still looking for input on that one).

The path failure is from http://hg.python.org/cpython/rev/f341b99bb370 which Brian committed.

The test_reprlib failure is because of a race condition where an importlib.invalidate_caches() call is needed.
msg158503 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2012-04-16 21:20
Not sure why test_extension_import_fail is failing - not seeing that here.
msg158517 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-17 00:48
New changeset eae7cc54d28b by Brett Cannon in branch 'default':
Issue #14599: Make test_reprlib robust against import cache race
http://hg.python.org/cpython/rev/eae7cc54d28b
msg158519 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-17 00:50
New changeset 00a07720a570 by Brett Cannon in branch 'default':
Issue #14599: Fix an import caching race condition.
http://hg.python.org/cpython/rev/00a07720a570
msg158520 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-04-17 00:51
I think the failure is from a cache race condition. Hopefully the invalidate_caches() call will fix things.
msg158875 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-20 19:23
New changeset 573010778eed by Brett Cannon in branch 'default':
Issue #14599: Generalize a test for ImportError.path and add support
http://hg.python.org/cpython/rev/573010778eed
msg158876 - (view) Author: Roundup Robot (python-dev) Date: 2012-04-20 19:31
New changeset 56aa4cda11a8 by Brett Cannon in branch 'default':
Issue #14599: Support ImportError.path on AIX and HPUX when loading
http://hg.python.org/cpython/rev/56aa4cda11a8
msg162598 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-06-10 20:48
I think the issue mentioned in eae7cc54d28b just occurred again:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/6807/steps/test/logs/stdio

======================================================================
ERROR: test_module (test.test_reprlib.LongReprTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_reprlib.py", line 257, in test_module
    from areallylongpackageandmodulenametotestreprtruncation.areallylongpackageandmodulenametotestreprtruncation import areallylongpackageandmodulenametotestreprtruncation
  File "<frozen importlib._bootstrap>", line 1333, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 1286, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1253, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 776, in load_module
  File "<frozen importlib._bootstrap>", line 757, in load_module
  File "<frozen importlib._bootstrap>", line 408, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 633, in _load_module
  File "<frozen importlib._bootstrap>", line 743, in get_code
  File "<frozen importlib._bootstrap>", line 819, in set_data
  File "<frozen importlib._bootstrap>", line 127, in _write_atomic
FileNotFoundError: [Errno 2] No such file or directory

----------------------------------------------------------------------
msg162621 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-11 15:05
If you look at line 127 in importlib/_bootstrap.py you will see that it is an os.open() call to open the bytecode file for exclusive writing. I'm willing to bet the buildbot didn't have the directory writable or something and that triggered the issue. Regardless, this has nothing to do with the finder's cache.

Antoine might have a little more insight since he wrote the atomic code initially. If not then this issue should be closed.
msg162642 - (view) Author: Roumen Petrov (rpetrov) * Date: 2012-06-11 20:07
There is one long standing issue with length of the build path ...
msg162646 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-11 21:51
Roumen, what issue is that? Do you have an issue # you can share?
msg162654 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2012-06-12 00:47
Brett, I reopened this because you said earlier that the test_reprlib
failure is due to a race condition where an invalidate_caches()
call is needed.

You're quite right of course that the new occurrence could be caused
by something unrelated. I can't reproduce the failure on Windows 7,
so it might be limited to XP.
msg162679 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-12 14:56
It's not a problem, Stefan. I just happened to have already added the importlib.invalidate_caches() call to test_reprlib so I know that isn't the issue.
msg162727 - (view) Author: Roumen Petrov (rpetrov) * Date: 2012-06-13 20:41
The issue is related to MAXPATHLEN limit and how is implemented . One part of request are already closed , another part wait . Some people port patches other wrote that would not use very long path. I'm in the second group.

This buildbot use path with length 54 and since build tree is detected and to this path code adds  'build'. For installed python  code will use TEMP environment variable.
The test are run in subdir test_python_${PID} and depending of PID the path is with different length. Note this.

So try to reproduce with installed python and TEMP set to path like this C:\12345678\12345678\12345678\12345678\12345678\12345678 and run test. The limit of 260(-1) will be reached and FileNotFoundError will be raised.
The default value of TEMP is to short to trigger error.
msg162800 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-06-14 15:29
Thanks for the info, Roumen. It looks like there is a single use of MAXPATHLEN still in import.c which I hope to eliminate, so hopefully that will rectify this problem.
msg165394 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-13 17:57
New changeset 9e164b404983 by Brett Cannon in branch 'default':
Issues #15169, #14599: Make PyImport_ExecCodeModuleWithPathnames() use
http://hg.python.org/cpython/rev/9e164b404983
msg165396 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2012-07-13 17:58
MAXPATHLEN no longer shows up in Python/import.c.
History
Date User Action Args
2012-07-13 17:58:39brett.cannonsetstatus: open -> closed
resolution: fixed
dependencies: + Clear C code under PyImport_ExecCodeModuleObject()
messages: + msg165396
2012-07-13 17:57:12python-devsetmessages: + msg165394
2012-06-14 15:29:06brett.cannonsetmessages: + msg162800
2012-06-13 20:41:27rpetrovsetmessages: + msg162727
2012-06-12 14:56:13brett.cannonsetmessages: + msg162679
2012-06-12 00:47:32skrahsetmessages: + msg162654
2012-06-11 21:51:52brett.cannonsetmessages: + msg162646
2012-06-11 20:07:05rpetrovsetnosy: + rpetrov
messages: + msg162642
2012-06-11 15:05:52brett.cannonsetmessages: + msg162621
2012-06-10 20:48:11skrahsetstatus: closed -> open

nosy: + skrah
messages: + msg162598

resolution: fixed -> (no value)
2012-04-20 19:54:12brett.cannonsetstatus: open -> closed
assignee: brett.cannon
resolution: fixed
2012-04-20 19:31:18python-devsetmessages: + msg158876
2012-04-20 19:23:18python-devsetmessages: + msg158875
2012-04-20 17:45:13brett.cannonsettitle: Windows test_import failure -> Windows test_import failure thanks to ImportError.path
2012-04-17 00:51:57brett.cannonsetmessages: + msg158520
2012-04-17 00:50:39python-devsetmessages: + msg158519
2012-04-17 00:48:56python-devsetnosy: + python-dev
messages: + msg158517
2012-04-16 21:20:31brian.curtinsetmessages: + msg158503
2012-04-16 21:17:28brett.cannonsetnosy: + brian.curtin
messages: + msg158501
2012-04-16 17:52:20r.david.murraycreate