classification
Title: test_packaging and test_distutils failures under Windows
Type: behavior Stage: resolved
Components: Distutils, Distutils2, Tests Versions: Python 3.2, Python 3.3, Python 2.7, 3rd party
process
Status: closed Resolution: fixed
Dependencies: 12785 Superseder:
Assigned To: eric.araujo Nosy List: alexis, benjamin.peterson, eric.araujo, georg.brandl, higery, jkloth, nadeem.vawda, paul.moore, python-dev, rpetrov, vstinner
Priority: release blocker Keywords: patch

Created on 2011-08-02 00:54 by pitrou, last changed 2011-09-17 15:26 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
test_sdist.diff jkloth, 2011-08-20 00:36
tmp-debug.diff eric.araujo, 2011-08-21 16:39
windows-build_ext.diff eric.araujo, 2011-08-23 23:06 debugged version review
Messages (34)
msg141542 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-08-02 00:54
Seen on the buildbots, seems very recent:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%202.7/builds/968
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.2/builds/440
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3232
msg141576 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-02 18:20
Here’s the distutils error:

ERROR: test_manual_manifest (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1802, in gzopen
    fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
  File "D:\Buildslave\3.x.moore-windows\build\lib\gzip.py", line 145, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: 'c:\\docume~1\\pydev\\locals~1\\temp\\tmptlkdmc\\dist\\fake-1.0.tar.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\distutils\tests\test_sdist.py", line 385, in test_manual_manifest
    archive = tarfile.open(archive_name)
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1736, in open
    return func(name, "r", fileobj, **kwargs)
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1806, in gzopen
    fileobj.close()
AttributeError: 'NoneType' object has no attribute 'close'

It looks like an issue with a temporary file removed too soon.  I’ll look into it shortly.


For packaging, the bug is related to os.path.relpath.  I actually want to remove the use of relpath because of another bug (not reported yet, will do soon), so we’ll have to live with this failure for a while until the code is changed.
msg142470 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-19 15:59
I don’t understand this part:

Traceback (most recent call last):
  File "distutils\tests\test_sdist.py", line 385, in test_manual_manifest
    archive = tarfile.open(archive_name)
  File "tarfile.py", line 1736, in open
    return func(name, "r", fileobj, **kwargs)
  File "tarfile.py", line 1806, in gzopen
    fileobj.close()
AttributeError: 'NoneType' object has no attribute 'close'

The code is protected by a skipUnless(ZLIB_SUPPORT), so how can the tarfile object be None?
msg142496 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-08-19 21:10
This failure needs to be gone in time for 3.2.2. Please find an appropriate way to fix it, even if it is the temporary disabling of a faulty test.
msg142509 - (view) Author: Jeremy Kloth (jkloth) * Date: 2011-08-20 00:36
The failure was due to the sdist command having different default formats for each platform (i.e., POSIX: gztar, Win32: zip).  Patch attached.
msg142547 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-20 18:02
New changeset 4136acaf03de by Éric Araujo in branch 'default':
Fix sdist test on Windows (#12678).  Patch by Jeremy Kloth.
http://hg.python.org/cpython/rev/4136acaf03de
msg142629 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-21 16:19
Looks like the patch is not enough:
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.2/builds/477/steps/test/logs/stdio/text

Please see #12785 for a patch to packaging.database that should fix test_packaging failures and needs testing on Windows.
msg142632 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-21 16:39
I have to log off for today; another committer can use the attached patch so that we get more info.
msg142634 - (view) Author: Roumen Petrov (rpetrov) * Date: 2011-08-21 17:46
diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
index 440af98..520289c 100644
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -381,6 +381,15 @@ class SDistTestCase(PyPIRCCommandTestCase):
 
         self.assertEqual(manifest, ['README.manual'])
 
+        if os.name == 'nt':
+            archive_name = join(self.tmp_dir, 'dist', 'fake-1.0.zip')
+            with zipfile.ZipFile(archive_name, "r", ) as zipfp:
+                archive = zipfp.infolist()
+                filenames = [zipinfo.filename for zipinfo in archive]
+            self.assertEqual(sorted(filenames), ['fake-1.0/PKG-INFO',
+                                                 'fake-1.0/README.manual'])
+            return
+
         archive_name = join(self.tmp_dir, 'dist', 'fake-1.0.tar.gz')
         archive = tarfile.open(archive_name)
         try:
msg142636 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-21 18:20
Roumen: I thought too about checking the zipfile on Windows, but with Jeremy’s patch the test force tar.gz generation, so it should amount to the same with less code.
msg142641 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2011-08-21 19:49
> Looks like the patch is not enough:
> http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.2/builds/477/steps/test/logs/stdio/text

That's a 3.2 builder. You only committed Jeremy's fix to default. I've tested
the fix on 3.2 on my own XP system, and it seems to fix the sdist failure.
test_install.InstallTestCase.test_record_extensions still fails, though.
msg142645 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-21 20:36
New changeset c8e73a89150e by Nadeem Vawda in branch '3.2':
Issue #12678: Fix distutils sdist test on Windows.
http://hg.python.org/cpython/rev/c8e73a89150e
msg142646 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-21 20:40
New changeset d40856e424fd by Nadeem Vawda in branch '2.7':
Issue #12678: Fix distutils sdist test on Windows.
http://hg.python.org/cpython/rev/d40856e424fd
msg142649 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2011-08-21 22:10
The sdist failure is now fixed on the 3.2 XP builders.
test_record_extensions still needs attention, though.

I also tried backporting Jeremy's fix to 2.7, but that doesn't seem to
have worked; the sdist tests is still failing there. Someone who
actually knows what they're doing should probably have a look at it at
some point.
msg142772 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-22 23:30
>> Looks like the patch is not enough:
> That's a 3.2 builder. You only committed Jeremy's fix to default.
*puts on the stupid hat*

> I also tried backporting Jeremy's fix to 2.7, but that doesn't seem to
> have worked; the sdist tests is still failing there.
I’ll commit the tmp-debug patch so that we can get info from the buildbot, unless Jeremy or someone can investigate.

> test_record_extensions still needs attention, though.
I was hoping 49b4b4ba9f93 would be enough :(
msg142782 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2011-08-23 06:01
> I’ll commit the tmp-debug patch so that we can get info from the buildbot,
> unless Jeremy or someone can investigate.

If you can wait until this evening (CEST), I'll run the tests with that
patch applied locally. It might be easier to get to the bottom of this if
we co-ordinate via IRC.
msg142783 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-23 06:05
I’m in CEST too (France), but I’m not sure I’ll have Internet access this evening.  Would tomorrow afternoon work for you?  I don’t want you to have to dive in distutils more than you should :)
msg142790 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-23 07:50
higery: Would you have a bit of free time to help use here?
msg142805 - (view) Author: higery (higery) Date: 2011-08-23 11:34
>>higery: Would you have a bit of free time to help use here?

Certainly yes.

I have run the test_packaging.py, the main error message is about "ValueError: path is on mount 'c:', start on mount 'D:'". One sample is:

ERROR: test_uses (packaging.tests.test_database.TestDistribution)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\add-develop-command\lib\packaging\tests\test_database.py", line 148, in setUp
    os.path.join(distinfo_dir, file)))
  File "D:\add-develop-command\lib\packaging\tests\test_database.py", line 36, in record_pieces
    path = relpath(file, sys.prefix)
  File "D:\add-develop-command\lib\ntpath.py", line 622, in relpath
    raise ValueError(error)
ValueError: path is on mount 'c:', start on mount 'D:'
msg142806 - (view) Author: higery (higery) Date: 2011-08-23 11:47
For test_distutils.py, there is not an 'os.link()' function on Windows platform, so all the tests using this function will report error.
msg142807 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-23 11:50
Thanks!  There’s a patch on #12785 that may fix the test_database bug.  Here we try to see the distutils test_sdist problem on 2.7.
msg142819 - (view) Author: higery (higery) Date: 2011-08-23 12:59
>>Here we try to see the distutils test_sdist problem on 2.7.

So sorry that there is not a developed cpython 2.7 on my pc, and my currently installed Python2.7 has not installed the _msi module, so the running of the tests does not work well.
msg142860 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-23 19:42
New changeset aa27461759f2 by Éric Araujo in branch 'default':
Try to fix packaging tests using build_ext on Windows (#12678)
http://hg.python.org/cpython/rev/aa27461759f2
msg142862 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-23 20:38
Nadeem, please try this patch.
msg142864 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2011-08-23 21:20
Testing on 3.2, the latest patch doesn't fix the test_record_extensions failure :/

It does change the traceback for the exception, though.

Before:

    ======================================================================
    ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 647, in link
        self.spawn([self.linker] + ld_args)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 34, in spawn
        _spawn_nt(cmd, search_path, dry_run=dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 75, in _spawn_nt
        "command '%s' failed with exit status %d" % (cmd[0], rc))
    distutils.errors.DistutilsExecError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\tests\test_install.py", line 207, in test_record_extensions
        buildcmd.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 345, in run
        self.build_extensions()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 454, in build_extensions
        self.build_extension(ext)
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 541, in build_extension
        target_lang=language)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 717, in link_shared_object
        extra_preargs, extra_postargs, build_temp, target_lang)
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 649, in link
        raise LinkError(msg)
    distutils.errors.LinkError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104


After:

    ======================================================================
    ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 647, in link
        self.spawn([self.linker] + ld_args)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 34, in spawn
        _spawn_nt(cmd, search_path, dry_run=dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 75, in _spawn_nt
        "command '%s' failed with exit status %d" % (cmd[0], rc))
    distutils.errors.DistutilsExecError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\tests\test_install.py", line 221, in test_record_extensions
        cmd.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\install.py", line 569, in run
        self.run_command('build')
      File "C:\Code\cpython\python-3.2\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Code\cpython\python-3.2\lib\distutils\dist.py", line 936, in run_command
        cmd_obj.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build.py", line 126, in run
        self.run_command(cmd_name)
      File "C:\Code\cpython\python-3.2\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Code\cpython\python-3.2\lib\distutils\dist.py", line 936, in run_command
        cmd_obj.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 345, in run
        self.build_extensions()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 454, in build_extensions
        self.build_extension(ext)
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 541, in build_extension
        target_lang=language)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 717, in link_shared_object
        extra_preargs, extra_postargs, build_temp, target_lang)
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 649, in link
        raise LinkError(msg)
    distutils.errors.LinkError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104
msg142871 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-23 23:59
New changeset 20944ea49392 by Éric Araujo in branch '3.2':
Fix distutils tests on Windows (#12678).
http://hg.python.org/cpython/rev/20944ea49392
msg142872 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-24 00:15
New changeset 1696e2789d91 by Éric Araujo in branch 'default':
Fix test_packaging on Windows (#12678).
http://hg.python.org/cpython/rev/1696e2789d91
msg142873 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-24 00:18
If the 3.x buildbots are smiling tomorrow, I will backport to 2.7 and distutils2.
msg142975 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-25 16:14
New changeset 262e03bbe7a9 by Éric Araujo in branch 'default':
Another (hopefully last) fix for test_packaging on Windws (#12678)
http://hg.python.org/cpython/rev/262e03bbe7a9
msg143012 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-26 14:37
New changeset 8ad1670c0f1f by Éric Araujo in branch '2.7':
Try to fix test_distutils on Windows (#12678)
http://hg.python.org/cpython/rev/8ad1670c0f1f
msg143475 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-09-04 06:41
New changeset 6fe19f421a16 by Nadeem Vawda in branch '3.2':
Issue #12678: Fix distutils sdist test on Windows.
http://hg.python.org/cpython/rev/6fe19f421a16

New changeset b173cfbf49a0 by Éric Araujo in branch '3.2':
Fix distutils tests on Windows (#12678).
http://hg.python.org/cpython/rev/b173cfbf49a0
msg144051 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-09-14 20:55
test_packaging is still failing on Windows, example:

======================================================================
ERROR: test_uses (packaging.tests.test_database.TestDistribution)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\tests\test_database.py", line 148, in setUp
    os.path.join(distinfo_dir, file)))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\tests\test_database.py", line 36, in record_pieces
    path = relpath(file, sys.prefix)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\ntpath.py", line 622, in relpath
    raise ValueError(error)
ValueError: path is on mount 'c:', start on mount 'D:'

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/5186/steps/test/logs/stdio
msg144052 - (view) Author: Nadeem Vawda (nadeem.vawda) * (Python committer) Date: 2011-09-14 21:14
Issue 12785 has a patch for the test_database failures.
msg144194 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-09-17 15:26
Distutils and packaging tests now pass on the three builbots.
History
Date User Action Args
2011-09-17 15:26:07eric.araujosetstatus: open -> closed
resolution: fixed
messages: + msg144194

stage: needs patch -> resolved
2011-09-14 21:14:14nadeem.vawdasetmessages: + msg144052
2011-09-14 20:55:37vstinnersetnosy: + vstinner
messages: + msg144051
2011-09-04 06:41:15python-devsetmessages: + msg143475
2011-08-26 14:37:34python-devsetmessages: + msg143012
2011-08-25 16:14:26python-devsetmessages: + msg142975
2011-08-24 00:18:25eric.araujosetmessages: + msg142873
versions: + 3rd party
2011-08-24 00:15:36python-devsetmessages: + msg142872
2011-08-23 23:59:22python-devsetmessages: + msg142871
2011-08-23 23:06:56eric.araujosetfiles: - windows-build_ext.diff
2011-08-23 23:06:47eric.araujosetfiles: + windows-build_ext.diff
2011-08-23 21:21:27nadeem.vawdasetmessages: - msg142863
2011-08-23 21:20:26nadeem.vawdasetmessages: + msg142864
2011-08-23 21:16:37nadeem.vawdasetmessages: + msg142863
2011-08-23 20:38:11eric.araujosetfiles: + windows-build_ext.diff

messages: + msg142862
2011-08-23 19:42:05python-devsetmessages: + msg142860
2011-08-23 12:59:14higerysetmessages: + msg142819
2011-08-23 11:50:57eric.araujosetmessages: + msg142807
2011-08-23 11:47:44higerysetmessages: + msg142806
2011-08-23 11:34:34higerysetmessages: + msg142805
2011-08-23 07:50:10eric.araujosetnosy: + higery
messages: + msg142790
2011-08-23 06:05:43eric.araujosetmessages: + msg142783
2011-08-23 06:01:59nadeem.vawdasetmessages: + msg142782
2011-08-23 01:57:01pitrousetnosy: - pitrou
2011-08-22 23:30:42eric.araujosetmessages: + msg142772
2011-08-21 22:10:59nadeem.vawdasetmessages: + msg142649
2011-08-21 20:40:27python-devsetmessages: + msg142646
2011-08-21 20:36:42python-devsetmessages: + msg142645
2011-08-21 19:49:52nadeem.vawdasetnosy: + nadeem.vawda
messages: + msg142641
2011-08-21 18:20:12eric.araujosetmessages: + msg142636
2011-08-21 17:46:03rpetrovsetnosy: + rpetrov
messages: + msg142634
2011-08-21 16:39:36eric.araujosetfiles: + tmp-debug.diff

messages: + msg142632
2011-08-21 16:19:01eric.araujosetmessages: + msg142629
2011-08-21 11:00:14paul.mooresetnosy: + paul.moore
2011-08-20 18:02:33python-devsetnosy: + python-dev
messages: + msg142547
2011-08-20 00:36:52jklothsetfiles: + test_sdist.diff

nosy: + jkloth
messages: + msg142509

keywords: + patch
2011-08-19 21:10:38georg.brandlsetpriority: high -> release blocker
nosy: + georg.brandl, benjamin.peterson
messages: + msg142496

2011-08-19 15:59:44eric.araujosetmessages: + msg142470
2011-08-19 15:57:40eric.araujosetdependencies: + list_distinfo_file is wrong
2011-08-02 18:20:18eric.araujosetmessages: + msg141576
2011-08-02 00:54:15pitroucreate