Seen on the buildbots, seems very recent:

http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%202.7/builds/968
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.2/builds/440
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3232

Here’s the distutils error:

ERROR: test_manual_manifest (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1802, in gzopen
    fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
  File "D:\Buildslave\3.x.moore-windows\build\lib\gzip.py", line 145, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: 'c:\\docume~1\\pydev\\locals~1\\temp\\tmptlkdmc\\dist\\fake-1.0.tar.gz'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Buildslave\3.x.moore-windows\build\lib\distutils\tests\test_sdist.py", line 385, in test_manual_manifest
    archive = tarfile.open(archive_name)
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1736, in open
    return func(name, "r", fileobj, **kwargs)
  File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1806, in gzopen
    fileobj.close()
AttributeError: 'NoneType' object has no attribute 'close'

It looks like an issue with a temporary file removed too soon.  I’ll look into it shortly.


For packaging, the bug is related to os.path.relpath.  I actually want to remove the use of relpath because of another bug (not reported yet, will do soon), so we’ll have to live with this failure for a while until the code is changed.

I don’t understand this part:

Traceback (most recent call last):
  File "distutils\tests\test_sdist.py", line 385, in test_manual_manifest
    archive = tarfile.open(archive_name)
  File "tarfile.py", line 1736, in open
    return func(name, "r", fileobj, **kwargs)
  File "tarfile.py", line 1806, in gzopen
    fileobj.close()
AttributeError: 'NoneType' object has no attribute 'close'

The code is protected by a skipUnless(ZLIB_SUPPORT), so how can the tarfile object be None?

This failure needs to be gone in time for 3.2.2. Please find an appropriate way to fix it, even if it is the temporary disabling of a faulty test.

The failure was due to the sdist command having different default formats for each platform (i.e., POSIX: gztar, Win32: zip).  Patch attached.

New changeset 4136acaf03de by Éric Araujo in branch 'default':
Fix sdist test on Windows (#12678).  Patch by Jeremy Kloth.
http://hg.python.org/cpython/rev/4136acaf03de

Looks like the patch is not enough:
http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.2/builds/477/steps/test/logs/stdio/text

Please see #12785 for a patch to packaging.database that should fix test_packaging failures and needs testing on Windows.

I have to log off for today; another committer can use the attached patch so that we get more info.

diff --git a/Lib/distutils/tests/test_sdist.py b/Lib/distutils/tests/test_sdist.py
index 440af98..520289c 100644
--- a/Lib/distutils/tests/test_sdist.py
+++ b/Lib/distutils/tests/test_sdist.py
@@ -381,6 +381,15 @@ class SDistTestCase(PyPIRCCommandTestCase):
 
         self.assertEqual(manifest, ['README.manual'])
 
+        if os.name == 'nt':
+            archive_name = join(self.tmp_dir, 'dist', 'fake-1.0.zip')
+            with zipfile.ZipFile(archive_name, "r", ) as zipfp:
+                archive = zipfp.infolist()
+                filenames = [zipinfo.filename for zipinfo in archive]
+            self.assertEqual(sorted(filenames), ['fake-1.0/PKG-INFO',
+                                                 'fake-1.0/README.manual'])
+            return
+
         archive_name = join(self.tmp_dir, 'dist', 'fake-1.0.tar.gz')
         archive = tarfile.open(archive_name)
         try:

Roumen: I thought too about checking the zipfile on Windows, but with Jeremy’s patch the test force tar.gz generation, so it should amount to the same with less code.

> Looks like the patch is not enough:
> http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.2/builds/477/steps/test/logs/stdio/text

That's a 3.2 builder. You only committed Jeremy's fix to default. I've tested
the fix on 3.2 on my own XP system, and it seems to fix the sdist failure.
test_install.InstallTestCase.test_record_extensions still fails, though.

New changeset c8e73a89150e by Nadeem Vawda in branch '3.2':
Issue #12678: Fix distutils sdist test on Windows.
http://hg.python.org/cpython/rev/c8e73a89150e

New changeset d40856e424fd by Nadeem Vawda in branch '2.7':
Issue #12678: Fix distutils sdist test on Windows.
http://hg.python.org/cpython/rev/d40856e424fd

The sdist failure is now fixed on the 3.2 XP builders.
test_record_extensions still needs attention, though.

I also tried backporting Jeremy's fix to 2.7, but that doesn't seem to
have worked; the sdist tests is still failing there. Someone who
actually knows what they're doing should probably have a look at it at
some point.

>> Looks like the patch is not enough:
> That's a 3.2 builder. You only committed Jeremy's fix to default.
*puts on the stupid hat*

> I also tried backporting Jeremy's fix to 2.7, but that doesn't seem to
> have worked; the sdist tests is still failing there.
I’ll commit the tmp-debug patch so that we can get info from the buildbot, unless Jeremy or someone can investigate.

> test_record_extensions still needs attention, though.
I was hoping 49b4b4ba9f93 would be enough :(

> I’ll commit the tmp-debug patch so that we can get info from the buildbot,
> unless Jeremy or someone can investigate.

If you can wait until this evening (CEST), I'll run the tests with that
patch applied locally. It might be easier to get to the bottom of this if
we co-ordinate via IRC.

I’m in CEST too (France), but I’m not sure I’ll have Internet access this evening.  Would tomorrow afternoon work for you?  I don’t want you to have to dive in distutils more than you should :)

higery: Would you have a bit of free time to help use here?

>>higery: Would you have a bit of free time to help use here?

Certainly yes.

I have run the test_packaging.py, the main error message is about "ValueError: path is on mount 'c:', start on mount 'D:'". One sample is:

ERROR: test_uses (packaging.tests.test_database.TestDistribution)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\add-develop-command\lib\packaging\tests\test_database.py", line 148, in setUp
    os.path.join(distinfo_dir, file)))
  File "D:\add-develop-command\lib\packaging\tests\test_database.py", line 36, in record_pieces
    path = relpath(file, sys.prefix)
  File "D:\add-develop-command\lib\ntpath.py", line 622, in relpath
    raise ValueError(error)
ValueError: path is on mount 'c:', start on mount 'D:'

For test_distutils.py, there is not an 'os.link()' function on Windows platform, so all the tests using this function will report error.

Thanks!  There’s a patch on #12785 that may fix the test_database bug.  Here we try to see the distutils test_sdist problem on 2.7.

>>Here we try to see the distutils test_sdist problem on 2.7.

So sorry that there is not a developed cpython 2.7 on my pc, and my currently installed Python2.7 has not installed the _msi module, so the running of the tests does not work well.

New changeset aa27461759f2 by Éric Araujo in branch 'default':
Try to fix packaging tests using build_ext on Windows (#12678)
http://hg.python.org/cpython/rev/aa27461759f2

Nadeem, please try this patch.

Testing on 3.2, the latest patch doesn't fix the test_record_extensions failure :/

It does change the traceback for the exception, though.

Before:

    ======================================================================
    ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 647, in link
        self.spawn([self.linker] + ld_args)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 34, in spawn
        _spawn_nt(cmd, search_path, dry_run=dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 75, in _spawn_nt
        "command '%s' failed with exit status %d" % (cmd[0], rc))
    distutils.errors.DistutilsExecError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\tests\test_install.py", line 207, in test_record_extensions
        buildcmd.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 345, in run
        self.build_extensions()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 454, in build_extensions
        self.build_extension(ext)
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 541, in build_extension
        target_lang=language)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 717, in link_shared_object
        extra_preargs, extra_postargs, build_temp, target_lang)
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 649, in link
        raise LinkError(msg)
    distutils.errors.LinkError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104


After:

    ======================================================================
    ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 647, in link
        self.spawn([self.linker] + ld_args)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 909, in spawn
        spawn(cmd, dry_run=self.dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 34, in spawn
        _spawn_nt(cmd, search_path, dry_run=dry_run)
      File "C:\Code\cpython\python-3.2\lib\distutils\spawn.py", line 75, in _spawn_nt
        "command '%s' failed with exit status %d" % (cmd[0], rc))
    distutils.errors.DistutilsExecError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "C:\Code\cpython\python-3.2\lib\distutils\tests\test_install.py", line 221, in test_record_extensions
        cmd.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\install.py", line 569, in run
        self.run_command('build')
      File "C:\Code\cpython\python-3.2\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Code\cpython\python-3.2\lib\distutils\dist.py", line 936, in run_command
        cmd_obj.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build.py", line 126, in run
        self.run_command(cmd_name)
      File "C:\Code\cpython\python-3.2\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Code\cpython\python-3.2\lib\distutils\dist.py", line 936, in run_command
        cmd_obj.run()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 345, in run
        self.build_extensions()
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 454, in build_extensions
        self.build_extension(ext)
      File "C:\Code\cpython\python-3.2\lib\distutils\command\build_ext.py", line 541, in build_extension
        target_lang=language)
      File "C:\Code\cpython\python-3.2\lib\distutils\ccompiler.py", line 717, in link_shared_object
        extra_preargs, extra_postargs, build_temp, target_lang)
      File "C:\Code\cpython\python-3.2\lib\distutils\msvc9compiler.py", line 649, in link
        raise LinkError(msg)
    distutils.errors.LinkError: command '"C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\link.exe"' failed with exit status 1104

New changeset 20944ea49392 by Éric Araujo in branch '3.2':
Fix distutils tests on Windows (#12678).
http://hg.python.org/cpython/rev/20944ea49392

New changeset 1696e2789d91 by Éric Araujo in branch 'default':
Fix test_packaging on Windows (#12678).
http://hg.python.org/cpython/rev/1696e2789d91

If the 3.x buildbots are smiling tomorrow, I will backport to 2.7 and distutils2.

New changeset 262e03bbe7a9 by Éric Araujo in branch 'default':
Another (hopefully last) fix for test_packaging on Windws (#12678)
http://hg.python.org/cpython/rev/262e03bbe7a9

New changeset 8ad1670c0f1f by Éric Araujo in branch '2.7':
Try to fix test_distutils on Windows (#12678)
http://hg.python.org/cpython/rev/8ad1670c0f1f

New changeset 6fe19f421a16 by Nadeem Vawda in branch '3.2':
Issue #12678: Fix distutils sdist test on Windows.
http://hg.python.org/cpython/rev/6fe19f421a16

New changeset b173cfbf49a0 by Éric Araujo in branch '3.2':
Fix distutils tests on Windows (#12678).
http://hg.python.org/cpython/rev/b173cfbf49a0

test_packaging is still failing on Windows, example:

======================================================================
ERROR: test_uses (packaging.tests.test_database.TestDistribution)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\tests\test_database.py", line 148, in setUp
    os.path.join(distinfo_dir, file)))
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\packaging\tests\test_database.py", line 36, in record_pieces
    path = relpath(file, sys.prefix)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\ntpath.py", line 622, in relpath
    raise ValueError(error)
ValueError: path is on mount 'c:', start on mount 'D:'

http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/5186/steps/test/logs/stdio

Issue 12785 has a patch for the test_database failures.

Distutils and packaging tests now pass on the three builbots.