this test module looks for sysconfig.get_config_var('srcdir') which in turns uses the sys,executable path.

multiprocess seems to change it in every process, leading to the errors.

To reproduce:

./python Lib/test/regrtest.py -j2 -v test_packaging

A workaround is to skip the test in case the file is not found, but we need to fix it because it boils down to sysconfig being broken in multiprocess

What's the difference with bpo-12132?

Oops. It's a duplicate. Keeping this one since the problem was narrowed to multiprocessing/sys,executable and sysconfig

Oops. It's a duplicate. Keeping this one since the problem was
narrowed to multiprocessing/sys,executable and sysconfig

I don't think it has anything to do with sys.executable. As the other
issue says, it seems the test (or packaging/sysconfig itself) is simply
unable to locate the root of the current checkout (which should be
trivial using __file__).

sysconfig is looking for the source dir when

  sysconfig.get_config_var('srcdir')

is called.

And this is done like this:

if sys.executable:
    _PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable))
else:
    # sys.executable can be empty if argv[0] has been changed and Python is
    # unable to retrieve the real program name
    _PROJECT_BASE = _safe_realpath(os.getcwd())

Because sys.executable (argv[0] in fact) is not filled when you call multiprocess vvia the -j option.

So yeah, this has to do with sys.executable

Because sys.executable (argv[0] in fact) is not filled when you call
multiprocess vvia the -j option.

sys.executable is set perfectly well when running regrtest in
multiprocess mode, otherwise many other tests would fail.

Try:
$ grep "sys.executable" Lib/test/*
Lib/test/regrtest.py:585: base_cmd = [sys.executable] + opt_args + ['-m', 'test.regrtest']
Lib/test/script_helper.py:20: cmd_line = [sys.executable]
Lib/test/script_helper.py:60: cmd_line = [sys.executable, '-E']
Lib/test/test_base64.py:231: args = (sys.executable, '-m', 'base64') + args
Lib/test/test_capi.py:42: p = subprocess.Popen([sys.executable, "-c",
[etc.]

Again, it seems sysconfig simply has a bug which you can reproduce on
the command line:

$ pwd
/home/antoine/cpython/default/build
$ ../python -c "import sys, sysconfig; print(sys.executable,
sysconfig.get_config_var('srcdir'))"
/home/antoine/cpython/default/build/../python /home/antoine/cpython/default/build

The test should be changed anyway to avoid the dependency on "srcdir", whose value has no meaning when the tests are not run from a simple in-source-directory build. The test could create its own temp copy of xxmodule.c.

@ned: right. done, and fixes bpo-12132

Here are patches to install a copy of xxmodule.c in the distutils tests directory (for 3.3, 3.2, and 2.7) and a copy in packaging tests for 3.3. With them in place, test_build_ext/test_command_build_ext now executes when the tests are run from an installed Python where no source directory is available and each test case run copies the c file into its temp dir.

Currently, those test are being silently skipped in the installed case or, worse, picking up the wrong version of xxmodule.c depending on what sysconfig.get_config_vars('srcdir') returns (this happened to me when I reused a source directory name in some builds). For 3.2, a patch is included to backport the Distutils part of the fix for bpo-12132.

Is it really necessary to copy this file? I haven't looked at the tests, but I'm wondering if they are using xxmodule.c only because it already existed. Could the test instead use a much simpler .c file that it creates on the fly?

I think one of the objectives of the test is to see that it works with the C API of that particular Python release as captured in its version of xxmodule.c. And, while it could be argued that both distutils and packaging don't need their own copies, I think it is cleaner to keep the two testing environment uncoupled.

Path LGTM.

Also +1 on keeping distutils and packaging wholly separate, including in tests infrastructure. It’s just one file.

See also bpo-10764.

Are the patches good to go? And would you like me to apply them?

I think they are ready, I gave a +1 (with a typo: path instead of patch :) two messages ago.

New changeset c8ffa3891d5e by Ned Deily in branch '2.7':
Issue bpo-12141: Install a copy of template C module file so that
http://hg.python.org/cpython/rev/c8ffa3891d5e

New changeset de226a510b52 by Ned Deily in branch '3.2':
Issue bpo-12141: Install a copy of template C module file so that
http://hg.python.org/cpython/rev/de226a510b52

New changeset ef8e9e99de88 by Ned Deily in branch 'default':
Issue bpo-12141: Install copies of template C module file so that
http://hg.python.org/cpython/rev/ef8e9e99de88

Patches applied as described above for 3.3, 3.2.1, and 2.7.3. I'm setting the status of the issue to pending and, assuming there are no buildbot failures in the near future, I will close it unless anyone sees a reason not to.

BTW, doesn’t the change to Makefile.pre.in need to be ported to the MSI build system too?

New changeset 7d9fa30c5588 by Éric Araujo in branch '3.2':
Refactor the copying of xxmodule.c in distutils tests (bpo-12141).
http://hg.python.org/cpython/rev/7d9fa30c5588

New changeset 900738175779 by Éric Araujo in branch 'default':
Refactor the copying of xxmodule.c in packaging tests (bpo-12141).
http://hg.python.org/cpython/rev/900738175779

New changeset 402c4bbedf9c by Éric Araujo in branch '3.2':
Refactor the copying of xxmodule.c in distutils tests (bpo-12141).
http://hg.python.org/cpython/rev/402c4bbedf9c