classification
Title: sysconfig.get_config_vars('srcdir') fails in specific cases
Type: Stage: resolved
Components: Distutils, Distutils2, Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: tarek Nosy List: alexis, eric.araujo, nadeem.vawda, ned.deily, pitrou, python-dev, r.david.murray, tarek
Priority: normal Keywords: patch

Created on 2011-05-21 22:32 by tarek, last changed 2011-09-04 06:41 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue12141_3x.patch ned.deily, 2011-06-13 19:02 3.3 version for packaging and distutils
issue12132_backport_32.patch ned.deily, 2011-06-13 19:03 3.2 backport of distutils part of issue12132
issue12141_32.patch ned.deily, 2011-06-13 19:04 3.2 version for distutils
issue12141_27.patch ned.deily, 2011-06-13 19:04 2.7 version for distutils
Messages (20)
msg136483 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-21 22:32
this test module looks for sysconfig.get_config_var('srcdir') which in turns uses the sys,executable path.

multiprocess seems to change it in every process, leading to the errors.

To reproduce:

./python Lib/test/regrtest.py -j2 -v test_packaging

A workaround is to skip the test in case the file is not found, but we need to fix it because it boils down to sysconfig being broken in multiprocess
msg136484 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-21 22:34
What's the difference with issue12132?
msg136486 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-21 22:40
Oops. It's a duplicate. Keeping this one since the problem was narrowed to multiprocessing/sys,executable and sysconfig
msg136487 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-21 22:50
> Oops. It's a duplicate. Keeping this one since the problem was
> narrowed to multiprocessing/sys,executable and sysconfig

I don't think it has anything to do with sys.executable. As the other
issue says, it seems the test (or packaging/sysconfig itself) is simply
unable to locate the root of the current checkout (which should be
trivial using __file__).
msg136489 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-21 22:59
sysconfig is looking for the source dir when 

  sysconfig.get_config_var('srcdir')

is called.

And this is done like this:


if sys.executable:
    _PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable))
else:
    # sys.executable can be empty if argv[0] has been changed and Python is
    # unable to retrieve the real program name
    _PROJECT_BASE = _safe_realpath(os.getcwd())


Because sys.executable (argv[0] in fact) is not filled when you call multiprocess vvia the -j option.

So yeah, this has to do with sys.executable
msg136493 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-05-21 23:24
> Because sys.executable (argv[0] in fact) is not filled when you call
> multiprocess vvia the -j option.

sys.executable is set perfectly well when running regrtest in
multiprocess mode, otherwise many other tests would fail.

Try:
$ grep "sys.executable" Lib/test/*
Lib/test/regrtest.py:585:        base_cmd = [sys.executable] + opt_args + ['-m', 'test.regrtest']
Lib/test/script_helper.py:20:    cmd_line = [sys.executable]
Lib/test/script_helper.py:60:    cmd_line = [sys.executable, '-E']
Lib/test/test_base64.py:231:        args = (sys.executable, '-m', 'base64') + args
Lib/test/test_capi.py:42:        p = subprocess.Popen([sys.executable, "-c",
[etc.]

Again, it seems sysconfig simply has a bug which you can reproduce on
the command line:

$ pwd
/home/antoine/cpython/default/build
$ ../python -c "import sys, sysconfig; print(sys.executable,
sysconfig.get_config_var('srcdir'))"
/home/antoine/cpython/default/build/../python /home/antoine/cpython/default/build
msg136494 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-05-21 23:46
The test should be changed anyway to avoid the dependency on "srcdir", whose value has no meaning when the tests are not run from a simple in-source-directory build.  The test could create its own temp copy of xxmodule.c.
msg136568 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2011-05-22 20:15
@ned: right. done, and fixes issue12132
msg138269 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-06-13 19:02
Here are patches to install a copy of xxmodule.c in the distutils tests directory (for 3.3, 3.2, and 2.7) and a copy in packaging tests for 3.3.  With them in place, test_build_ext/test_command_build_ext now executes when the tests are run from an installed Python where no source directory is available and each test case run copies the c file into its temp dir.
 
Currently, those test are being silently skipped in the installed case or, worse, picking up the wrong version of xxmodule.c depending on what sysconfig.get_config_vars('srcdir') returns (this happened to me when I reused a source directory name in some builds).   For 3.2, a patch is included to backport the Distutils part of the fix for issue12132.
msg138283 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-06-13 22:29
Is it really necessary to copy this file?  I haven't looked at the tests, but I'm wondering if they are using xxmodule.c only because it already existed.  Could the test instead use a much simpler .c file that it creates on the fly?
msg138284 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-06-13 22:37
I think one of the objectives of the test is to see that it works with the C API of that particular Python release as captured in its version of xxmodule.c.  And, while it could be argued that both distutils and packaging don't need their own copies, I think it is cleaner to keep the two testing environment uncoupled.
msg138319 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-14 14:58
Path LGTM.

Also +1 on keeping distutils and packaging wholly separate, including in tests infrastructure.  It’s just one file.
msg138609 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-18 21:45
See also #10764.
msg139001 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-06-24 23:23
Are the patches good to go?  And would you like me to apply them?
msg139012 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-06-25 00:48
I think they are ready, I gave a +1 (with a typo: path instead of patch :) two messages ago.
msg139349 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-06-28 07:54
New changeset c8ffa3891d5e by Ned Deily in branch '2.7':
Issue #12141: Install a copy of template C module file so that
http://hg.python.org/cpython/rev/c8ffa3891d5e

New changeset de226a510b52 by Ned Deily in branch '3.2':
Issue #12141: Install a copy of template C module file so that
http://hg.python.org/cpython/rev/de226a510b52

New changeset ef8e9e99de88 by Ned Deily in branch 'default':
Issue #12141: Install copies of template C module file so that
http://hg.python.org/cpython/rev/ef8e9e99de88
msg139350 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2011-06-28 08:03
Patches applied as described above for 3.3, 3.2.1, and 2.7.3. I'm setting the status of the issue to pending and, assuming there are no buildbot failures in the near future, I will close it unless anyone sees a reason not to.
msg140438 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-15 16:07
BTW, doesn’t the change to Makefile.pre.in need to be ported to the MSI build system too?
msg142548 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-08-20 18:02
New changeset 7d9fa30c5588 by Éric Araujo in branch '3.2':
Refactor the copying of xxmodule.c in distutils tests (#12141).
http://hg.python.org/cpython/rev/7d9fa30c5588

New changeset 900738175779 by Éric Araujo in branch 'default':
Refactor the copying of xxmodule.c in packaging tests (#12141).
http://hg.python.org/cpython/rev/900738175779
msg143478 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-09-04 06:41
New changeset 402c4bbedf9c by Éric Araujo in branch '3.2':
Refactor the copying of xxmodule.c in distutils tests (#12141).
http://hg.python.org/cpython/rev/402c4bbedf9c
History
Date User Action Args
2011-09-04 06:41:51python-devsetmessages: + msg143478
2011-08-20 18:02:33python-devsetmessages: + msg142548
2011-07-15 16:07:01eric.araujosetmessages: + msg140438
2011-06-29 00:21:51ned.deilysetstatus: pending -> closed
2011-06-28 08:03:07ned.deilysetstatus: open -> pending
resolution: fixed
messages: + msg139350

stage: patch review -> resolved
2011-06-28 07:55:00python-devsetnosy: + python-dev
messages: + msg139349
2011-06-25 00:48:42eric.araujosetmessages: + msg139012
versions: - Python 3.1
2011-06-24 23:23:02ned.deilysetmessages: + msg139001
2011-06-18 21:45:44eric.araujosetmessages: + msg138609
2011-06-14 14:58:52eric.araujosetmessages: + msg138319
2011-06-13 22:37:46ned.deilysetmessages: + msg138284
2011-06-13 22:29:24r.david.murraysetnosy: + r.david.murray
messages: + msg138283
2011-06-13 19:04:48ned.deilysetfiles: + issue12141_27.patch
2011-06-13 19:04:13ned.deilysetfiles: + issue12141_32.patch
2011-06-13 19:03:37ned.deilysetfiles: + issue12132_backport_32.patch
2011-06-13 19:02:42ned.deilysetfiles: + issue12141_3x.patch

components: + Distutils, Distutils2

keywords: + patch
nosy: + alexis
messages: + msg138269
stage: patch review
2011-05-23 15:05:29eric.araujosetnosy: + eric.araujo

versions: - Python 3.4
2011-05-22 20:15:00tareksetversions: + Python 3.1, Python 2.7, Python 3.2, Python 3.3, Python 3.4
title: --multiprocessing fails with packaging.tests.test_command_build_ext -> sysconfig.get_config_vars('srcdir') fails in specific cases
messages: + msg136568

assignee: pitrou -> tarek
components: + Library (Lib)
2011-05-21 23:46:46ned.deilysetnosy: + ned.deily
messages: + msg136494
2011-05-21 23:24:38pitrousetmessages: + msg136493
2011-05-21 22:59:11tareksetmessages: + msg136489
2011-05-21 22:50:18pitrousetmessages: + msg136487
2011-05-21 22:50:10nadeem.vawdasetnosy: + nadeem.vawda
2011-05-21 22:40:59tareksetmessages: + msg136486
2011-05-21 22:39:35tareklinkissue12132 superseder
2011-05-21 22:34:26pitrousetmessages: + msg136484
2011-05-21 22:32:30tarekcreate