classification
Title: _sysconfigdata is generated in srcdir, not builddir
Type: behavior Stage: resolved
Components: Build Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, devurandom, doko, eric.araujo, georg.brandl, ned.deily, pitrou, python-dev, sbt, trent
Priority: critical Keywords: patch

Created on 2012-07-08 21:26 by doko, last changed 2013-02-04 17:36 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
sysconfigdata.diff doko, 2012-07-10 20:11 review
alt_sysconfigdata.patch sbt, 2012-09-05 17:05 review
sysconfig.py.patch trent, 2012-10-17 11:38 review
Messages (27)
msg165036 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-07-08 21:26
_sysconfigdata is generated in srcdir, not builddir, so if you do two consecutive differently builds in different builddirs and using the same srcdir, then the _sysconfigdata of the second build wins. _sysconfigdata.py has to be built in the builddir, not the srcdir.
msg165139 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-07-10 01:37
looks like the module can be put into $builddir/`cat pybuilddir.txt`. However pybuilddir.txt is written/computed by the just built python. So either

 - implement distutils.util.get_platform in configure.ac

 - or write a temporary pybuilddir.txt, call the just built
   python -c 'print("build/lib.%s-%" % (distutils.util.get_platform(), sys.version[:3]))' to write the final pybuilddir.txt, move the _sysconfigdata.py to this location.
   and remove the pybuilddir.txt writing in setup.py
msg165201 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-07-10 20:11
proposed patch, tested with configure, build and install
msg165206 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-10 21:57
The patch is rather ugly, I think. You should arrange for sysconfigdata to be generated directly at the right place instead (not sure how, perhaps it should be done from setup.py).

Also, there's no need for this to be a release blocker.
msg165285 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-07-12 07:27
(Once this issue is resolved, a permanent fix for the minor OS X test case failure of Issue15188 can be developed and applied.)
msg165322 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-07-12 16:54
while I appreciate the fix for #13150, it's the patch for #13150 which introduces the ugliness to build the file in the srcdir in the first place.

re-setting the bug severity. the current behaviour can result in a bogus installation; please let the release manager decide on it.
msg165343 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-07-13 00:03
like other platform dependent files _sysconfigdata.py should be installed in plat-*
msg165344 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-13 00:13
> like other platform dependent files _sysconfigdata.py should be installed in plat-*

What are you talking about? plat-* files are only OS-specific, while
_sysconfigdata also depends on configure options and other stuff.
msg166031 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-07-21 14:26
The bug certainly is a valid annoyance, but I won't hold up beta2 for it.
msg168554 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-08-19 10:54
It seems there is no urge to fix this before 3.3 -- fine with me.
msg169865 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-05 10:55
If sysconfig._generate_posix_vars() creates the build directory and pybuilddir.txt (instead of setup.py) then it could write _sysconfigdata.py in the correct place.  Then setup.py would not have to mess with its own sys.path.

One issue with having _sysconfigdata.py in the build dir is that distutils copies it (along with everything else in the build dir) to .../lib/python3.x/lib-dynload.  So I think Matthias' patch results in two copies of _sysconfigdata.py being installed (plus __pycache__/_sysconfigdata.pyc).
msg169869 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-05 13:01
Alternative patch.
msg169873 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-09-05 14:52
I did not test the patch but it does not look bad.
msg169878 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-09-05 15:35
looks ok, two issues are:

builddir should have added
  +("-pydebug" if hasattr(sys, "gettotalrefcount") else "")

the installation installs this to the lib-dynload directory, which maybe is not desired.
msg169880 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-05 16:55
> builddir should have added
>  +("-pydebug" if hasattr(sys, "gettotalrefcount") else "")

Oops, yes.

> the installation installs this to the lib-dynload directory, which 
> maybe is not desired.

The lines

+	-rm $(DESTDIR)$(DESTSHARED)/_sysconfigdata.py
+	-rm -r $(DESTDIR)$(DESTSHARED)/__pycache__

in the patch remove them from lib-dynload.
msg169881 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-05 17:01
Updated patch.
msg169882 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-09-05 17:04
Try again.
msg169947 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2012-09-06 22:35
looks fine. from my point of view this should go to the 3.3 branch after the 3.3.0 release, and to the trunk.
msg172813 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-13 17:18
The previous 'alt_sysconfigdata.patch' didn't apply cleanly to 3.3/3.x.  I've attached an updated patch that does.

Any objections to applying this to 3.3 -> 3.x now that the freeze is over?
msg173034 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-16 12:17
New changeset 24d52d3060e8 by Trent Nelson in branch '3.3':
Issue #15298: ensure _sysconfigdata is generated in build directory,
http://hg.python.org/cpython/rev/24d52d3060e8

New changeset f85c3f4d9b98 by Trent Nelson in branch 'default':
Merge issue #15298: ensure _sysconfigdata is generated in build directory,
http://hg.python.org/cpython/rev/f85c3f4d9b98
msg173137 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-10-17 07:35
There seems to be a bootstrap issue here.  Building with ./configure --with-pydebug --prefix=... on OS X in a clean source directory (hg purge --all), 'make' makes it to building the static libpython .a but then dies on the sysconfig generate-posix-vars step:

ar rc libpython3.4dm.a Modules/_threadmodule.o  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/operator.o  Modules/_collectionsmodule.o  Modules/itertoolsmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/symtablemodule.o  Modules/xxsubtype.o
/usr/bin/ranlib: file: libpython3.4dm.a(dynamic_annotations.o) has no symbols
/usr/bin/ranlib: file: libpython3.4dm.a(pymath.o) has no symbols
ranlib libpython3.4dm.a
ranlib: file: libpython3.4dm.a(dynamic_annotations.o) has no symbols
ranlib: file: libpython3.4dm.a(pymath.o) has no symbols
/usr/bin/clang   -framework CoreFoundation -o python Modules/python.o libpython3.4dm.a -ldl  -framework CoreFoundation     
./python -E -S -m sysconfig --generate-posix-vars
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 683, in <module>
    _main()
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 671, in _main
    _generate_posix_vars()
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 374, in _generate_posix_vars
    pybuilddir = 'build/lib.%s-%s' % (get_platform(), sys.version[:3])
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 651, in get_platform
    get_config_vars(),
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 520, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 393, in _init_posix
    from _sysconfigdata import build_time_vars
ImportError: No module named '_sysconfigdata'
msg173141 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-17 08:08
Ah, this is because of this OS X-specific snippet in sysconfig.get_platform():

    elif osname[:6] == "darwin":
        import _osx_support
        osname, release, machine = _osx_support.get_platform_osx(
                                            get_config_vars(),
                                            osname, release, machine)

get_config_vars() results in _init_posix() eventually being called, which attempts to import _sysconfigdata.

Ugh.  Looking into a patch now.
msg173143 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-17 08:24
New changeset 66959d419369 by Trent Nelson in branch '3.3':
Issue #15298: fix an OS X bootstrap issue with _sysconfigdata.py.
http://hg.python.org/cpython/rev/66959d419369

New changeset 429774e8d9f9 by Trent Nelson in branch 'default':
Merge issue #15298: fix an OS X bootstrap issue with _sysconfigdata.py.
http://hg.python.org/cpython/rev/429774e8d9f9
msg173160 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-17 11:38
That last commit fixes in-tree builds, but broke out-of-tree builds.  I've refactored the patch (sysconfig.py.patch) to fix the issue, but it's getting progressively more hacky, so I wanted to see what other people think before proceeding.

(That being said, it works a *lot* better than the previous solution of writing _sysconfigdata.py to the root builddir, then moving it.  That didn't work at all for out-of-tree builds.  This solution slots _sysconfigdata directly into sys.modules, which _init_posix() is more than happy with.)
msg173214 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-17 22:08
New changeset 1250498db562 by Trent Nelson in branch '3.3':
Issue #15298: refactor previous fix from 66959d419369.
http://hg.python.org/cpython/rev/1250498db562

New changeset a0614e041bc8 by Trent Nelson in branch 'default':
Merge issue #15298.
http://hg.python.org/cpython/rev/a0614e041bc8
msg181336 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-02-04 15:35
Can this be closed?
msg181355 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-04 17:36
On OS X, Trent's fixes solved the bootstrap issue and _sysconfigdata.py is now created in buildir.  Closing.
History
Date User Action Args
2013-02-04 17:36:35ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg181355

stage: resolved
2013-02-04 15:35:50eric.araujosetmessages: + msg181336
2012-10-17 22:08:52python-devsetmessages: + msg173214
2012-10-17 11:38:54trentsetfiles: + sysconfig.py.patch

messages: + msg173160
2012-10-17 08:47:19devurandomsetnosy: + devurandom
2012-10-17 08:24:59python-devsetmessages: + msg173143
2012-10-17 08:08:33trentsetmessages: + msg173141
2012-10-17 07:35:58ned.deilysetmessages: + msg173137
2012-10-16 12:17:46python-devsetnosy: + python-dev
messages: + msg173034
2012-10-13 22:42:50trentsetfiles: - issue15298-alt_sysconfigdata2.patch
2012-10-13 17:18:18trentsetfiles: + issue15298-alt_sysconfigdata2.patch

messages: + msg172813
versions: + Python 3.4
2012-10-13 16:46:20trentsetnosy: + trent
2012-09-06 22:35:26dokosetkeywords: - needs review

messages: + msg169947
2012-09-05 17:06:04sbtsetfiles: - alt_sysconfigdata.patch
2012-09-05 17:05:55sbtsetfiles: - alt_sysconfigdata.patch
2012-09-05 17:05:44sbtsetfiles: + alt_sysconfigdata.patch
2012-09-05 17:04:58sbtsetfiles: + alt_sysconfigdata.patch

messages: + msg169882
2012-09-05 17:02:41sbtsetfiles: - alt_sysconfigdata.patch
2012-09-05 17:01:28sbtsetfiles: + alt_sysconfigdata.patch

messages: + msg169881
2012-09-05 16:55:33sbtsetmessages: + msg169880
2012-09-05 15:35:30dokosetmessages: + msg169878
2012-09-05 14:52:45eric.araujosetmessages: + msg169873
2012-09-05 13:01:39sbtsetfiles: + alt_sysconfigdata.patch

messages: + msg169869
2012-09-05 10:55:02sbtsetnosy: + sbt
messages: + msg169865
2012-09-04 07:47:20Arfreversetnosy: + Arfrever
2012-08-19 10:54:55georg.brandlsetpriority: deferred blocker -> critical

messages: + msg168554
2012-07-22 10:12:16ned.deilyunlinkissue15188 dependencies
2012-07-21 14:26:33georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg166031
2012-07-13 00:13:29pitrousetmessages: + msg165344
2012-07-13 00:03:29dokosetmessages: + msg165343
2012-07-12 17:18:06pitrousetnosy: + georg.brandl
2012-07-12 16:54:34dokosetpriority: normal -> release blocker

messages: + msg165322
2012-07-12 07:30:00ned.deilylinkissue15188 dependencies
2012-07-12 07:27:52ned.deilysetnosy: + ned.deily
messages: + msg165285
2012-07-10 21:57:51pitrousetpriority: release blocker -> normal

messages: + msg165206
2012-07-10 20:11:21dokosetkeywords: + needs review, patch
files: + sysconfigdata.diff
messages: + msg165201
2012-07-10 01:37:37dokosetmessages: + msg165139
stage: needs patch -> (no value)
2012-07-09 05:03:42eric.araujosetnosy: + pitrou, eric.araujo

stage: needs patch
2012-07-08 21:26:15dokocreate