Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_sysconfigdata is generated in srcdir, not builddir #59503

Closed
doko42 opened this issue Jul 8, 2012 · 27 comments
Closed

_sysconfigdata is generated in srcdir, not builddir #59503

doko42 opened this issue Jul 8, 2012 · 27 comments
Labels
build The build process and cross-build type-bug An unexpected behavior, bug, or error

Comments

@doko42
Copy link
Member

doko42 commented Jul 8, 2012

BPO 15298
Nosy @birkenfeld, @doko42, @pitrou, @ned-deily, @tpn, @merwok
Files
  • sysconfigdata.diff
  • alt_sysconfigdata.patch
  • sysconfig.py.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2013-02-04.17:36:35.281>
    created_at = <Date 2012-07-08.21:26:15.939>
    labels = ['type-bug', 'build']
    title = '_sysconfigdata is generated in srcdir, not builddir'
    updated_at = <Date 2013-02-04.17:36:35.280>
    user = 'https://github.com/doko42'

    bugs.python.org fields:

    activity = <Date 2013-02-04.17:36:35.280>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2013-02-04.17:36:35.281>
    closer = 'ned.deily'
    components = ['Build']
    creation = <Date 2012-07-08.21:26:15.939>
    creator = 'doko'
    dependencies = []
    files = ['26345', '27127', '27601']
    hgrepos = []
    issue_num = 15298
    keywords = ['patch']
    message_count = 27.0
    messages = ['165036', '165139', '165201', '165206', '165285', '165322', '165343', '165344', '166031', '168554', '169865', '169869', '169873', '169878', '169880', '169881', '169882', '169947', '172813', '173034', '173137', '173141', '173143', '173160', '173214', '181336', '181355']
    nosy_count = 10.0
    nosy_names = ['georg.brandl', 'doko', 'pitrou', 'ned.deily', 'trent', 'eric.araujo', 'Arfrever', 'devurandom', 'python-dev', 'sbt']
    pr_nums = []
    priority = 'critical'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue15298'
    versions = ['Python 3.3', 'Python 3.4']

    @doko42
    Copy link
    Member Author

    doko42 commented Jul 8, 2012

    _sysconfigdata is generated in srcdir, not builddir, so if you do two consecutive differently builds in different builddirs and using the same srcdir, then the _sysconfigdata of the second build wins. _sysconfigdata.py has to be built in the builddir, not the srcdir.

    @doko42 doko42 added release-blocker build The build process and cross-build type-bug An unexpected behavior, bug, or error labels Jul 8, 2012
    @doko42
    Copy link
    Member Author

    doko42 commented Jul 10, 2012

    looks like the module can be put into $builddir/cat pybuilddir.txt. However pybuilddir.txt is written/computed by the just built python. So either

    • implement distutils.util.get_platform in configure.ac

    • or write a temporary pybuilddir.txt, call the just built
      python -c 'print("build/lib.%s-%" % (distutils.util.get_platform(), sys.version[:3]))' to write the final pybuilddir.txt, move the _sysconfigdata.py to this location.
      and remove the pybuilddir.txt writing in setup.py

    @doko42
    Copy link
    Member Author

    doko42 commented Jul 10, 2012

    proposed patch, tested with configure, build and install

    @pitrou
    Copy link
    Member

    pitrou commented Jul 10, 2012

    The patch is rather ugly, I think. You should arrange for sysconfigdata to be generated directly at the right place instead (not sure how, perhaps it should be done from setup.py).

    Also, there's no need for this to be a release blocker.

    @ned-deily
    Copy link
    Member

    (Once this issue is resolved, a permanent fix for the minor OS X test case failure of bpo-15188 can be developed and applied.)

    @doko42
    Copy link
    Member Author

    doko42 commented Jul 12, 2012

    while I appreciate the fix for bpo-13150, it's the patch for bpo-13150 which introduces the ugliness to build the file in the srcdir in the first place.

    re-setting the bug severity. the current behaviour can result in a bogus installation; please let the release manager decide on it.

    @doko42
    Copy link
    Member Author

    doko42 commented Jul 13, 2012

    like other platform dependent files _sysconfigdata.py should be installed in plat-*

    @pitrou
    Copy link
    Member

    pitrou commented Jul 13, 2012

    like other platform dependent files _sysconfigdata.py should be installed in plat-*

    What are you talking about? plat-* files are only OS-specific, while
    _sysconfigdata also depends on configure options and other stuff.

    @birkenfeld
    Copy link
    Member

    The bug certainly is a valid annoyance, but I won't hold up beta2 for it.

    @birkenfeld
    Copy link
    Member

    It seems there is no urge to fix this before 3.3 -- fine with me.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Sep 5, 2012

    If sysconfig._generate_posix_vars() creates the build directory and pybuilddir.txt (instead of setup.py) then it could write _sysconfigdata.py in the correct place. Then setup.py would not have to mess with its own sys.path.

    One issue with having _sysconfigdata.py in the build dir is that distutils copies it (along with everything else in the build dir) to .../lib/python3.x/lib-dynload. So I think Matthias' patch results in two copies of _sysconfigdata.py being installed (plus __pycache__/_sysconfigdata.pyc).

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Sep 5, 2012

    Alternative patch.

    @merwok
    Copy link
    Member

    merwok commented Sep 5, 2012

    I did not test the patch but it does not look bad.

    @doko42
    Copy link
    Member Author

    doko42 commented Sep 5, 2012

    looks ok, two issues are:

    builddir should have added
    +("-pydebug" if hasattr(sys, "gettotalrefcount") else "")

    the installation installs this to the lib-dynload directory, which maybe is not desired.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Sep 5, 2012

    builddir should have added
    +("-pydebug" if hasattr(sys, "gettotalrefcount") else "")

    Oops, yes.

    the installation installs this to the lib-dynload directory, which
    maybe is not desired.

    The lines

    + -rm $(DESTDIR)$(DESTSHARED)/_sysconfigdata.py
    + -rm -r $(DESTDIR)$(DESTSHARED)/pycache

    in the patch remove them from lib-dynload.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Sep 5, 2012

    Updated patch.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Sep 5, 2012

    Try again.

    @doko42
    Copy link
    Member Author

    doko42 commented Sep 6, 2012

    looks fine. from my point of view this should go to the 3.3 branch after the 3.3.0 release, and to the trunk.

    @tpn
    Copy link
    Member

    tpn commented Oct 13, 2012

    The previous 'alt_sysconfigdata.patch' didn't apply cleanly to 3.3/3.x. I've attached an updated patch that does.

    Any objections to applying this to 3.3 -> 3.x now that the freeze is over?

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 16, 2012

    New changeset 24d52d3060e8 by Trent Nelson in branch '3.3':
    Issue bpo-15298: ensure _sysconfigdata is generated in build directory,
    http://hg.python.org/cpython/rev/24d52d3060e8

    New changeset f85c3f4d9b98 by Trent Nelson in branch 'default':
    Merge issue bpo-15298: ensure _sysconfigdata is generated in build directory,
    http://hg.python.org/cpython/rev/f85c3f4d9b98

    @ned-deily
    Copy link
    Member

    There seems to be a bootstrap issue here. Building with ./configure --with-pydebug --prefix=... on OS X in a clean source directory (hg purge --all), 'make' makes it to building the static libpython .a but then dies on the sysconfig generate-posix-vars step:

    ar rc libpython3.4dm.a Modules/_threadmodule.o  Modules/signalmodule.o  Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/operator.o  Modules/_collectionsmodule.o  Modules/itertoolsmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/symtablemodule.o  Modules/xxsubtype.o
    /usr/bin/ranlib: file: libpython3.4dm.a(dynamic_annotations.o) has no symbols
    /usr/bin/ranlib: file: libpython3.4dm.a(pymath.o) has no symbols
    ranlib libpython3.4dm.a
    ranlib: file: libpython3.4dm.a(dynamic_annotations.o) has no symbols
    ranlib: file: libpython3.4dm.a(pymath.o) has no symbols
    /usr/bin/clang   -framework CoreFoundation -o python Modules/python.o libpython3.4dm.a -ldl  -framework CoreFoundation     
    ./python -E -S -m sysconfig --generate-posix-vars
    Could not find platform dependent libraries <exec_prefix>
    Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
    Traceback (most recent call last):
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/runpy.py", line 160, in _run_module_as_main
        "__main__", fname, loader, pkg_name)
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/runpy.py", line 73, in _run_code
        exec(code, run_globals)
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 683, in <module>
        _main()
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 671, in _main
        _generate_posix_vars()
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 374, in _generate_posix_vars
        pybuilddir = 'build/lib.%s-%s' % (get_platform(), sys.version[:3])
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 651, in get_platform
        get_config_vars(),
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 520, in get_config_vars
        _init_posix(_CONFIG_VARS)
      File "/py/dev/default/b10.7_t10.7_x4.3_cclang_d/unix/source/Lib/sysconfig.py", line 393, in _init_posix
        from _sysconfigdata import build_time_vars
    ImportError: No module named '_sysconfigdata'

    @tpn
    Copy link
    Member

    tpn commented Oct 17, 2012

    Ah, this is because of this OS X-specific snippet in sysconfig.get_platform():

    elif osname[:6] == "darwin":
        import _osx_support
        osname, release, machine = _osx_support.get_platform_osx(
                                            get_config_vars(),
                                            osname, release, machine)
    

    get_config_vars() results in _init_posix() eventually being called, which attempts to import _sysconfigdata.

    Ugh. Looking into a patch now.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 17, 2012

    New changeset 66959d419369 by Trent Nelson in branch '3.3':
    Issue bpo-15298: fix an OS X bootstrap issue with _sysconfigdata.py.
    http://hg.python.org/cpython/rev/66959d419369

    New changeset 429774e8d9f9 by Trent Nelson in branch 'default':
    Merge issue bpo-15298: fix an OS X bootstrap issue with _sysconfigdata.py.
    http://hg.python.org/cpython/rev/429774e8d9f9

    @tpn
    Copy link
    Member

    tpn commented Oct 17, 2012

    That last commit fixes in-tree builds, but broke out-of-tree builds. I've refactored the patch (sysconfig.py.patch) to fix the issue, but it's getting progressively more hacky, so I wanted to see what other people think before proceeding.

    (That being said, it works a *lot* better than the previous solution of writing _sysconfigdata.py to the root builddir, then moving it. That didn't work at all for out-of-tree builds. This solution slots _sysconfigdata directly into sys.modules, which _init_posix() is more than happy with.)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 17, 2012

    New changeset 1250498db562 by Trent Nelson in branch '3.3':
    Issue bpo-15298: refactor previous fix from 66959d419369.
    http://hg.python.org/cpython/rev/1250498db562

    New changeset a0614e041bc8 by Trent Nelson in branch 'default':
    Merge issue bpo-15298.
    http://hg.python.org/cpython/rev/a0614e041bc8

    @merwok
    Copy link
    Member

    merwok commented Feb 4, 2013

    Can this be closed?

    @ned-deily
    Copy link
    Member

    On OS X, Trent's fixes solved the bootstrap issue and _sysconfigdata.py is now created in buildir. Closing.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants