Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to build Python out-of-tree when source tree is readonly. #60023

Closed
tpn opened this issue Aug 30, 2012 · 44 comments
Closed

Unable to build Python out-of-tree when source tree is readonly. #60023

tpn opened this issue Aug 30, 2012 · 44 comments
Assignees
Labels
build The build process and cross-build

Comments

@tpn
Copy link
Member

tpn commented Aug 30, 2012

BPO 15819
Nosy @loewis, @doko42, @gpshead, @pitrou, @vstinner, @ned-deily, @tpn, @ericsnowcurrently, @vadmium, @zware, @koobs
PRs
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
  • Dependencies
  • bpo-28066: [Patch] Fix the ability to cross compile Python when doing a rebuild of importlib.h
  • Files
  • readonly_src.patch
  • hgtouch.diff
  • unused-flags.patch: Unneeded code in Py 2
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tpn'
    closed_at = <Date 2016-09-16.02:48:55.797>
    created_at = <Date 2012-08-30.06:49:21.582>
    labels = ['build']
    title = 'Unable to build Python out-of-tree when source tree is\treadonly.'
    updated_at = <Date 2017-03-31.16:36:17.495>
    user = 'https://github.com/tpn'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:17.495>
    actor = 'dstufft'
    assignee = 'trent'
    closed = True
    closed_date = <Date 2016-09-16.02:48:55.797>
    closer = 'martin.panter'
    components = []
    creation = <Date 2012-08-30.06:49:21.582>
    creator = 'trent'
    dependencies = ['28066']
    files = ['27054', '28874', '44574']
    hgrepos = []
    issue_num = 15819
    keywords = ['patch']
    message_count = 44.0
    messages = ['169430', '169431', '169432', '169433', '169434', '169435', '169437', '169444', '169458', '169461', '169462', '169463', '169480', '169484', '169486', '169489', '169490', '169534', '169535', '169541', '169542', '169544', '169545', '169546', '169552', '169558', '169569', '169595', '173038', '173041', '173042', '173046', '173144', '180788', '181177', '203348', '275751', '275768', '275910', '275948', '275995', '276260', '276265', '276667']
    nosy_count = 15.0
    nosy_names = ['loewis', 'doko', 'gregory.p.smith', 'pitrou', 'vstinner', 'ned.deily', 'trent', 'rpetrov', 'Arfrever', 'python-dev', 'sbt', 'eric.snow', 'martin.panter', 'zach.ware', 'koobs']
    pr_nums = ['552']
    priority = 'low'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue15819'
    versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

    @tpn
    Copy link
    Member Author

    tpn commented Aug 30, 2012

    This is somewhat related to Issue bpo-9860. You can't currently build Python out-of-tree when the source directory is readonly. This is because there are a few Parser/pgen/AST/python_asdl.py steps that try and write to $(srcdir).

    The attached patch fixes this.

    @tpn tpn self-assigned this Aug 30, 2012
    @tpn tpn added the build The build process and cross-build label Aug 30, 2012
    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 30, 2012

    Instead of using "mkdir -p", it should use $(MKDIR_P) (which actually isn't normally "mkdir -p" :-) Other than that, the patch looks fine.

    @tpn
    Copy link
    Member Author

    tpn commented Aug 30, 2012

    Hmmmmm. Now I'm confused. Is this what you meant?

    -------------------

    % hg diff
    diff -r 019a2390b014 Makefile.pre.in
    --- a/Makefile.pre.in   Tue Aug 21 23:41:43 2012 +0000
    +++ b/Makefile.pre.in   Thu Aug 30 07:16:55 2012 +0000
    @@ -43,6 +43,8 @@
     
     GNULD=         @GNULD@
     
    +MKDIR_P=       @MKDIR_P@
    +
     # Shell used by make (some versions default to the login shell, which is bad)
     SHELL=         /bin/sh
     
    @@ -223,8 +225,8 @@
     
     ##########################################################################
     # Grammar
    -GRAMMAR_H=     $(srcdir)/Include/graminit.h
    -GRAMMAR_C=     $(srcdir)/Python/graminit.c
    +GRAMMAR_H=     Include/graminit.h
    +GRAMMAR_C=     Python/graminit.c
     GRAMMAR_INPUT= $(srcdir)/Grammar/Grammar
     
     
    @@ -266,9 +268,9 @@
     
     ##########################################################################
     # AST
    -AST_H_DIR=     $(srcdir)/Include
    +AST_H_DIR=     Include
     AST_H=         $(AST_H_DIR)/Python-ast.h
    -AST_C_DIR=     $(srcdir)/Python
    +AST_C_DIR=     Python
     AST_C=         $(AST_C_DIR)/Python-ast.c
     AST_ASDL=      $(srcdir)/Parser/Python.asdl
     
    @@ -605,9 +607,11 @@
     Parser/pgenmain.o:     $(srcdir)/Include/parsetok.h
     
     $(AST_H): $(AST_ASDL) $(ASDLGEN_FILES)
    +       $(MKDIR_P) $(AST_H_DIR)
            $(ASDLGEN) -h $(AST_H_DIR) $(AST_ASDL)
     
     $(AST_C): $(AST_ASDL) $(ASDLGEN_FILES)
    +       $(MKDIR_P) $(AST_C_DIR)
            $(ASDLGEN) -c $(AST_C_DIR) $(AST_ASDL)
     
     Python/compile.o Python/symtable.o Python/ast.o: $(GRAMMAR_H) $(AST_H)
    diff -r 019a2390b014 configure
    --- a/configure Tue Aug 21 23:41:43 2012 +0000
    +++ b/configure Thu Aug 30 07:16:55 2012 +0000
    @@ -658,6 +658,7 @@
     GNULD
     LINKCC
     LDVERSION
    +MKDIR_P
     RUNSHARED
     INSTSONAME
     LDLIBRARYDIR
    @@ -5262,6 +5263,7 @@
         HGBRANCH=""
     fi
     
    +MKDIR_P="mkdir -p"
     
     DISABLE_ASDLGEN=""
     # Extract the first word of "python", so it can be a program name with args.

    Given your "which actually isn't normally mkdir -p" comment... my patch doesn't seem right.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 30, 2012

    Am 30.08.12 09:19, schrieb Trent Nelson:

    Hmmmmm. Now I'm confused. Is this what you meant?

    No. Makefile.pre.in *already* has a definition of MKDIR_P, no need to
    add one.

    @loewis loewis mannequin changed the title Unable to build Python out-of-tree when source tree is readonly. Unable to build Python out-of-tree when source tree is readonly. Aug 30, 2012
    @tpn
    Copy link
    Member Author

    tpn commented Aug 30, 2012

    That's what I figured you meant initially... Until I couldn't find any MKDIR_P definitions, and couldn't get it working without the hack above.

    I thought it might be an implicit make variable as well, but, not so much. (Tested with BSD make and gmake on FreeBSD.)

    @akheron
    Copy link
    Member

    akheron commented Aug 30, 2012

    In 3.3, Makefile.pre.in already defines MKDIR_P, in 3.2 it doesn't.

    @tpn
    Copy link
    Member Author

    tpn commented Aug 30, 2012

    Ah, I see the MKDIR_P changes in 3.x/configure as well.

    Objections to backporting to 3.2?

    Is 2.7 off the table?

    (Building from a readonly source is handy for Snakebite; saves having a dozen machines needing to use their own hg clone.)

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 30, 2012

    Am 30.08.12 09:47, schrieb Trent Nelson:

    Objections to backporting to 3.2?

    Is 2.7 off the table?

    No, adding AC_PROG_MKDIR_P into all of them is fine.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Aug 30, 2012

    New changeset 62044cd5b19b by Trent Nelson in branch '3.2':
    Issue bpo-15819: Fix out-of-tree builds from a readonly source.
    http://hg.python.org/cpython/rev/62044cd5b19b

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Aug 30, 2012

    New changeset af36536988fd by Trent Nelson in branch 'default':
    Block 78809:62044cd5b19b (Issue bpo-15819) from 3.2.
    http://hg.python.org/cpython/rev/af36536988fd

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Aug 30, 2012

    New changeset ab6ab44921b2 by Trent Nelson in branch 'default':
    Issue bpo-15819: Fix out-of-tree builds from a readonly source.
    http://hg.python.org/cpython/rev/ab6ab44921b2

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Aug 30, 2012

    New changeset 2dde5a7439fd by Trent Nelson in branch '2.7':
    Issue bpo-15819: Fix out-of-tree builds from a readonly source.
    http://hg.python.org/cpython/rev/2dde5a7439fd

    @tpn tpn closed this as completed Aug 30, 2012
    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Aug 30, 2012

    I tried building from out-of-tree with a read-only srcdir, but I found two other files which the build process tries to create in the source directory:

    $(srcdir)/Objects/typeslots.inc
    $(srcdir)/Lib/_sysconfigdata.py
    

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 30, 2012

    What I liked about Trent's original bug report is that it didn't just bring up a generic issue (cannot build with read-only source code), but clarified that his concern really was graminit.

    So I'd like to declare that this (graminit) is the focus of this issue: any other aspects of read-only source trees are out of scope, and should be reported separately.

    That said:

    As for typeslots.inc: this shouldn't really be an issue for a released version of Python, since its tarball should include the generated file, and the timestamps should be in an order so that generation is unnecessary. This should be the case for all generated files that are also checked into Mercurial. For checkouts from Mercurial, you should do "make touch" after an update, as this will bring the timestamps in the right order.

    It seems that graminit is missing from .hgtouch (in which case Trent's patch probably wouldn't have been necessary, except that it also fixes older branches). But that is again a different issue.

    As for sysconfigdata: this really should be a separate issue. It's more difficult, since installation and packaging needs to pick up the file even if it's not in the source tree.

    @ned-deily
    Copy link
    Member

    The sysconfigdata problem is already documented in two open issues: bpo-14774 and bpo-15298.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 30, 2012

    As a further follow-up: the original code (in 67ed8a6905c3, by Neil Schemenauer from 2001), he explicitly changed GRAMMAR_C/H to refer to srcdir, claiming that errors from building them would be ignored.

    In r87558, for which I can't find a subversion revsision for, Victor Stinner dropped the Makefile "-" marker, making the read-only case fail.

    In reviewing the patch more, I think it is actually incorrect: it now generates the files into Include, but they aren't considered there since (builddir)/Include is not added to the -I options.

    So the two options now are:
    a) revert the patch, and restore Neil's original approach, or
    b) add Include to the -I options, if srcdir is not "."

    Reopening for consideration.

    @loewis loewis mannequin reopened this Aug 30, 2012
    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 30, 2012

    Reconsidering more: it would be desirable if even an out-of-source-tree compilation wouldn't require Python for bootstrapping; with the committed patch, it does since it will always run ASDLGEN.

    If the source version of the generated files is fresh enough, they should just be copied to the destination.

    I'm not quite sure how to achieve that with Make rules. This probably *is* a separate issue (which would become irrelevant if the patch here is reverted).

    @tpn
    Copy link
    Member Author

    tpn commented Aug 31, 2012

    Well, that escalated quickly :-)

    I think I should clarify my use case that resulted in raising this bug.

    1. Create a fresh hg clone, say, ~/hg/cpython-3.2.
    2. Make it readonly: zfs set readonly=on tank/home/cpython/hg/cpython-3.2
    3. cd /tmp/cpython-3.2-build
    4. /home/cpython/hg/cpython-3.2/configure --srcdir=/home/cpython/hg/cpython-3.2/configure

    So, my patches address that use case. The thing I like about the approach above is that I can set the source directory to readonly immediately -- I don't have to do an intermediate ./configure in that directory in order to generate a Makefile so I can make touch. (Side note: I'm almost certain make touch wouldn't work for my use case.)

    Richard: what was your use case? What steps did you take that resulted in getting typeslots.inc and _sysconfigdata.py written to?

    ....although I just tried an out-of-tree build with 'default' using the exact steps above and got the same error with _sysconfigdata.py.

    Martin: I think I understand your arguments, and I (think) I agree with them. Especially your last point with regards to always running ASDLGEN.

    I think I'm +1 on reverting and trying the '-' approach originally used by Neil.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 31, 2012

    Am 31.08.12 15:19, schrieb Trent Nelson:

    I think I should clarify my use case that resulted in raising this
    bug.

    1. Create a fresh hg clone, say, ~/hg/cpython-3.2.
    2. Make it readonly

    The use case is really uncontended, and long-standing. People want
    to build from CD-ROM sources, from sources mounted via NFS,
    from sources owned by some other user - there are really many
    reasons why you want to build in a separate directory, and why
    you then do not want the original source be clobbered.

    Richard: what was your use case?

    I guess pretty much the same (though perhaps not ZFS).

    What steps did you take that
    resulted in getting typeslots.inc and _sysconfigdata.py written to?

    Touch Include/typeslots.h (before making the checkout readonly)
    to see this problem. On a hg checkout, it really depends on the order
    in which Mercurial creates the files to either run or not run into one
    of these issues.

    You were lucky that the timestamps happened to be in an order were
    in an order that caused typeslots.inc to be newer than its sources.
    AFAIK, hg uses alphapbetical order when updating files, so this may
    have helped.

    I think I'm +1 on reverting and trying the '-' approach originally
    used by Neil.

    Please don't. I think Victor dropped the - because it would mask
    errors from running pgen. pgen might fail, but pgen.stamp is still
    generated. Then, on a subsequent run, pgen would not run again.
    Of course, pgen.stamp is a later addition (by me), so this wasn't
    originally a problem.

    I think the regular case should be that the time stamps are in
    the right order in the source tree, so none of the generators
    should run. But if they happen to be in the wrong order (either
    because of Mercurial, or because one of the sources was actually
    modified), then the generators should run and output into the
    build directory.

    Since the committed patch is already a step in that direction,
    I think it now needs to be completed. It's mandatory (IMO)
    that the generated files are actually used; that the generators
    are not run if not needed is optional and can be fixed later.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Aug 31, 2012

    Richard: what was your use case?

    Building on the main host and a VM without having to commit and synchronize temporary changes. (The VM has read-only access to the host's repositories).

    What steps did you take that resulted in getting typeslots.inc and
    _sysconfigdata.py written to?

    As Martin says it is probably just the timestamps which caused typeslots.inc to be rebuilt. In 3.3 _sysconfigdata.py is rebuilt whenever the python binary is.

    I suspect a bigger issue is the fact that *.pyc files cannot be written to $(srcdir)/Lib/pycache. This means that to complete the build, $(PYTHON_FOR_BUILD) should probably include the -B flag to prevent it from trying to write *.pyc files to a read-only location.

    But that still leaves you with a python which only works with the -B flag. Maybe library files could be linked/copied to some writable directory early in sys.path.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 31, 2012

    Am 31.08.12 16:40, schrieb Richard Oudkerk:

    I suspect a bigger issue is the fact that *.pyc files cannot be
    written to $(srcdir)/Lib/pycache. This means that to complete
    the build, $(PYTHON_FOR_BUILD) should probably include the -B flag to
    prevent it from trying to write *.pyc files to a read-only location.

    Why is that? python will work just fine if it can't write pyc files.

    But that still leaves you with a python which only works with the -B
    flag. Maybe library files could be linked/copied to some writable
    directory early in sys.path.

    Just for running setup.py? I don't see the need for that.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Aug 31, 2012

    Why is that? python will work just fine if it can't write pyc files.

    When I hacked around the _sysconfigdata.py issue, running the created python produced

    Fatal Python error: Py_Initialize: Unable to get the locale encoding
    Traceback (most recent call last):
      File "/mnt/hgfs/Repos/cpython-dirty/Lib/encodings/__init__.py", line 31, in <module>
      File "<frozen importlib._bootstrap>", line 1562, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1529, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 590, in _check_name_wrapper
      File "<frozen importlib._bootstrap>", line 1027, in load_module
      File "<frozen importlib._bootstrap>", line 1008, in load_module
      File "<frozen importlib._bootstrap>", line 566, in module_for_loader_wrapper
      File "<frozen importlib._bootstrap>", line 858, in _load_module
      File "<frozen importlib._bootstrap>", line 994, in get_code
      File "<frozen importlib._bootstrap>", line 1055, in _cache_bytecode
      File "<frozen importlib._bootstrap>", line 1078, in set_data
      File "<frozen importlib._bootstrap>", line 128, in _write_atomic
    OSError: [Errno 5] Input/output error: '/mnt/hgfs/Repos/cpython-dirty/Lib/__pycache__/codecs.cpython-33.pyc.3076609376'
    Aborted

    I took this to be a failure to write the bytecompiled file. (A regression?) Running "./python -B" does work correctly for me.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Aug 31, 2012

    It looks like the code which writes the byte-compiled codes checks for and ignores PermissionError (EACCES, EPERM) and FileExistsError (EEXIST).

    However, for me it is EIO which is raised.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 31, 2012

    Am 31.08.12 17:32, schrieb Richard Oudkerk:

    File "<frozen importlib._bootstrap>", line 128, in _write_atomic
    OSError: [Errno 5] Input/output error: '/mnt/hgfs/Repos/cpython-dirty/Lib/pycache/codecs.cpython-33.pyc.3076609376'
    Aborted

    I took this to be a failure to write the bytecompiled file. (A regression?)

    Sounds like a regression to me. However, errno 5 is EIO.
    Normally, you should get PermissionError(errno=13, EACCES).
    I could understand EROFS as well. This sounds like a bug in
    the VM software, which really should report EROFS if the volume is
    read-only, not EIO.

    You might want to check whether this really is a regression;
    in any case, I consider Python's reaction reasonable and then
    your setup as just not supported. However, I could also accept
    that a work-around is added for treating EIO also as an ignorable
    error. In any case, this is a separate issue.

    @ericsnowcurrently
    Copy link
    Member

    Doesn't EIO cover pathologies which we don't want to silence? I could see adding EROFS though.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented Aug 31, 2012

    Am 31.08.12 18:18, schrieb Eric Snow:

    Doesn't EIO cover pathologies which we don't want to silence? I could see adding EROFS though.

    [This should *really* be its own issue]
    Sure, EIO indicates normally something pathological, such as a broken
    cable to the disk. I'm not sure it needs to be reported, though, at
    least not when coming out of open(): we are trying to write to the
    cache, and that failed really bad. However, this is not bad for Python:
    if we don't have a cache file, we can still continue executing.

    If we get EIO in the middle of writing, this would be worse.
    So as a work-around, I could propose to map OSError(EIO) coming
    out of open into PermissionError(EIO) (inside _bootstrap).
    But as I said: I'd also be fine with declaring this VM software
    unsupported. If this work-around is added, we should record the
    exact software for which it is a work-around, so we can remove
    it in the future.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Aug 31, 2012

    I have opened a new issue:

    http://bugs.python.org/issue15833
    

    It is a regression because in Python 3.2 all failures to open the file for writing were supressed.

    But as I said: I'd also be fine with declaring this VM software
    unsupported. If this work-around is added, we should record the
    exact software for which it is a work-around, so we can remove
    it in the future.

    It is VMware Player 4.0.0.

    @tpn
    Copy link
    Member Author

    tpn commented Sep 1, 2012

    Since the committed patch is already a step in that direction,
    I think it now needs to be completed. It's mandatory (IMO)
    that the generated files are actually used; that the generators
    are not run if not needed is optional and can be fixed later.

    Martin: I'll work on another patch over the next few days and post it here for review. It seems like your comments regarding our options are still valid (despite this thread getting sidetracked): continue with the premise that all generated files get generated in the build directory, and make sure that build directory gets used during the build (i.e. -I./Include).

    I'll start with that and see how I get on.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 16, 2012

    New changeset 617591e7d708 by Trent Nelson in branch '3.2':
    Issue bpo-15819: additional fixes for out-of-tree builds from read-only src.
    http://hg.python.org/cpython/rev/617591e7d708

    New changeset abb00e23681a by Trent Nelson in branch '3.3':
    Merge issue bpo-15819: additional fixes for out-of-tree builds from read-only src.
    http://hg.python.org/cpython/rev/abb00e23681a

    New changeset 3420fbd87646 by Trent Nelson in branch 'default':
    Merge issue bpo-15819: additional fixes for out-of-tree builds from read-only src.
    http://hg.python.org/cpython/rev/3420fbd87646

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 16, 2012

    New changeset e0a2b14a3cf9 by Trent Nelson in branch '2.7':
    Issue bpo-15819: additional fixes for out-of-tree builds from read-only src.
    http://hg.python.org/cpython/rev/e0a2b14a3cf9

    @tpn
    Copy link
    Member Author

    tpn commented Oct 16, 2012

    Things should be looking a lot better now across the board. I just committed fixes where appropriate to 2.7, 3.2, 3.3 and 3.x for this issue and two related issues:

    http://bugs.python.org/issue15298
    http://bugs.python.org/issue15833

    The only thing that stands out now for 3.[23x] is $(srcdir)/Lib/$(PLATDIR). Do we need a patch along the following lines?

    % hg diff
    diff -r 617591e7d708 Makefile.pre.in
    --- a/Makefile.pre.in   Tue Oct 16 08:41:32 2012 -0400
    +++ b/Makefile.pre.in   Tue Oct 16 09:46:53 2012 -0400
    @@ -947,7 +947,7 @@
                    multiprocessing multiprocessing/dummy \
                    unittest unittest/test \
                    curses pydoc_data $(MACHDEPS)
    -libinstall:    build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
    +libinstall:    build_all Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
            @for i in $(SCRIPTDIR) $(LIBDEST); \
            do \
                    if test ! -d $(DESTDIR)$$i; then \
    @@ -1031,14 +1031,14 @@
                    ./$(BUILDPYTHON) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt
     
     # Create the PLATDIR source directory, if one wasn't distributed..
    -$(srcdir)/Lib/$(PLATDIR):
    -       mkdir $(srcdir)/Lib/$(PLATDIR)
    -       cp $(srcdir)/Lib/plat-generic/regen $(srcdir)/Lib/$(PLATDIR)/regen
    +Lib/$(PLATDIR):
    +       mkdir Lib/$(PLATDIR)
    +       cp $(srcdir)/Lib/plat-generic/regen Lib/$(PLATDIR)/regen
            export PATH; PATH="`pwd`:$$PATH"; \
            export PYTHONPATH; PYTHONPATH="`pwd`/Lib"; \
            export DYLD_FRAMEWORK_PATH; DYLD_FRAMEWORK_PATH="`pwd`"; \
            export EXE; EXE="$(BUILDEXE)"; \
    -       cd $(srcdir)/Lib/$(PLATDIR); $(RUNSHARED) ./regen
    +       cd Lib/$(PLATDIR); $(RUNSHARED) ./regen
     
     python-config: $(srcdir)/Misc/python-config.in
            # Substitution happens here, as the completely-expanded BINDIR

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 16, 2012

    New changeset c35db4960d1c by Trent Nelson in branch '2.7':
    Issue bpo-15819: tweak logic in previous commit (e0a2b14a3cf9).
    http://hg.python.org/cpython/rev/c35db4960d1c

    New changeset 5707c8a36cc0 by Trent Nelson in branch '3.2':
    Issue bpo-15819: tweak logic in previous commit (617591e7d708).
    http://hg.python.org/cpython/rev/5707c8a36cc0

    New changeset 85863c4a93db by Trent Nelson in branch '3.3':
    Merge issue bpo-15819: tweak logic in previous commit.
    http://hg.python.org/cpython/rev/85863c4a93db

    New changeset d282018bff10 by Trent Nelson in branch 'default':
    Merge issue bpo-15819: tweak logic in previous commit.
    http://hg.python.org/cpython/rev/d282018bff10

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Oct 17, 2012

    New changeset 7e879a453bb3 by Trent Nelson in branch '2.7':
    Issue bpo-15819: use standard autoconf preset output variables.
    http://hg.python.org/cpython/rev/7e879a453bb3

    New changeset 65b3c41052b6 by Trent Nelson in branch '3.2':
    Issue bpo-15819: use standard autoconf preset output variables.
    http://hg.python.org/cpython/rev/65b3c41052b6

    New changeset b78e97b84f35 by Trent Nelson in branch '3.3':
    Issue bpo-15819: use standard autoconf preset output variables.
    http://hg.python.org/cpython/rev/b78e97b84f35

    New changeset f9eb516bea88 by Trent Nelson in branch 'default':
    Issue bpo-15819: use standard autoconf preset output variables.
    http://hg.python.org/cpython/rev/f9eb516bea88

    @doko42
    Copy link
    Member

    doko42 commented Jan 27, 2013

    just saw the comment about .hgtouch in http://bugs.python.org/issue15819#msg169484

    attaching a patch for it here (as proposed on python-committers).

    @rpetrov
    Copy link
    Mannequin

    rpetrov mannequin commented Feb 2, 2013

    To update .hgtouch is not enough.
    Grammar, AST and importlib should left in source tree as before.

    @koobs
    Copy link

    koobs commented Nov 19, 2013

    I'm not sure how I missed this, but our FreeBSD ports that build out-of-tree required adding a local patchthat reverts the Makefile.pre.in change in ab6ab44921b2 when we updated each port to the latest releases of 2.7, 3.2 and 3.3:

    https://svnweb.freebsd.org/ports?view=revision&revision=318353

    Without it, builds fail with:

    cc -c -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./../Include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c
    cc: Python/Python-ast.c: No such file or directory
    cc: No input files specified
    *** [Python/Python-ast.o] Error code 1

    Stop in /freebsd/ports/lang/python27/work/Python-2.7.6/portbld.shared.

    I also note Trents msg173160 in issue bpo-15298, which may be related?, but seems only resolved for OSX.

    What needs to be done to remove the need for this patch?

    @vadmium
    Copy link
    Member

    vadmium commented Sep 11, 2016

    Perhaps there should be a comment explaining that you can’t write to $(srcdir). Or even a buildbot that does an out-of-tree build. Every now and then I seem to be defending against patches that want to change targets back to $(srcdir).

    Also, Trent’s last change, revision 65b3c41052b6, does not seem to work. Apparently $abs_srcdir etc are only valid when substituted in a makefile, and are not valid in the configure script: <https://lists.gnu.org/archive/html/autoconf/2010-07/msg00021.html\>. I will try to post a patch that addresses this to bpo-28066.

    @koobs
    Copy link

    koobs commented Sep 11, 2016

    @martin Maybe Zach (nosey'd) can help us get an out-of-tree builder set up?

    If this is relevant for 3.5+, versions should be updated accordingly.

    @vadmium
    Copy link
    Member

    vadmium commented Sep 12, 2016

    I’m not sure what the overall status of this bug is, so I will leave the versions as they are. Are the three comments from 2013 relevant, and is there anything I can do to help?

    The Python 2 code already unconditionally adds -IInclude (see bpo-786737), and does not use BASECPPFLAGS. I think it would better to remove Trent’s code, as in unused-flags.patch.

    Python 3 also has this unconditional -IInclude, so maybe we can eliminate one of them in Python 3?

    gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -IObjects -IInclude -IPython -I. -IInclude -I../Include -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c

    @vadmium
    Copy link
    Member

    vadmium commented Sep 12, 2016

    Matthias seems to have already applied his patch in Python 3.3 and 3.4+: revision f2cc3d8b88bb.

    Roumen: Is your problem still relevant? If so, perhaps open a separate bug to elaborate.

    Koobs: Is your problem with finding Python/Python-ast.c still relevant? Is the $(ASDLGEN) makefile rule being run? I tried building both Python 2 and 3 outside the source tree, with Gnu and BSD Make. In all cases, if I hack the timestamps to force $(ASDLGEN) to run and remove the original source Python/Python-ast.c, it is correctly generated and compiled in the build directory. If I hack the timestamps the other way so that $(ASDLGEN) is not run, the file from the source directory is compiled into the build directory. So everything seems to be working reasonably well.

    @koobs
    Copy link

    koobs commented Sep 12, 2016

    @martin tldr; That was three years ago so I don't know if the issue remains. Perhaps a new out of tree builder will help answer that question.

    We(FreeBSD) gave up^W switched away from out of tree builds due to lack of support getting issues fixed and the frequency at which commits would alternately break in tree builds then out of tree builds.

    Unfortunately that also lost us the flexibility to create dual static/shared builds.

    The relevant downstream commit log: https://svnweb.freebsd.org/ports?view=revision&revision=350610

    The patch we added (reverting 2dde5a7439fd) that was mentioned in msg203348 was:

    https://svnweb.freebsd.org/ports/head/lang/python27/files/patch-Makefile.pre.in?view=markup&pathrev=318353

    If the current code is the same as per the 2dde5a7439fd change, then I'd guess it would still be an issue.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 13, 2016

    New changeset 9cabcb4411ac by Martin Panter in branch '2.7':
    Issue bpo-15819: Drop dead code; -IInclude already added unconditionally
    https://hg.python.org/cpython/rev/9cabcb4411ac

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 13, 2016

    New changeset 36550e4f9b4c by Martin Panter in branch '3.5':
    Issue bpo-15819: Remove old unconditional -IInclude option
    https://hg.python.org/cpython/rev/36550e4f9b4c

    New changeset a90daae58323 by Martin Panter in branch '3.6':
    Issue bpo-15819: Merge include search from 3.5 into 3.6
    https://hg.python.org/cpython/rev/a90daae58323

    New changeset 2ac00ff072b8 by Martin Panter in branch 'default':
    Issue bpo-15819: Merge include search from 3.6
    https://hg.python.org/cpython/rev/2ac00ff072b8

    @vadmium
    Copy link
    Member

    vadmium commented Sep 16, 2016

    Just to clarify, the current status is that the revision that Koobs identified is still in place, meaning that the AST files are regenerated into the build tree, not the source tree.

    I don’t think there is much else to do for this bug, unless Koobs can provide more info why the Free BSD build could not find Python/Python-ast.c, or Roumen can get back with more info. Closing for the moment, but I am happy to help if I can resolve any problems that remain.

    @vadmium vadmium closed this as completed Sep 16, 2016
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants