classification
Title: Unable to build Python out-of-tree when source tree is readonly.
Type: compile error Stage: resolved
Components: Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: 28066 Superseder:
Assigned To: trent Nosy List: Arfrever, doko, eric.snow, gregory.p.smith, haypo, koobs, loewis, martin.panter, ned.deily, pitrou, python-dev, rpetrov, sbt, trent, zach.ware
Priority: low Keywords: patch

Created on 2012-08-30 06:49 by trent, last changed 2016-09-16 02:48 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
readonly_src.patch trent, 2012-08-30 06:49 review
hgtouch.diff doko, 2013-01-27 20:27 review
unused-flags.patch martin.panter, 2016-09-12 01:49 Unneeded code in Py 2 review
Messages (44)
msg169430 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-08-30 06:49
This is somewhat related to Issue #9860.  You can't currently build Python out-of-tree when the source directory is readonly.  This is because there are a few Parser/pgen/AST/python_asdl.py steps that try and write to $(srcdir).

The attached patch fixes this.
msg169431 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-30 06:59
Instead of using "mkdir -p", it should use $(MKDIR_P) (which actually isn't normally "mkdir -p" :-) Other than that, the patch looks fine.
msg169432 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-08-30 07:19
Hmmmmm.  Now I'm confused.  Is this what you meant?

-------------------
% hg diff
diff -r 019a2390b014 Makefile.pre.in
--- a/Makefile.pre.in   Tue Aug 21 23:41:43 2012 +0000
+++ b/Makefile.pre.in   Thu Aug 30 07:16:55 2012 +0000
@@ -43,6 +43,8 @@
 
 GNULD=         @GNULD@
 
+MKDIR_P=       @MKDIR_P@
+
 # Shell used by make (some versions default to the login shell, which is bad)
 SHELL=         /bin/sh
 
@@ -223,8 +225,8 @@
 
 ##########################################################################
 # Grammar
-GRAMMAR_H=     $(srcdir)/Include/graminit.h
-GRAMMAR_C=     $(srcdir)/Python/graminit.c
+GRAMMAR_H=     Include/graminit.h
+GRAMMAR_C=     Python/graminit.c
 GRAMMAR_INPUT= $(srcdir)/Grammar/Grammar
 
 
@@ -266,9 +268,9 @@
 
 ##########################################################################
 # AST
-AST_H_DIR=     $(srcdir)/Include
+AST_H_DIR=     Include
 AST_H=         $(AST_H_DIR)/Python-ast.h
-AST_C_DIR=     $(srcdir)/Python
+AST_C_DIR=     Python
 AST_C=         $(AST_C_DIR)/Python-ast.c
 AST_ASDL=      $(srcdir)/Parser/Python.asdl
 
@@ -605,9 +607,11 @@
 Parser/pgenmain.o:     $(srcdir)/Include/parsetok.h
 
 $(AST_H): $(AST_ASDL) $(ASDLGEN_FILES)
+       $(MKDIR_P) $(AST_H_DIR)
        $(ASDLGEN) -h $(AST_H_DIR) $(AST_ASDL)
 
 $(AST_C): $(AST_ASDL) $(ASDLGEN_FILES)
+       $(MKDIR_P) $(AST_C_DIR)
        $(ASDLGEN) -c $(AST_C_DIR) $(AST_ASDL)
 
 Python/compile.o Python/symtable.o Python/ast.o: $(GRAMMAR_H) $(AST_H)
diff -r 019a2390b014 configure
--- a/configure Tue Aug 21 23:41:43 2012 +0000
+++ b/configure Thu Aug 30 07:16:55 2012 +0000
@@ -658,6 +658,7 @@
 GNULD
 LINKCC
 LDVERSION
+MKDIR_P
 RUNSHARED
 INSTSONAME
 LDLIBRARYDIR
@@ -5262,6 +5263,7 @@
     HGBRANCH=""
 fi
 
+MKDIR_P="mkdir -p"
 
 DISABLE_ASDLGEN=""
 # Extract the first word of "python", so it can be a program name with args.
-------------------

Given your "which actually isn't normally mkdir -p" comment... my patch doesn't seem right.
msg169433 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-30 07:28
Am 30.08.12 09:19, schrieb Trent Nelson:
> Hmmmmm.  Now I'm confused.  Is this what you meant?

No. Makefile.pre.in *already* has a definition of MKDIR_P, no need to 
add one.
msg169434 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-08-30 07:30
That's what I figured you meant initially... Until I couldn't find any MKDIR_P definitions, and couldn't get it working without the hack above.

I thought it might be an implicit make variable as well, but, not so much.  (Tested with BSD make and gmake on FreeBSD.)
msg169435 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2012-08-30 07:34
In 3.3, Makefile.pre.in already defines MKDIR_P, in 3.2 it doesn't.
msg169437 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-08-30 07:47
Ah, I see the MKDIR_P changes in 3.x/configure as well.

Objections to backporting to 3.2?

Is 2.7 off the table?

(Building from a readonly source is handy for Snakebite; saves having a dozen machines needing to use their own hg clone.)
msg169444 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-30 10:00
Am 30.08.12 09:47, schrieb Trent Nelson:
> Objections to backporting to 3.2?
>
> Is 2.7 off the table?

No, adding AC_PROG_MKDIR_P into all of them is fine.
msg169458 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-30 14:32
New changeset 62044cd5b19b by Trent Nelson in branch '3.2':
Issue #15819: Fix out-of-tree builds from a readonly source.
http://hg.python.org/cpython/rev/62044cd5b19b
msg169461 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-30 14:36
New changeset af36536988fd by Trent Nelson in branch 'default':
Block 78809:62044cd5b19b (Issue #15819) from 3.2.
http://hg.python.org/cpython/rev/af36536988fd
msg169462 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-30 14:52
New changeset ab6ab44921b2 by Trent Nelson in branch 'default':
Issue #15819: Fix out-of-tree builds from a readonly source.
http://hg.python.org/cpython/rev/ab6ab44921b2
msg169463 - (view) Author: Roundup Robot (python-dev) Date: 2012-08-30 14:56
New changeset 2dde5a7439fd by Trent Nelson in branch '2.7':
Issue #15819: Fix out-of-tree builds from a readonly source.
http://hg.python.org/cpython/rev/2dde5a7439fd
msg169480 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-08-30 18:51
I tried building from out-of-tree with a read-only srcdir, but I found two other files which the build process tries to create in the source directory:

    $(srcdir)/Objects/typeslots.inc
    $(srcdir)/Lib/_sysconfigdata.py
msg169484 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-30 19:09
What I liked about Trent's original bug report is that it didn't just bring up a generic issue (cannot build with read-only source code), but clarified that his concern really was graminit.

So I'd like to declare that this (graminit) is the focus of this issue: any other aspects of read-only source trees are out of scope, and should be reported separately.

That said:

As for typeslots.inc: this shouldn't really be an issue for a released version of Python, since its tarball should include the generated file, and the timestamps should be in an order so that generation is unnecessary. This should be the case for all generated files that are also checked into Mercurial. For checkouts from Mercurial, you should do "make touch" after an update, as this will bring the timestamps in the right order.

It seems that graminit is missing from .hgtouch (in which case Trent's patch probably wouldn't have been necessary, except that it also fixes older branches). But that is again a different issue.

As for sysconfigdata: this really should be a separate issue. It's more difficult, since installation and packaging needs to pick up the file even if it's not in the source tree.
msg169486 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-08-30 19:13
The sysconfigdata problem is already documented in two open issues: Issue14774 and Issue15298.
msg169489 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-30 19:50
As a further follow-up: the original code (in 67ed8a6905c3, by Neil Schemenauer from 2001), he explicitly changed GRAMMAR_C/H to refer to srcdir, claiming that errors from building them would be ignored.

In r87558, for which I can't find a subversion revsision for, Victor Stinner dropped the Makefile "-" marker, making the read-only case fail.

In reviewing the patch more, I think it is actually incorrect: it now generates the files into Include, but they aren't considered there since (builddir)/Include is not added to the -I options.

So the two options now are:
a) revert the patch, and restore Neil's original approach, or
b) add Include to the -I options, if srcdir is not "."

Reopening for consideration.
msg169490 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-30 20:02
Reconsidering more: it would be desirable if even an out-of-source-tree compilation wouldn't require Python for bootstrapping; with the committed patch, it does since it will always run ASDLGEN.

If the source version of the generated files is fresh enough, they should just be copied to the destination.

I'm not quite sure how to achieve that with Make rules. This probably *is* a separate issue (which would become irrelevant if the patch here is reverted).
msg169534 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-08-31 13:19
Well, that escalated quickly :-)

I think I should clarify my use case that resulted in raising this bug.

1. Create a fresh hg clone, say, ~/hg/cpython-3.2.
2. Make it readonly: `zfs set readonly=on tank/home/cpython/hg/cpython-3.2`
3. cd /tmp/cpython-3.2-build
4. /home/cpython/hg/cpython-3.2/configure --srcdir=/home/cpython/hg/cpython-3.2/configure

So, my patches address that use case.  The thing I like about the approach above is that I can set the source directory to readonly immediately -- I don't have to do an intermediate ./configure in that directory in order to generate a Makefile so I can `make touch`.  (Side note: I'm almost certain `make touch` wouldn't work for my use case.)

Richard: what was your use case?  What steps did you take that resulted in getting typeslots.inc and _sysconfigdata.py written to?

....although I just tried an out-of-tree build with 'default' using the exact steps above and got the same error with _sysconfigdata.py.

Martin: I think I understand your arguments, and I (think) I agree with them.  Especially your last point with regards to always running ASDLGEN.

I think I'm +1 on reverting and trying the '-' approach originally used by Neil.
msg169535 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-31 13:56
Am 31.08.12 15:19, schrieb Trent Nelson:
> I think I should clarify my use case that resulted in raising this
> bug.
>
> 1. Create a fresh hg clone, say, ~/hg/cpython-3.2.
> 2. Make it readonly

The use case is really uncontended, and long-standing. People want
to build from CD-ROM sources, from sources mounted via NFS,
from sources owned by some other user - there are really many
reasons why you want to build in a separate directory, and why
you then do not want the original source be clobbered.

> Richard: what was your use case?

I guess pretty much the same (though perhaps not ZFS).

> What steps did you take that
> resulted in getting typeslots.inc and _sysconfigdata.py written to?

Touch Include/typeslots.h (before making the checkout readonly)
to see this problem. On a hg checkout, it really depends on the order
in which Mercurial creates the files to either run or not run into one
of these issues.

You were lucky that the timestamps happened to be in an order were
in an order that caused typeslots.inc to be newer than its sources.
AFAIK, hg uses alphapbetical order when updating files, so this may
have helped.

> I think I'm +1 on reverting and trying the '-' approach originally
> used by Neil.

Please don't. I think Victor dropped the - because it would mask
errors from running pgen. pgen might fail, but pgen.stamp is still
generated. Then, on a subsequent run, pgen would not run again.
Of course, pgen.stamp is a later addition (by me), so this wasn't
originally a problem.

I think the regular case should be that the time stamps are in
the right order in the source tree, so none of the generators
should run. But if they happen to be in the wrong order (either
because of Mercurial, or because one of the sources was actually
modified), then the generators should run and output into the
build directory.

Since the committed patch is already a step in that direction,
I think it now needs to be completed. It's mandatory (IMO)
that the generated files are actually used; that the generators
are not run if not needed is optional and can be fixed later.
msg169541 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-08-31 14:40
> Richard: what was your use case?

Building on the main host and a VM without having to commit and synchronize temporary changes.  (The VM has read-only access to the host's repositories).

> What steps did you take that resulted in getting typeslots.inc and 
> _sysconfigdata.py written to?

As Martin says it is probably just the timestamps which caused typeslots.inc to be rebuilt.  In 3.3 _sysconfigdata.py is rebuilt whenever the python binary is.

I suspect a bigger issue is the fact that *.pyc files cannot be written to $(srcdir)/Lib/__pycache__.  This means that to complete the build, $(PYTHON_FOR_BUILD) should probably include the -B flag to prevent it from trying to write *.pyc files to a read-only location.

But that still leaves you with a python which only works with the -B flag. Maybe library files could be linked/copied to some writable directory early in sys.path.
msg169542 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-31 14:57
Am 31.08.12 16:40, schrieb Richard Oudkerk:
> I suspect a bigger issue is the fact that *.pyc files cannot be
> written to $(srcdir)/Lib/__pycache__.  This means that to complete
> the build, $(PYTHON_FOR_BUILD) should probably include the -B flag to
> prevent it from trying to write *.pyc files to a read-only location.

Why is that? python will work just fine if it can't write pyc files.

> But that still leaves you with a python which only works with the -B
> flag. Maybe library files could be linked/copied to some writable
> directory early in sys.path.

Just for running setup.py? I don't see the need for that.
msg169544 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-08-31 15:32
> Why is that? python will work just fine if it can't write pyc files.

When I hacked around the _sysconfigdata.py issue, running the created python produced

Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
  File "/mnt/hgfs/Repos/cpython-dirty/Lib/encodings/__init__.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 1562, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1529, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 590, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1027, in load_module
  File "<frozen importlib._bootstrap>", line 1008, in load_module
  File "<frozen importlib._bootstrap>", line 566, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 858, in _load_module
  File "<frozen importlib._bootstrap>", line 994, in get_code
  File "<frozen importlib._bootstrap>", line 1055, in _cache_bytecode
  File "<frozen importlib._bootstrap>", line 1078, in set_data
  File "<frozen importlib._bootstrap>", line 128, in _write_atomic
OSError: [Errno 5] Input/output error: '/mnt/hgfs/Repos/cpython-dirty/Lib/__pycache__/codecs.cpython-33.pyc.3076609376'
Aborted

I took this to be a failure to write the bytecompiled file.  (A regression?)  Running "./python -B" does work correctly for me.
msg169545 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-08-31 15:40
It looks like the code which writes the byte-compiled codes checks for and ignores PermissionError (EACCES, EPERM) and FileExistsError (EEXIST).

However, for me it is EIO which is raised.
msg169546 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-31 15:49
Am 31.08.12 17:32, schrieb Richard Oudkerk:
>    File "<frozen importlib._bootstrap>", line 128, in _write_atomic
> OSError: [Errno 5] Input/output error: '/mnt/hgfs/Repos/cpython-dirty/Lib/__pycache__/codecs.cpython-33.pyc.3076609376'
> Aborted
>
> I took this to be a failure to write the bytecompiled file.  (A regression?)

Sounds like a regression to me. However, errno 5 is EIO.
Normally, you should get PermissionError(errno=13, EACCES).
I could understand EROFS as well. This sounds like a bug in
the VM software, which really should report EROFS if the volume is
read-only, not EIO.

You might want to check whether this really is a regression;
in any case, I consider Python's reaction reasonable and then
your setup as just not supported. However, I could also accept
that a work-around is added for treating EIO also as an ignorable
error. In any case, this is a separate issue.
msg169552 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-08-31 16:18
Doesn't EIO cover pathologies which we don't want to silence?  I could see adding EROFS though.
msg169558 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-08-31 16:45
Am 31.08.12 18:18, schrieb Eric Snow:
> Doesn't EIO cover pathologies which we don't want to silence?  I could see adding EROFS though.

[This should *really* be its own issue]
Sure, EIO indicates normally something pathological, such as a broken
cable to the disk. I'm not sure it needs to be reported, though, at
least not when coming out of open(): we are trying to write to the 
cache, and that failed really bad. However, this is not bad for Python:
if we don't have a cache file, we can still continue executing.

If we get EIO in the middle of writing, this would be worse.
So as a work-around, I could propose to map OSError(EIO) coming
out of open into PermissionError(EIO) (inside _bootstrap).
But as I said: I'd also be fine with declaring this VM software
unsupported. If this work-around is added, we should record the
exact software for which it is a work-around, so we can remove
it in the future.
msg169569 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-08-31 17:22
I have opened a new issue:

    http://bugs.python.org/issue15833

It is a regression because in Python 3.2 all failures to open the file for writing were supressed.

> But as I said: I'd also be fine with declaring this VM software
> unsupported. If this work-around is added, we should record the
> exact software for which it is a work-around, so we can remove
> it in the future.

It is VMware Player 4.0.0.
msg169595 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-09-01 00:03
> Since the committed patch is already a step in that direction,
> I think it now needs to be completed. It's mandatory (IMO)
> that the generated files are actually used; that the generators
> are not run if not needed is optional and can be fixed later.

Martin: I'll work on another patch over the next few days and post it here for review.  It seems like your comments regarding our options are still valid (despite this thread getting sidetracked): continue with the premise that all generated files get generated in the build directory, and make sure that build directory gets used during the build (i.e. -I./Include).

I'll start with that and see how I get on.
msg173038 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-16 12:57
New changeset 617591e7d708 by Trent Nelson in branch '3.2':
Issue #15819: additional fixes for out-of-tree builds from read-only src.
http://hg.python.org/cpython/rev/617591e7d708

New changeset abb00e23681a by Trent Nelson in branch '3.3':
Merge issue #15819: additional fixes for out-of-tree builds from read-only src.
http://hg.python.org/cpython/rev/abb00e23681a

New changeset 3420fbd87646 by Trent Nelson in branch 'default':
Merge issue #15819: additional fixes for out-of-tree builds from read-only src.
http://hg.python.org/cpython/rev/3420fbd87646
msg173041 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-16 13:43
New changeset e0a2b14a3cf9 by Trent Nelson in branch '2.7':
Issue #15819: additional fixes for out-of-tree builds from read-only src.
http://hg.python.org/cpython/rev/e0a2b14a3cf9
msg173042 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-10-16 13:52
Things should be looking a lot better now across the board.  I just committed fixes where appropriate to 2.7, 3.2, 3.3 and 3.x for this issue and two related issues: 

http://bugs.python.org/issue15298
http://bugs.python.org/issue15833

The only thing that stands out now for 3.[23x] is $(srcdir)/Lib/$(PLATDIR).  Do we need a patch along the following lines?

% hg diff
diff -r 617591e7d708 Makefile.pre.in
--- a/Makefile.pre.in   Tue Oct 16 08:41:32 2012 -0400
+++ b/Makefile.pre.in   Tue Oct 16 09:46:53 2012 -0400
@@ -947,7 +947,7 @@
                multiprocessing multiprocessing/dummy \
                unittest unittest/test \
                curses pydoc_data $(MACHDEPS)
-libinstall:    build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
+libinstall:    build_all Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
        @for i in $(SCRIPTDIR) $(LIBDEST); \
        do \
                if test ! -d $(DESTDIR)$$i; then \
@@ -1031,14 +1031,14 @@
                ./$(BUILDPYTHON) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt
 
 # Create the PLATDIR source directory, if one wasn't distributed..
-$(srcdir)/Lib/$(PLATDIR):
-       mkdir $(srcdir)/Lib/$(PLATDIR)
-       cp $(srcdir)/Lib/plat-generic/regen $(srcdir)/Lib/$(PLATDIR)/regen
+Lib/$(PLATDIR):
+       mkdir Lib/$(PLATDIR)
+       cp $(srcdir)/Lib/plat-generic/regen Lib/$(PLATDIR)/regen
        export PATH; PATH="`pwd`:$$PATH"; \
        export PYTHONPATH; PYTHONPATH="`pwd`/Lib"; \
        export DYLD_FRAMEWORK_PATH; DYLD_FRAMEWORK_PATH="`pwd`"; \
        export EXE; EXE="$(BUILDEXE)"; \
-       cd $(srcdir)/Lib/$(PLATDIR); $(RUNSHARED) ./regen
+       cd Lib/$(PLATDIR); $(RUNSHARED) ./regen
 
 python-config: $(srcdir)/Misc/python-config.in
        # Substitution happens here, as the completely-expanded BINDIR
msg173046 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-16 15:11
New changeset c35db4960d1c by Trent Nelson in branch '2.7':
Issue #15819: tweak logic in previous commit (e0a2b14a3cf9).
http://hg.python.org/cpython/rev/c35db4960d1c

New changeset 5707c8a36cc0 by Trent Nelson in branch '3.2':
Issue #15819: tweak logic in previous commit (617591e7d708).
http://hg.python.org/cpython/rev/5707c8a36cc0

New changeset 85863c4a93db by Trent Nelson in branch '3.3':
Merge issue #15819: tweak logic in previous commit.
http://hg.python.org/cpython/rev/85863c4a93db

New changeset d282018bff10 by Trent Nelson in branch 'default':
Merge issue #15819: tweak logic in previous commit.
http://hg.python.org/cpython/rev/d282018bff10
msg173144 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-17 08:49
New changeset 7e879a453bb3 by Trent Nelson in branch '2.7':
Issue #15819: use standard autoconf preset output variables.
http://hg.python.org/cpython/rev/7e879a453bb3

New changeset 65b3c41052b6 by Trent Nelson in branch '3.2':
Issue #15819: use standard autoconf preset output variables.
http://hg.python.org/cpython/rev/65b3c41052b6

New changeset b78e97b84f35 by Trent Nelson in branch '3.3':
Issue #15819: use standard autoconf preset output variables.
http://hg.python.org/cpython/rev/b78e97b84f35

New changeset f9eb516bea88 by Trent Nelson in branch 'default':
Issue #15819: use standard autoconf preset output variables.
http://hg.python.org/cpython/rev/f9eb516bea88
msg180788 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-01-27 20:27
just saw the comment about .hgtouch in http://bugs.python.org/issue15819#msg169484

attaching a patch for it here (as proposed on python-committers).
msg181177 - (view) Author: Roumen Petrov (rpetrov) * Date: 2013-02-02 16:33
To update .hgtouch is not enough.
Grammar, AST and importlib should left in source tree as before.
msg203348 - (view) Author: Kubilay Kocak (koobs) Date: 2013-11-19 10:08
I'm not sure how I missed this, but our FreeBSD ports that build out-of-tree required adding a local patchthat reverts the Makefile.pre.in change in ab6ab44921b2 when we updated each port to the latest releases of 2.7, 3.2 and 3.3:

https://svnweb.freebsd.org/ports?view=revision&revision=318353

Without it, builds fail with:

cc -c -fno-strict-aliasing -O2 -pipe  -fno-strict-aliasing -DNDEBUG  -I. -IInclude -I./../Include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c
cc: Python/Python-ast.c: No such file or directory
cc: No input files specified
*** [Python/Python-ast.o] Error code 1

Stop in /freebsd/ports/lang/python27/work/Python-2.7.6/portbld.shared.

I also note Trents msg173160 in issue #15298, which may be related?, but seems only resolved for OSX.

What needs to be done to remove the need for this patch?
msg275751 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-11 09:08
Perhaps there should be a comment explaining that you can’t write to $(srcdir). Or even a buildbot that does an out-of-tree build. Every now and then I seem to be defending against patches that want to change targets back to $(srcdir).

Also, Trent’s last change, revision 65b3c41052b6, does not seem to work. Apparently $abs_srcdir etc are only valid when substituted in a makefile, and are not valid in the configure script: <https://lists.gnu.org/archive/html/autoconf/2010-07/msg00021.html>. I will try to post a patch that addresses this to Issue 28066.
msg275768 - (view) Author: Kubilay Kocak (koobs) Date: 2016-09-11 10:32
@Martin Maybe Zach (nosey'd) can help us get an out-of-tree builder set up?

If this is relevant for 3.5+, versions should be updated accordingly.
msg275910 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-12 01:49
I’m not sure what the overall status of this bug is, so I will leave the versions as they are. Are the three comments from 2013 relevant, and is there anything I can do to help?

The Python 2 code already unconditionally adds -IInclude (see Issue 786737), and does not use BASECPPFLAGS. I think it would better to remove Trent’s code, as in unused-flags.patch.

Python 3 also has this unconditional -IInclude, so maybe we can eliminate one of them in Python 3?

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -Werror=declaration-after-statement  -IObjects -IInclude -IPython -I. -IInclude -I../Include    -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c
msg275948 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-12 04:24
Matthias seems to have already applied his patch in Python 3.3 and 3.4+: revision f2cc3d8b88bb.

Roumen: Is your problem still relevant? If so, perhaps open a separate bug to elaborate.

Koobs: Is your problem with finding Python/Python-ast.c still relevant? Is the $(ASDLGEN) makefile rule being run? I tried building both Python 2 and 3 outside the source tree, with Gnu and BSD Make. In all cases, if I hack the timestamps to force $(ASDLGEN) to run and remove the original source Python/Python-ast.c, it is correctly generated and compiled in the build directory. If I hack the timestamps the other way so that $(ASDLGEN) is not run, the file from the source directory is compiled into the build directory. So everything seems to be working reasonably well.
msg275995 - (view) Author: Kubilay Kocak (koobs) Date: 2016-09-12 08:32
@Martin tldr; That was three years ago so I don't know if the issue remains. Perhaps a new out of tree builder will help answer that question.

We(FreeBSD) gave up^W switched away from out of tree builds due to lack of support getting issues fixed and the frequency at which commits would alternately break in tree builds then out of tree builds.

Unfortunately that also lost us the flexibility to create dual static/shared builds.

The relevant downstream commit log: https://svnweb.freebsd.org/ports?view=revision&revision=350610

The patch we added (reverting 2dde5a7439fd) that was mentioned in msg203348 was: 

https://svnweb.freebsd.org/ports/head/lang/python27/files/patch-Makefile.pre.in?view=markup&pathrev=318353

If the current code is the same as per the 2dde5a7439fd change, then I'd guess it would still be an issue.
msg276260 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-13 12:26
New changeset 9cabcb4411ac by Martin Panter in branch '2.7':
Issue #15819: Drop dead code; -IInclude already added unconditionally
https://hg.python.org/cpython/rev/9cabcb4411ac
msg276265 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-13 13:29
New changeset 36550e4f9b4c by Martin Panter in branch '3.5':
Issue #15819: Remove old unconditional -IInclude option
https://hg.python.org/cpython/rev/36550e4f9b4c

New changeset a90daae58323 by Martin Panter in branch '3.6':
Issue #15819: Merge include search from 3.5 into 3.6
https://hg.python.org/cpython/rev/a90daae58323

New changeset 2ac00ff072b8 by Martin Panter in branch 'default':
Issue #15819: Merge include search from 3.6
https://hg.python.org/cpython/rev/2ac00ff072b8
msg276667 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-16 02:48
Just to clarify, the current status is that the revision that Koobs identified is still in place, meaning that the AST files are regenerated into the build tree, not the source tree.

I don’t think there is much else to do for this bug, unless Koobs can provide more info why the Free BSD build could not find Python/Python-ast.c, or Roumen can get back with more info. Closing for the moment, but I am happy to help if I can resolve any problems that remain.
History
Date User Action Args
2016-09-16 02:48:55martin.pantersetstatus: open -> closed
resolution: fixed
messages: + msg276667

stage: patch review -> resolved
2016-09-13 13:29:29python-devsetmessages: + msg276265
2016-09-13 12:26:16python-devsetmessages: + msg276260
2016-09-12 09:41:11petri.lehtinensetnosy: - petri.lehtinen
2016-09-12 08:32:25koobssetmessages: + msg275995
2016-09-12 04:24:06martin.pantersetmessages: + msg275948
2016-09-12 01:49:51martin.pantersetfiles: + unused-flags.patch

messages: + msg275910
stage: patch review
2016-09-11 10:32:40koobssetnosy: + zach.ware
messages: + msg275768
2016-09-11 09:08:20martin.pantersetnosy: + martin.panter
dependencies: + [Patch] Fix the ability to cross compile Python when doing a rebuild of importlib.h
messages: + msg275751
2014-03-04 08:19:25gregory.p.smithsetnosy: + gregory.p.smith
2013-11-19 10:08:05koobssetnosy: + koobs
messages: + msg203348
2013-05-22 20:29:47ned.deilylinkissue17649 superseder
2013-02-02 16:33:59rpetrovsetnosy: + rpetrov
messages: + msg181177
2013-01-27 20:27:51dokosetfiles: + hgtouch.diff
nosy: + doko
messages: + msg180788

2012-10-17 08:49:44python-devsetmessages: + msg173144
2012-10-16 15:11:25python-devsetmessages: + msg173046
2012-10-16 13:52:14trentsetnosy: + pitrou
messages: + msg173042
2012-10-16 13:43:19python-devsetmessages: + msg173041
2012-10-16 12:57:24python-devsetmessages: + msg173038
2012-10-14 08:44:09hayposetnosy: + haypo
2012-09-04 07:47:54Arfreversetnosy: + Arfrever
2012-09-01 00:03:14trentsetmessages: + msg169595
2012-08-31 17:22:51sbtsetmessages: + msg169569
2012-08-31 16:45:43loewissetmessages: + msg169558
2012-08-31 16:18:03eric.snowsetnosy: + eric.snow
messages: + msg169552
2012-08-31 15:49:06loewissetmessages: + msg169546
2012-08-31 15:40:05sbtsetmessages: + msg169545
2012-08-31 15:32:08sbtsetmessages: + msg169544
2012-08-31 14:57:22loewissetmessages: + msg169542
2012-08-31 14:40:37sbtsetmessages: + msg169541
2012-08-31 13:56:30loewissetmessages: + msg169535
2012-08-31 13:19:09trentsetmessages: + msg169534
2012-08-31 10:27:29asvetlovsetstage: resolved -> (no value)
2012-08-30 20:02:51loewissetmessages: + msg169490
2012-08-30 19:50:49loewissetstatus: closed -> open

messages: + msg169489
2012-08-30 19:13:47ned.deilysetnosy: + ned.deily
messages: + msg169486
2012-08-30 19:09:33loewissetmessages: + msg169484
2012-08-30 18:51:09sbtsetnosy: + sbt
messages: + msg169480
2012-08-30 14:57:20trentsetstatus: open -> closed
stage: patch review -> resolved
2012-08-30 14:56:23python-devsetmessages: + msg169463
2012-08-30 14:52:53python-devsetmessages: + msg169462
2012-08-30 14:36:37python-devsetmessages: + msg169461
2012-08-30 14:32:13python-devsetnosy: + python-dev
messages: + msg169458
2012-08-30 10:00:06loewissetmessages: + msg169444
2012-08-30 07:47:23trentsetmessages: + msg169437
2012-08-30 07:34:06petri.lehtinensetnosy: + petri.lehtinen
messages: + msg169435
2012-08-30 07:30:09trentsetmessages: + msg169434
2012-08-30 07:28:15loewissetmessages: + msg169433
title: Unable to build Python out-of-tree when source tree is readonly. -> Unable to build Python out-of-tree when source tree is readonly.
2012-08-30 07:19:41trentsetmessages: + msg169432
2012-08-30 06:59:27loewissetnosy: + loewis
messages: + msg169431
2012-08-30 06:49:31trentsetstage: needs patch -> patch review
2012-08-30 06:49:21trentcreate