classification
Title: Building python outside of source directory fails
Type: compile error Stage: resolved
Components: Interpreter Core Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: belopolsky, dmalcolm, doko, eric.araujo, ned.deily, pitrou, python-dev, rpetrov, trent
Priority: low Keywords:

Created on 2010-09-15 14:08 by belopolsky, last changed 2014-06-29 20:31 by belopolsky. This issue is now closed.

Messages (16)
msg116448 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-15 14:08
When testing config options, it is often useful to build python in a directory different from the root of the source tree.  This is supported by autoconf based builds as follows: you cd to the desired directory run configure with an explicit path. For example:

$ cd ../altdir
$ ../python-source/configure
$ make

However, this fails for python 3.1 and 3.2 with the following error:


i686-apple-darwin10-gcc-4.2.1: Parser/tokenizer_pgen.o: No such file or directory
i686-apple-darwin10-gcc-4.2.1: Parser/printgrammar.o: No such file or directory
i686-apple-darwin10-gcc-4.2.1: Parser/pgenmain.o: No such file or directory
make: *** [Parser/pgen] Error 1
msg116451 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-15 14:45
I think the problem is with the following explicit rules in Makefile:


Parser/tokenizer_pgen.o:        $(srcdir)/Parser/tokenizer.c

Parser/pgenmain.o:      $(srcdir)/Include/parsetok.h

It looks like these rules are not VPATH friendly.  Per Antoine's suggestions, adding build experts to the nosy list.
msg116456 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-15 15:18
It works here (Linux). Did you try make distclean first? Also, try to remove the various files created by configure.

(what is VPATH?)
msg116458 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-15 15:24
It turns out that make only gets confused when target files (Parser/pgenmain.o etc.) exist in the source directory.  Thus an obvious workaround is to build all versions in alternative directories or delete object files from source tree before alternative builds.

This is still fragile and does not explain why for some targets, VPATH candidates are ignored and for some are not.
msg116459 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-15 15:30
> (what is VPATH?)

VPATH is a search path that make uses to find targets and prerequisites.  configure sets it to its own location when invoked from a directory other than cwd.
msg116490 - (view) Author: Roumen Petrov (rpetrov) * Date: 2010-09-15 21:45
Please could you remove object files from source tree then try again.

This is invalid issue.
msg116493 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-15 21:56
On Wed, Sep 15, 2010 at 5:45 PM, Roumen Petrov <report@bugs.python.org> wrote:
..
> Please could you remove object files from source tree then try again.
>
It looks like you missed my penultimate comment.

> This is invalid issue.

At the very least, Building Python [1] section can be expanded to
cover building outside of the source tree.  Some developers are not
even aware that this is supported.

[1] http://docs.python.org/using/unix.html#building-python
msg116972 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-09-20 21:09
Does this apply to 2.7 too?
msg117061 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-09-21 15:07
> Does this apply to 2.7 too?

I think it does, but it would be hard to sell any improvements in this area as a bug fix.
msg117096 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2010-09-21 19:16
FYIW, the OS X installer build script (Mac/BuildScript/build-installer.py for years has, and continues to, depend on building outside of the source directory.  When I build one, I always start with a clean source directory.  There are so many opportunities for the configure and make stages to go wrong that it seems not worth the effort to try to plug all of the holes of the type you have identified here when attempting to build with a "dirty" source directory.  I wouldn't trust using one.
msg126016 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2011-01-11 15:54
Here is a related bug:

$ make patchcheck
./python.exe ../py3k-commit/Tools/scripts/patchcheck.py
Getting the list of files that have been added/changed ... need a checkout to get modified files
[49399 refs]
make: *** [patchcheck] Error 1
msg139895 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-05 16:20
I am working on a patch to make patchcheck use os.path.join(sysconfig.get_config_var('srcdir'), etc.) to look for the .hg dir and open files (to do its checks) with the right paths.
msg141496 - (view) Author: Roundup Robot (python-dev) Date: 2011-08-01 12:45
New changeset 313a71664781 by Éric Araujo in branch '3.2':
Let “make patchcheck” work for out-of-dir builds (#9860)
http://hg.python.org/cpython/rev/313a71664781

New changeset 5993f91598ce by Éric Araujo in branch 'default':
Merge fixes for #9860, #11104/#8688 and #12331 from 3.2
http://hg.python.org/cpython/rev/5993f91598ce
msg141500 - (view) Author: Roundup Robot (python-dev) Date: 2011-08-01 12:49
New changeset bea11ce24bb0 by Éric Araujo in branch '2.7':
Let “make patchcheck” work for out-of-dir builds (#9860)
http://hg.python.org/cpython/rev/bea11ce24bb0
msg169465 - (view) Author: Trent Nelson (trent) * (Python committer) Date: 2012-08-30 14:59
FWIW, `make patchcheck` fails for me as follows:


% make patchcheck
./python /home/trent/hg/cpython-3.2/Tools/scripts/patchcheck.py
Getting the list of files that have been added/changed ... abort: no repository found in '/tmp/cpython-3.2-build' (.hg not found)!
0 files
Fixing whitespace ... 0 files
Fixing C file whitespace ... 0 files
Fixing docs whitespace ... 0 files
Docs modified ... NO
Misc/ACKS updated ... NO
Misc/NEWS updated ... NO
[51233 refs]

patchcheck.py needs to be modified so that it cd's into SRCDIR before running any hg commands.
msg173497 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-10-22 05:39
FYI hg -R path_to_repo works, we don't have to cd.
History
Date User Action Args
2014-06-29 20:31:16belopolskysetstatus: open -> closed
resolution: fixed
stage: needs patch -> resolved
2012-10-22 05:39:05eric.araujosetmessages: + msg173497
2012-08-30 14:59:59trentsetmessages: + msg169465
2012-08-29 15:05:59trentsetnosy: + trent
2011-08-01 12:49:58python-devsetmessages: + msg141500
2011-08-01 12:45:24python-devsetnosy: + python-dev
messages: + msg141496
2011-07-05 16:20:23eric.araujosetmessages: + msg139895
2011-01-11 15:54:49belopolskysetnosy: doko, belopolsky, pitrou, ned.deily, eric.araujo, rpetrov, dmalcolm
messages: + msg126016
versions: + Python 3.3, - Python 3.1
2010-09-21 19:16:34ned.deilysetnosy: + ned.deily
messages: + msg117096
2010-09-21 15:07:26belopolskysetmessages: + msg117061
2010-09-20 21:09:13eric.araujosetnosy: + eric.araujo
messages: + msg116972
2010-09-15 21:56:11belopolskysetmessages: + msg116493
2010-09-15 21:45:49rpetrovsetnosy: + rpetrov
messages: + msg116490
2010-09-15 15:30:45belopolskysetmessages: + msg116459
2010-09-15 15:24:43belopolskysetpriority: normal -> low

messages: + msg116458
2010-09-15 15:18:24pitrousetnosy: + pitrou
messages: + msg116456
2010-09-15 14:45:49belopolskysetnosy: + doko, dmalcolm
messages: + msg116451
2010-09-15 14:08:13belopolskycreate