Building with--

./configure --with-pydebug && make -j2

errors out after switching branches from default to 2.7 when the system Python is Python 3 (on Mac OS X 10.7.4 using MacPorts).

To reproduce:

$ sudo port select python python32
$ python
No such file or directory
$ python3 -V
Python 3.2.3
$ hg update default
$ ./configure --with-pydebug && make -j2
$ hg update 2.7
$ cp Modules/Setup.dist Modules/Setup
$ ./configure --with-pydebug && make -j2
...
config.status: creating pyconfig.h
config.status: pyconfig.h is unchanged
creating Modules/Setup
creating Modules/Setup.local
creating Makefile
./install-sh -c -d Include
./Parser/asdl_c.py -h Include ./Parser/Python.asdl
gcc -c -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c
gcc -c -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c
  File "./Parser/asdl_c.py", line 1
    print "Must specify exactly one output file"
                                               ^
SyntaxError: invalid syntax
make: *** [Include/Python-ast.h] Error 1
make: *** Waiting for unfinished jobs....

Running "make distclean" doesn't fix this. Here is one work-around:

$ sudo port select python python27
$ ./configure --with-pydebug && make -j2

Perhaps the right work-around can be documented in the devguide, or maybe this can be addressed another way.

I think this might be related to bpo-15923. It shouldn't be necessary to rebuild the AST definition unless you have modified the grammar and AST files.

The case that "python" is a Python 3 binary is not a supported installation (see PEP-394). asdl_c.py works on both 2.x and 3.x unmodified in the 3.x branch, however, backporting this to 2.7 would be a new feature (support for building on systems where "python" is Python 3).

The proper work-around is to "make touch" after a hg update to bring the files in the right order.

bpo-15923 is really a separate issue since it complains about an actual bug in asdl_c.py.

So I propose to close this as "won't fix".

The case that "python" is a Python 3 binary is not a supported installation

Just to clarify, in the original scenario, "python" did not refer to anything. From the original comment:

$ python
No such file or directory

("python2" and "python3" did refer to the respective versions.)

The proper work-around is to "make touch" after a hg update to bring the files in the right order.

I had tried this in the 2.7 branch and got:

$ make touch
make: *** No rule to make target `touch'.  Stop.

$ make touch
make: *** No rule to make target `touch'. Stop.

Martin meant:

touch Include/Python-ast.h Python/Python-ast.c

Yes, that works. Rather than closing this as "won't fix," however, I would suggest that we document the workaround in the devguide.

Switching this to a devguide issue so the work-around can be documented, for example in this section:

http://docs.python.org/devguide/faq.html#how-do-i-switch-between-branches-inside-my-working-copy

Attaching patch.

I started a new section in the FAQ called "Build Troubleshooting" which is something Nick suggested in the context of addressing this issue. I'm sure we could add a couple more questions to this section right now if we wanted to.

Adding Nick because he is the one that suggested adding a "Build Troubleshooting" section to the devguide FAQ (meant to add him above).

We should be able to add the "make touch" target to the 2.7 Makefile without running afoul of the "no new features" rule.

However, agreed this is mostly a docs problem. We're seeing a couple of common traps that people run into when first getting started, so it will be good to get answers up somewhere where people are likely to look (and where search engines will be able to see the error messages).

We should be able to add the "make touch" target to the 2.7 Makefile without running afoul of the "no new features" rule.

To keep things simpler, I'm going to create a separate issue for this so that it can be discussed and committed independently of this doc-only issue. If necessary, we can always modify the answers to existing FAQ's in that issue.

We should be able to add the "make touch" target to the 2.7 Makefile without running afoul of the "no new features" rule.

I created bpo-16004 for this: http://bugs.python.org/issue16004

Martin meant:
touch Include/Python-ast.h Python/Python-ast.c

No, I *actually* meant 'make touch'.

Just to clarify, in the original scenario, "python" did not refer to
anything. From the original comment:
$ python
No such file or directory

So can you find out why asdl_c.py actually printed this error? With no python executable in PATH, it shouldn't even run.

So can you find out why asdl_c.py actually printed this error?

I collected some additional information. Here is the beginning of Parser/asdl_c.py:

#! /usr/bin/env python
"""Generate C code from an ASDL description."""

And in the scenario of this issue I have--

$ python
-bash: /opt/local/Library/Frameworks/Python.framework/Versions/Current/bin/python: No such file or directory

But--

$ /usr/bin/env python -V
Python 3.2.3

Some more data:

$ /usr/bin/env python -c "import sys; print(sys.executable)"
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/Resources/Python.app/Contents/MacOS/Python
$ echo $PATH
/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/git/bin:/usr/texbin
$ which python
/opt/local/bin/python
$ /opt/local/bin/python -c "import sys; print(sys.executable)"
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/Resources/Python.app/Contents/MacOS/Python

It's not immediately clear to me why running python doesn't find python in the second $PATH entry even though which python does. Also, creating a new terminal doesn't seem to exhibit the same problem, so the error state may not be fully reproducible after all (at least on my machine).

In any case, Christian's work-around does resolve the issue for this particular error state while other solutions don't.

make touch is now in 2.7, so the patch should be updated.

I use Arch Linux which has /usr/bin/python as 3.3.0. I can confirm that "touch Include/Python-ast.h Python/Python-ast.c" fixes the problem.

But would it have been simpler just to convert the print statements to functions so that asdl_c.py would work with Python 3 as well as Python 2?

Do you want to provide a patch and test that it works?

The attached patch allows the same parser code to run under Python 2 and Python 3. I'm not sure how to test it other than by running the test suite which passed.

I left a review. To test it you could try to reproduce the steps described in the first message and see what happens.

Attached is the updated patch to include Ezio's review. Thanks Ezio!

Did anyone ended up updating the devguide about this? I can't find anything related in there.

New changeset 56ed149e597a by Eli Bendersky in branch 'default':
Mention 'make touch' in the devguide.
http://hg.python.org/devguide/rev/56ed149e597a

make touch was added to the bots in bpo-19106 and is now also recommended in the devguide for folks running into this problem. IMHO this is all we need here, because there's no point toiling to make the auto-generation scripts in every version to work with any other version of Python. It's error prone and keeps us back (there's no reason not to use modern Python practices in the ASDL generator of 3.4, say, and instead cling to Python 2.4).

So I'm going to close this issue in a few days unless new feedback emerges.