classification
Title: SyntaxError in asdl when building 2.7 with system Python 3
Type: enhancement Stage: resolved
Components: Devguide, Documentation Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arfrever, chris.jerdonek, christian.heimes, docs@python, eli.bendersky, ezio.melotti, loewis, ncoghlan, python-dev, roger.serwy, vstinner
Priority: normal Keywords: easy, needs review, patch

Created on 2012-09-18 14:26 by chris.jerdonek, last changed 2013-10-08 13:09 by eli.bendersky. This issue is now closed.

Files
File name Uploaded Description Edit
issue-15964-1.patch chris.jerdonek, 2012-09-22 13:09
patch_2and3.py roger.serwy, 2013-04-04 02:32 review
patch_2and3_rev1.py roger.serwy, 2013-04-11 04:23 review
Messages (24)
msg170646 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-18 14:26
Building with--

./configure --with-pydebug && make -j2

errors out after switching branches from default to 2.7 when the system Python is Python 3 (on Mac OS X 10.7.4 using MacPorts).

To reproduce:

$ sudo port select python python32
$ python
No such file or directory
$ python3 -V
Python 3.2.3
$ hg update default
$ ./configure --with-pydebug && make -j2
$ hg update 2.7
$ cp Modules/Setup.dist Modules/Setup
$ ./configure --with-pydebug && make -j2
...
config.status: creating pyconfig.h
config.status: pyconfig.h is unchanged
creating Modules/Setup
creating Modules/Setup.local
creating Makefile
./install-sh -c -d Include
./Parser/asdl_c.py -h Include ./Parser/Python.asdl
gcc -c -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c
gcc -c -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c
  File "./Parser/asdl_c.py", line 1
    print "Must specify exactly one output file"
                                               ^
SyntaxError: invalid syntax
make: *** [Include/Python-ast.h] Error 1
make: *** Waiting for unfinished jobs....

Running "make distclean" doesn't fix this.  Here is one work-around:

$ sudo port select python python27
$ ./configure --with-pydebug && make -j2

Perhaps the right work-around can be documented in the devguide, or maybe this can be addressed another way.
msg170682 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-09-18 22:09
I think this might be related to #15923. It shouldn't be necessary to rebuild the AST definition unless you have modified the grammar and AST files.
msg170720 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-19 10:34
The case that "python" is a Python 3 binary is not a supported installation (see PEP 394). asdl_c.py works on both 2.x and 3.x unmodified in the 3.x branch, however, backporting this to 2.7 would be a new feature (support for building on systems where "python" is Python 3).

The proper work-around is to "make touch" after a hg update to bring the files in the right order. 

#15923 is really a separate issue since it complains about an actual bug in asdl_c.py.

So I propose to close this as "won't fix".
msg170728 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-19 16:29
> The case that "python" is a Python 3 binary is not a supported installation

Just to clarify, in the original scenario, "python" did not refer to anything.  From the original comment:

$ python
No such file or directory

("python2" and "python3" did refer to the respective versions.)

> The proper work-around is to "make touch" after a hg update to bring the files in the right order.

I had tried this in the 2.7 branch and got:

$ make touch
make: *** No rule to make target `touch'.  Stop.
msg170729 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-09-19 16:40
> $ make touch
> make: *** No rule to make target `touch'.  Stop.

Martin meant:

  touch Include/Python-ast.h Python/Python-ast.c
msg170730 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-19 16:54
Yes, that works.  Rather than closing this as "won't fix," however, I would suggest that we document the workaround in the devguide.
msg170928 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-21 21:39
Switching this to a devguide issue so the work-around can be documented, for example in this section:

http://docs.python.org/devguide/faq.html#how-do-i-switch-between-branches-inside-my-working-copy
msg170991 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-22 13:09
Attaching patch.

I started a new section in the FAQ called "Build Troubleshooting" which is something Nick suggested in the context of addressing this issue.  I'm sure we could add a couple more questions to this section right now if we wanted to.
msg170992 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-22 13:12
Adding Nick because he is the one that suggested adding a "Build Troubleshooting" section to the devguide FAQ (meant to add him above).
msg171001 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2012-09-22 15:20
We should be able to add the "make touch" target to the 2.7 Makefile without running afoul of the "no new features" rule.

However, agreed this is mostly a docs problem. We're seeing a couple of common traps that people run into when first getting started, so it will be good to get answers up somewhere where people are likely to look (and where search engines will be able to see the error messages).
msg171018 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-23 08:55
> We should be able to add the "make touch" target to the 2.7 Makefile without running afoul of the "no new features" rule.

To keep things simpler, I'm going to create a separate issue for this so that it can be discussed and committed independently of this doc-only issue.  If necessary, we can always modify the answers to existing FAQ's in that issue.
msg171020 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-23 09:05
> We should be able to add the "make touch" target to the 2.7 Makefile without running afoul of the "no new features" rule.

I created issue 16004 for this: http://bugs.python.org/issue16004
msg171080 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-23 22:01
> Martin meant:
>  touch Include/Python-ast.h Python/Python-ast.c

No, I *actually* meant 'make touch'.
msg171081 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2012-09-23 22:05
> Just to clarify, in the original scenario, "python" did not refer to 
> anything.  From the original comment:
> $ python
> No such file or directory

So can you find out why asdl_c.py actually printed this error? With no python executable in PATH, it shouldn't even run.
msg171091 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2012-09-24 03:37
> So can you find out why asdl_c.py actually printed this error?

I collected some additional information.  Here is the beginning of Parser/asdl_c.py:

#! /usr/bin/env python
"""Generate C code from an ASDL description."""

And in the scenario of this issue I have--

$ python
-bash: /opt/local/Library/Frameworks/Python.framework/Versions/Current/bin/python: No such file or directory

But--

$ /usr/bin/env python -V
Python 3.2.3

Some more data:

$ /usr/bin/env python -c "import sys; print(sys.executable)"
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/Resources/Python.app/Contents/MacOS/Python
$ echo $PATH
/opt/local/Library/Frameworks/Python.framework/Versions/Current/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/git/bin:/usr/texbin
$ which python
/opt/local/bin/python
$ /opt/local/bin/python -c "import sys; print(sys.executable)"
/opt/local/Library/Frameworks/Python.framework/Versions/3.2/Resources/Python.app/Contents/MacOS/Python

It's not immediately clear to me why running `python` doesn't find python in the second $PATH entry even though `which python` does.  Also, creating a new terminal doesn't seem to exhibit the same problem, so the error state may not be fully reproducible after all (at least on my machine).

In any case, Christian's work-around does resolve the issue for this particular error state while other solutions don't.
msg184006 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-12 04:14
`make touch` is now in 2.7, so the patch should be updated.
msg185662 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-03-31 20:01
I use Arch Linux which has /usr/bin/python as 3.3.0. I can confirm that "touch Include/Python-ast.h Python/Python-ast.c" fixes the problem. 

But would it have been simpler just to convert the print statements to functions so that asdl_c.py would work with Python 3 as well as Python 2?
msg185982 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-04-04 00:27
Do you want to provide a patch and test that it works?
msg185995 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-04-04 02:32
The attached patch allows the same parser code to run under Python 2 and Python 3. I'm not sure how to test it other than by running the test suite which passed.
msg186507 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-04-10 15:25
I left a review.  To test it you could try to reproduce the steps described in the first message and see what happens.
msg186543 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2013-04-11 04:23
Attached is the updated patch to include Ezio's review. Thanks Ezio!
msg198449 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-09-26 18:38
Did anyone ended up updating the devguide about this? I can't find anything related in there.
msg198770 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-10-01 13:13
New changeset 56ed149e597a by Eli Bendersky in branch 'default':
Mention 'make touch' in the devguide.
http://hg.python.org/devguide/rev/56ed149e597a
msg198772 - (view) Author: Eli Bendersky (eli.bendersky) * (Python committer) Date: 2013-10-01 13:27
`make touch` was added to the bots in #19106 and is now also recommended in the devguide for folks running into this problem. IMHO this is all we need here, because there's no point toiling to make the auto-generation scripts in every version to work with any other version of Python. It's error prone and keeps us back (there's no reason not to use modern Python practices in the ASDL generator of 3.4, say, and instead cling to Python 2.4).

So I'm going to close this issue in a few days unless new feedback emerges.
History
Date User Action Args
2013-10-08 13:09:22eli.benderskysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2013-10-01 13:27:41eli.benderskysetmessages: + msg198772
2013-10-01 13:13:55python-devsetnosy: + python-dev
messages: + msg198770
2013-09-26 18:38:55eli.benderskysetnosy: + eli.bendersky
messages: + msg198449
2013-04-11 04:23:31roger.serwysetfiles: + patch_2and3_rev1.py

messages: + msg186543
2013-04-10 15:25:09ezio.melottisetmessages: + msg186507
2013-04-04 02:32:42roger.serwysetfiles: + patch_2and3.py

messages: + msg185995
2013-04-04 00:27:39ezio.melottisetmessages: + msg185982
2013-03-31 20:01:32roger.serwysetnosy: + roger.serwy
messages: + msg185662
2013-03-12 04:14:40ezio.melottisetmessages: + msg184006
2012-09-24 03:37:47chris.jerdoneksetmessages: + msg171091
2012-09-23 22:05:31loewissetmessages: + msg171081
2012-09-23 22:01:23loewissetmessages: + msg171080
2012-09-23 09:22:08chris.jerdoneksetkeywords: + needs review
stage: needs patch -> patch review
2012-09-23 09:05:57chris.jerdoneksetmessages: + msg171020
2012-09-23 08:55:14chris.jerdoneksetmessages: + msg171018
2012-09-22 15:20:25ncoghlansetmessages: + msg171001
2012-09-22 13:12:27chris.jerdoneksetnosy: + ncoghlan
messages: + msg170992
2012-09-22 13:09:05chris.jerdoneksetfiles: + issue-15964-1.patch
keywords: + patch
messages: + msg170991
2012-09-21 21:39:40chris.jerdoneksetassignee: docs@python
type: behavior -> enhancement
components: + Documentation, Devguide, - Build

keywords: + easy
nosy: + ezio.melotti, docs@python
messages: + msg170928
stage: needs patch
2012-09-19 16:54:37chris.jerdoneksetmessages: + msg170730
2012-09-19 16:40:31christian.heimessetmessages: + msg170729
2012-09-19 16:29:44chris.jerdoneksetmessages: + msg170728
2012-09-19 10:34:43loewissetnosy: + loewis
messages: + msg170720
2012-09-18 22:32:24Arfreversetnosy: + Arfrever
2012-09-18 22:09:55christian.heimessetnosy: + christian.heimes
messages: + msg170682
2012-09-18 21:48:49vstinnersetnosy: + vstinner
2012-09-18 14:26:18chris.jerdonekcreate