classification
Title: configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc
Type: Stage: resolved
Components: Build Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: koobs, martin.panter, python-dev, vstinner, xdegaye, yan12125
Priority: normal Keywords: patch

Created on 2016-03-29 10:38 by vstinner, last changed 2016-07-26 11:34 by koobs. This issue is now closed.

Files
File name Uploaded Description Edit
delete-on-error.patch martin.panter, 2016-07-11 04:34 review
py_for_gen_26662.patch xdegaye, 2016-07-17 17:15 review
py_for_gen_26662_2.patch xdegaye, 2016-07-19 13:16 review
py_for_gen_26662_3.patch xdegaye, 2016-07-22 09:29 patch with a new error message review
Messages (20)
msg262600 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 10:38
I built Python 3.6 (default) on FreeBSD using:

./configure --cache-file=config_debug.cache --with-pydebug CFLAGS=-O0
make

I'm using FreeBSD current, uname -a:

FreeBSD freebsd 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r296485: Tue Mar  8 07:04:36 UTC 2016     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

It looks like "cc" is clang LLVM compiler 3.8.0.

Extract of the make output:
---
[haypo@freebsd ~/prog/python/default]$ make
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
*** WARNING: importing extension "_testmultiphase" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_ssl" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_curses_panel" failed with <class 'RuntimeError'>: invalid slot offset
---
msg262602 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 10:55
Oh ok, I understood the issue.
---                                                       
[haypo@freebsd ~/prog/python/default]$ rm Objects/typeslots.inc 
[haypo@freebsd ~/prog/python/default]$ make
python ./Objects/typeslots.py < ./Include/typeslots.h > Objects/typeslots.inc
/bin/sh: python: not found
*** Error code 127

Stop.
make: stopped in /home/haypo/prog/python/default
---

I have "python2" and "python3" but no "python" on FreeBSD CURRENT. The configure script is smart enough to select "python3" as the ASDL generator, but it uses $(PYTHON) ("python") to build Objects/typeslots.inc.

We should use a smarter code to select the external "python" program to build Objects/typeslots.inc.

Moreover, when "make Objects/typeslots.inc" fails, it creates an empty Objects/typeslots.inc file:
---
Objects/typeobject.o: Objects/typeslots.inc
Objects/typeslots.inc: $(srcdir)/Include/typeslots.h $(srcdir)/Objects/typeslots.py
        $(PYTHON) $(srcdir)/Objects/typeslots.py < $(srcdir)/Include/typeslots.h > Objects/typeslots.inc
---

The Objects/typeslots.inc should be written directly by Objects/typeslots.py.
msg262634 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-29 22:32
Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

I noticed there are other boostrap scripts that require a “python” command via various mechanisms, e.g. Python/makeopcodetargets.py is run via “#! /usr/bin/env python” by default. It would at least be nice to make everything uniform, i.e. always use a /usr/bin/env hashbang, always use $(PYTHON), or always use this smarter external Python command.
msg262636 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-29 23:13
> Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

My initial issue is that "import ssl" fails with <class 'RuntimeError'>: invalid slot offset. This error is really strange and Google doesn't know it.

I probably missed something when I compiled Python, but I would prefer that Python build system detects such issues for me, rather than having to use workarounds.
msg262639 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-29 23:41
Maybe the configure.ac code for generating opcode.h is a better alternative. It looks like it prints a warning but would continue with the build:

if test "$PYTHON" = not-found; then
    OPCODEHGEN="@echo python: $PYTHON! cannot run Tools/scripts/generate_opcode_h.py"
else
    OPCODEHGEN="$PYTHON"
fi
msg269926 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-07-07 08:43
Hit by missing `python` command from Python/makeopcodetargets.py, too. Is `make touch` the correct way?
msg270159 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-11 04:34
Victor: Gnu Make has a special “.DELETE_ON_ERROR” target that will cause Objects/typeslots.inc to be removed if the “python” command fails, which is slightly better than truncating it. See delete-on-error.patch. It looks like you were not using Gnu Make, but maybe there is an equivalent for BSD’s Make.

Chi: Yes I think “make touch” may help you work around the problem. It should be safe to try anyway :)
msg270160 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-07-11 05:58
Thanks! It's now working find with `make touch` :)
msg270161 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-07-11 07:40
Right, I built Python on FreeBSD, it's likely that make was not GNU make.
msg270162 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-11 07:48
Survey of different mechanisms for running Python to generate files:

Tools/scripts/generate_opcode_h.py and Parser/asdl_c.py: configure.ac tries python3.6, python3, python, etc commands. It either runs the scripts with what it finds, or substitutes an “echo” command that emits an explanation but does not fail. One disadvantage of this was it needed a script to be made compatible with Python 2.5 <https://bugs.python.org/issue17861#msg217409>.

Python/makeopcodetargets.py: Always run as a Unix command with shebang “#! /usr/bin/env python”.

Objects/typeslots.py: Always run via $(PYTHON) command, which defaults to “python$(EXE)”. No magic configure stuff, but could be manually adjusted by using “make PYTHON=[. . .]”. This is the only remaining use of the $(PYTHON) makefile variable in 3.6, but it is shared with other stuff in 2.7.

An idea I had a while ago was to only run these scripts with a special Make command, eliminating the need for “make touch”. Going over a discussion earlier this year, Nick Coghlan <http://article.gmane.org/gmane.comp.python.devel/156682> said a disadvantage of this is that you have to remember the special Make command to use in the rare event that your work affects a generated. Maybe this could be overcome with documentation, warning messages, etc. Argument Clinic code and the configure script are not automatically regenerated.

Another potential idea is a flag (either to configure, or to Make) to manually disable or enable all running of Python scripts. When disabled, it would use the existing generated files, and perhaps warn if they are out of date. Or by default perhaps the warning could be an error, with the resolution to either tell it to use a pre-installed Python, or explicitly run the build with the existing generated files.
msg270652 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-17 17:15
This patch follows the first mechanism listed by Martin. The change in configure.ac fixes the problem described by Victor, it does not generate an empty Objects/typeslots.inc file when python is not found that would cause the cryptic " <class 'RuntimeError'>: invalid slot offset" errors on subsequent builds.

The change in Objects/typeslots.py is not strictly necessary to fix the problem but prevents typeslots.py to create an invalid typeslots.inc file through the previous stdout redirection mechanism when, for example, typeslots.py is modified inadvertently with a change that is not python2 compatible and typeslots.py is run by python2 and fails.
msg270653 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-17 17:21
Forgot to say that with the patch and no python in $PATH, the following error message is output:
    $ touch Include/typeslots.h
    $ make
    Cannot generate Objects/typeslots.inc, python not found !
    Re-run configure with python in PATH.
    make: *** [Makefile:908: Objects/typeslots.inc] Error 1
msg270667 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-17 23:05
The PYTHON_FOR_GEN scheme seems reasonable to me, as is changing the way typeslots script is run.

I do wonder about the advice in the message. If I ran into the problem, I would probably either override PYTHON_FOR_GEN when running Make, or try something like “make touch” instead. But that’t not a big deal.
msg270824 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-19 13:16
Following (and thanks to :) Martin's review and comments, this new patch:
* does not use the not portable '-e' echo option
* uses single quotes (') to avoid the escaping
* improves the error message

“make touch” does not always fix the problem, for example when Objects/typeslots.py has been modified or when this file has been touched, so it is not mentioned in the error message.
msg270904 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-21 03:05
The new patch looks good enough.

The main reason I complained about the error message is that it sounds like you need Python in order to build Python. Obviously you need Python to run a modified file like typeslots.py, but there is supposed to be an alternative if you don’t need to regenerate files. I think running “make touch” should fudge the timestamps so that Make does not run it. It works for me:

$ touch Objects/typeslots.py
$ make touch
cd .; \
	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Objects/typeslots.inc
$ make  # Does not run typeslots.py
gcc [. . .] Objects/typeobject.c
[. . .]

Unfortunately, I understand “make touch” requires Mercurial, which requires Python 2. That weakens my argument about bootstrapping Python, but it is still valid in some scenarios. Maybe we should recommend “make -t Objects/typeslots.inc” etc instead of “make touch”.
msg270913 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-21 11:49
So the error message could be ('$@' being the target):

Cannot generate $@, python not found !
To skip re-generation of $@ run 'make touch' or 'make -t $@'.
Otherwise, set python in PATH and run configure or run make with PYTHON_FOR_GEN=python.

Martin, what do you think ?
msg270915 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-21 12:03
Issue 24034 is a duplicate.
msg270968 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-07-22 03:37
I think that error message would be okay.
msg270980 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-22 09:29
Patch with a new error message.
msg271356 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-07-26 10:56
New changeset 5aff28f33b2a by Xavier de Gaye in branch '3.5':
Issue #26662: Set PYTHON_FOR_GEN in configure
https://hg.python.org/cpython/rev/5aff28f33b2a

New changeset a290f992e69a by Xavier de Gaye in branch 'default':
(merge from 3.5) Issue #26662: Set PYTHON_FOR_GEN in configure
https://hg.python.org/cpython/rev/a290f992e69a
History
Date User Action Args
2017-01-08 00:04:20martin.panterlinkissue21242 superseder
2016-07-26 11:34:34koobssetnosy: + koobs
2016-07-26 11:00:00xdegayesetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-07-26 10:56:36python-devsetnosy: + python-dev
messages: + msg271356
2016-07-22 09:29:01xdegayesetfiles: + py_for_gen_26662_3.patch

messages: + msg270980
2016-07-22 03:37:10martin.pantersetmessages: + msg270968
2016-07-21 12:03:34xdegayesetmessages: + msg270915
2016-07-21 11:49:14xdegayesetmessages: + msg270913
2016-07-21 03:05:36martin.pantersetmessages: + msg270904
stage: patch review -> commit review
2016-07-19 13:16:47xdegayesetfiles: + py_for_gen_26662_2.patch

messages: + msg270824
2016-07-17 23:05:16martin.pantersetmessages: + msg270667
2016-07-17 17:21:29xdegayesetmessages: + msg270653
2016-07-17 17:15:25xdegayesetfiles: + py_for_gen_26662.patch
versions: + Python 3.5
nosy: + xdegaye

messages: + msg270652

stage: patch review
2016-07-11 07:48:42martin.pantersetmessages: + msg270162
2016-07-11 07:40:09vstinnersetmessages: + msg270161
2016-07-11 05:58:43yan12125setmessages: + msg270160
2016-07-11 04:34:55martin.pantersetfiles: + delete-on-error.patch
keywords: + patch
messages: + msg270159
2016-07-07 08:43:55yan12125setnosy: + yan12125
messages: + msg269926
2016-03-29 23:41:27martin.pantersetmessages: + msg262639
2016-03-29 23:13:41vstinnersetmessages: + msg262636
2016-03-29 22:32:22martin.pantersetnosy: + martin.panter
messages: + msg262634
2016-03-29 21:59:03vstinnersetcomponents: + Build
2016-03-29 21:58:50vstinnersettitle: FreeBSD: "invalid slot offset" error on _ssl, _curses_panel and _testmultiphase extensions -> configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc
2016-03-29 10:55:04vstinnersetmessages: + msg262602
2016-03-29 10:38:22vstinnercreate