configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc #70849

vstinner · 2016-03-29T10:38:23Z

BPO	26662
Nosy	@vstinner, @xdegaye, @vadmium, @koobs, @yan12125
Files	delete-on-error.patch py_for_gen_26662.patch py_for_gen_26662_2.patch py_for_gen_26662_3.patch: patch with a new error message

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-07-26.11:00:00.993>
created_at = <Date 2016-03-29.10:38:22.939>
labels = ['build']
title = 'configure/Makefile doesn\'t check if "python" command works, needed to build Objects/typeslots.inc'
updated_at = <Date 2016-07-26.11:34:34.112>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2016-07-26.11:34:34.112>
actor = 'koobs'
assignee = 'none'
closed = True
closed_date = <Date 2016-07-26.11:00:00.993>
closer = 'xdegaye'
components = ['Build']
creation = <Date 2016-03-29.10:38:22.939>
creator = 'vstinner'
dependencies = []
files = ['43683', '43767', '43792', '43829']
hgrepos = []
issue_num = 26662
keywords = ['patch']
message_count = 20.0
messages = ['262600', '262602', '262634', '262636', '262639', '269926', '270159', '270160', '270161', '270162', '270652', '270653', '270667', '270824', '270904', '270913', '270915', '270968', '270980', '271356']
nosy_count = 6.0
nosy_names = ['vstinner', 'xdegaye', 'python-dev', 'martin.panter', 'koobs', 'yan12125']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26662'
versions = ['Python 3.5', 'Python 3.6']

vstinner · 2016-03-29T10:38:22Z

I built Python 3.6 (default) on FreeBSD using:

./configure --cache-file=config_debug.cache --with-pydebug CFLAGS=-O0
make

I'm using FreeBSD current, uname -a:

FreeBSD freebsd 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r296485: Tue Mar 8 07:04:36 UTC 2016 root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64

It looks like "cc" is clang LLVM compiler 3.8.0.

Extract of the make output:
---
[haypo@freebsd ~/prog/python/default]$ make
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
*** WARNING: importing extension "_testmultiphase" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_ssl" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_curses_panel" failed with <class 'RuntimeError'>: invalid slot offset
---

vstinner · 2016-03-29T10:55:04Z

Oh ok, I understood the issue.

[haypo@freebsd ~/prog/python/default]$ rm Objects/typeslots.inc
[haypo@freebsd ~/prog/python/default]$ make
python ./Objects/typeslots.py < ./Include/typeslots.h > Objects/typeslots.inc
/bin/sh: python: not found
*** Error code 127

Stop.
make: stopped in /home/haypo/prog/python/default
---

I have "python2" and "python3" but no "python" on FreeBSD CURRENT. The configure script is smart enough to select "python3" as the ASDL generator, but it uses $(PYTHON) ("python") to build Objects/typeslots.inc.

We should use a smarter code to select the external "python" program to build Objects/typeslots.inc.

Moreover, when "make Objects/typeslots.inc" fails, it creates an empty Objects/typeslots.inc file:
---
Objects/typeobject.o: Objects/typeslots.inc
Objects/typeslots.inc: $(srcdir)/Include/typeslots.h $(srcdir)/Objects/typeslots.py
$(PYTHON) $(srcdir)/Objects/typeslots.py < $(srcdir)/Include/typeslots.h > Objects/typeslots.inc
---

The Objects/typeslots.inc should be written directly by Objects/typeslots.py.

vadmium · 2016-03-29T22:32:22Z

Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

I noticed there are other boostrap scripts that require a “python” command via various mechanisms, e.g. Python/makeopcodetargets.py is run via “#! /usr/bin/env python” by default. It would at least be nice to make everything uniform, i.e. always use a /usr/bin/env hashbang, always use $(PYTHON), or always use this smarter external Python command.

vstinner · 2016-03-29T23:13:41Z

Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

My initial issue is that "import ssl" fails with <class 'RuntimeError'>: invalid slot offset. This error is really strange and Google doesn't know it.

I probably missed something when I compiled Python, but I would prefer that Python build system detects such issues for me, rather than having to use workarounds.

vadmium · 2016-03-29T23:41:27Z

Maybe the configure.ac code for generating opcode.h is a better alternative. It looks like it prints a warning but would continue with the build:

if test "$PYTHON" = not-found; then
    OPCODEHGEN="@echo python: $PYTHON! cannot run Tools/scripts/generate_opcode_h.py"
else
    OPCODEHGEN="$PYTHON"
fi

yan12125 · 2016-07-07T08:43:56Z

Hit by missing python command from Python/makeopcodetargets.py, too. Is make touch the correct way?

vadmium · 2016-07-11T04:34:54Z

Victor: Gnu Make has a special “.DELETE_ON_ERROR” target that will cause Objects/typeslots.inc to be removed if the “python” command fails, which is slightly better than truncating it. See delete-on-error.patch. It looks like you were not using Gnu Make, but maybe there is an equivalent for BSD’s Make.

Chi: Yes I think “make touch” may help you work around the problem. It should be safe to try anyway :)

yan12125 · 2016-07-11T05:58:44Z

Thanks! It's now working find with make touch :)

vstinner · 2016-07-11T07:40:10Z

Right, I built Python on FreeBSD, it's likely that make was not GNU make.

vadmium · 2016-07-11T07:48:42Z

Survey of different mechanisms for running Python to generate files:

Tools/scripts/generate_opcode_h.py and Parser/asdl_c.py: configure.ac tries python3.6, python3, python, etc commands. It either runs the scripts with what it finds, or substitutes an “echo” command that emits an explanation but does not fail. One disadvantage of this was it needed a script to be made compatible with Python 2.5 <https://bugs.python.org/issue17861#msg217409\>.

Python/makeopcodetargets.py: Always run as a Unix command with shebang “#! /usr/bin/env python”.

Objects/typeslots.py: Always run via $(PYTHON) command, which defaults to “python$(EXE)”. No magic configure stuff, but could be manually adjusted by using “make PYTHON=[. . .]”. This is the only remaining use of the $(PYTHON) makefile variable in 3.6, but it is shared with other stuff in 2.7.

An idea I had a while ago was to only run these scripts with a special Make command, eliminating the need for “make touch”. Going over a discussion earlier this year, Nick Coghlan <http://article.gmane.org/gmane.comp.python.devel/156682\> said a disadvantage of this is that you have to remember the special Make command to use in the rare event that your work affects a generated. Maybe this could be overcome with documentation, warning messages, etc. Argument Clinic code and the configure script are not automatically regenerated.

Another potential idea is a flag (either to configure, or to Make) to manually disable or enable all running of Python scripts. When disabled, it would use the existing generated files, and perhaps warn if they are out of date. Or by default perhaps the warning could be an error, with the resolution to either tell it to use a pre-installed Python, or explicitly run the build with the existing generated files.

xdegaye · 2016-07-17T17:15:25Z

This patch follows the first mechanism listed by Martin. The change in configure.ac fixes the problem described by Victor, it does not generate an empty Objects/typeslots.inc file when python is not found that would cause the cryptic " <class 'RuntimeError'>: invalid slot offset" errors on subsequent builds.

The change in Objects/typeslots.py is not strictly necessary to fix the problem but prevents typeslots.py to create an invalid typeslots.inc file through the previous stdout redirection mechanism when, for example, typeslots.py is modified inadvertently with a change that is not python2 compatible and typeslots.py is run by python2 and fails.

xdegaye · 2016-07-17T17:21:30Z

Forgot to say that with the patch and no python in $PATH, the following error message is output:
$ touch Include/typeslots.h
$ make
Cannot generate Objects/typeslots.inc, python not found !
Re-run configure with python in PATH.
make: *** [Makefile:908: Objects/typeslots.inc] Error 1

vadmium · 2016-07-17T23:05:16Z

The PYTHON_FOR_GEN scheme seems reasonable to me, as is changing the way typeslots script is run.

I do wonder about the advice in the message. If I ran into the problem, I would probably either override PYTHON_FOR_GEN when running Make, or try something like “make touch” instead. But that’t not a big deal.

xdegaye · 2016-07-19T13:16:47Z

Following (and thanks to :) Martin's review and comments, this new patch:

does not use the not portable '-e' echo option
uses single quotes (') to avoid the escaping
improves the error message

“make touch” does not always fix the problem, for example when Objects/typeslots.py has been modified or when this file has been touched, so it is not mentioned in the error message.

vadmium · 2016-07-21T03:05:35Z

The new patch looks good enough.

The main reason I complained about the error message is that it sounds like you need Python in order to build Python. Obviously you need Python to run a modified file like typeslots.py, but there is supposed to be an alternative if you don’t need to regenerate files. I think running “make touch” should fudge the timestamps so that Make does not run it. It works for me:

$ touch Objects/typeslots.py
$ make touch
cd .; \
	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Objects/typeslots.inc
$ make  # Does not run typeslots.py
gcc [. . .] Objects/typeobject.c
[. . .]

Unfortunately, I understand “make touch” requires Mercurial, which requires Python 2. That weakens my argument about bootstrapping Python, but it is still valid in some scenarios. Maybe we should recommend “make -t Objects/typeslots.inc” etc instead of “make touch”.

xdegaye · 2016-07-21T11:49:14Z

So the error message could be ('$@' being the target):

Cannot generate $@, python not found !
To skip re-generation of $@ run 'make touch' or 'make -t $@'.
Otherwise, set python in PATH and run configure or run make with PYTHON_FOR_GEN=python.

Martin, what do you think ?

xdegaye · 2016-07-21T12:03:34Z

bpo-24034 is a duplicate.

vadmium · 2016-07-22T03:37:10Z

I think that error message would be okay.

xdegaye · 2016-07-22T09:29:00Z

Patch with a new error message.

python-dev · 2016-07-26T10:56:36Z

New changeset 5aff28f33b2a by Xavier de Gaye in branch '3.5':
Issue bpo-26662: Set PYTHON_FOR_GEN in configure
https://hg.python.org/cpython/rev/5aff28f33b2a

New changeset a290f992e69a by Xavier de Gaye in branch 'default':
(merge from 3.5) Issue bpo-26662: Set PYTHON_FOR_GEN in configure
https://hg.python.org/cpython/rev/a290f992e69a

vstinner changed the title ~~FreeBSD: "invalid slot offset" error on _ssl, _curses_panel and _testmultiphase extensions~~ configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc Mar 29, 2016

vstinner added the build The build process and cross-build label Mar 29, 2016

xdegaye mannequin closed this as completed Jul 26, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc #70849

configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc #70849

vstinner commented Mar 29, 2016

vstinner commented Mar 29, 2016

vstinner commented Mar 29, 2016

vadmium commented Mar 29, 2016

vstinner commented Mar 29, 2016

vadmium commented Mar 29, 2016

yan12125 mannequin commented Jul 7, 2016

vadmium commented Jul 11, 2016

yan12125 mannequin commented Jul 11, 2016

vstinner commented Jul 11, 2016

vadmium commented Jul 11, 2016

xdegaye mannequin commented Jul 17, 2016

xdegaye mannequin commented Jul 17, 2016

vadmium commented Jul 17, 2016

xdegaye mannequin commented Jul 19, 2016

vadmium commented Jul 21, 2016

xdegaye mannequin commented Jul 21, 2016

xdegaye mannequin commented Jul 21, 2016

vadmium commented Jul 22, 2016

xdegaye mannequin commented Jul 22, 2016

python-dev mannequin commented Jul 26, 2016

configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc #70849

configure/Makefile doesn't check if "python" command works, needed to build Objects/typeslots.inc #70849

Comments

vstinner commented Mar 29, 2016

vstinner commented Mar 29, 2016

vstinner commented Mar 29, 2016

Oh ok, I understood the issue.

vadmium commented Mar 29, 2016

vstinner commented Mar 29, 2016

vadmium commented Mar 29, 2016

yan12125 mannequin commented Jul 7, 2016

vadmium commented Jul 11, 2016

yan12125 mannequin commented Jul 11, 2016

vstinner commented Jul 11, 2016

vadmium commented Jul 11, 2016

xdegaye mannequin commented Jul 17, 2016

xdegaye mannequin commented Jul 17, 2016

vadmium commented Jul 17, 2016

xdegaye mannequin commented Jul 19, 2016

vadmium commented Jul 21, 2016

xdegaye mannequin commented Jul 21, 2016

xdegaye mannequin commented Jul 21, 2016

vadmium commented Jul 22, 2016

xdegaye mannequin commented Jul 22, 2016

python-dev mannequin commented Jul 26, 2016