I built Python 3.6 (default) on FreeBSD using:

./configure --cache-file=config_debug.cache --with-pydebug CFLAGS=-O0
make

I'm using FreeBSD current, uname -a:

FreeBSD freebsd 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r296485: Tue Mar  8 07:04:36 UTC 2016     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

It looks like "cc" is clang LLVM compiler 3.8.0.

Extract of the make output:
---
[haypo@freebsd ~/prog/python/default]$ make
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
*** WARNING: importing extension "_testmultiphase" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_ssl" failed with <class 'RuntimeError'>: invalid slot offset
*** WARNING: importing extension "_curses_panel" failed with <class 'RuntimeError'>: invalid slot offset
---

Oh ok, I understood the issue.
---                                                       
[haypo@freebsd ~/prog/python/default]$ rm Objects/typeslots.inc 
[haypo@freebsd ~/prog/python/default]$ make
python ./Objects/typeslots.py < ./Include/typeslots.h > Objects/typeslots.inc
/bin/sh: python: not found
*** Error code 127

Stop.
make: stopped in /home/haypo/prog/python/default
---

I have "python2" and "python3" but no "python" on FreeBSD CURRENT. The configure script is smart enough to select "python3" as the ASDL generator, but it uses $(PYTHON) ("python") to build Objects/typeslots.inc.

We should use a smarter code to select the external "python" program to build Objects/typeslots.inc.

Moreover, when "make Objects/typeslots.inc" fails, it creates an empty Objects/typeslots.inc file:
---
Objects/typeobject.o: Objects/typeslots.inc
Objects/typeslots.inc: $(srcdir)/Include/typeslots.h $(srcdir)/Objects/typeslots.py
        $(PYTHON) $(srcdir)/Objects/typeslots.py < $(srcdir)/Include/typeslots.h > Objects/typeslots.inc
---

The Objects/typeslots.inc should be written directly by Objects/typeslots.py.

Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

I noticed there are other boostrap scripts that require a “python” command via various mechanisms, e.g. Python/makeopcodetargets.py is run via “#! /usr/bin/env python” by default. It would at least be nice to make everything uniform, i.e. always use a /usr/bin/env hashbang, always use $(PYTHON), or always use this smarter external Python command.

> Typeslots.inc is listed in the .hgtouch file, so you might be able to work around by running “make touch” before building.

My initial issue is that "import ssl" fails with <class 'RuntimeError'>: invalid slot offset. This error is really strange and Google doesn't know it.

I probably missed something when I compiled Python, but I would prefer that Python build system detects such issues for me, rather than having to use workarounds.

Maybe the configure.ac code for generating opcode.h is a better alternative. It looks like it prints a warning but would continue with the build:

if test "$PYTHON" = not-found; then
    OPCODEHGEN="@echo python: $PYTHON! cannot run Tools/scripts/generate_opcode_h.py"
else
    OPCODEHGEN="$PYTHON"
fi

Hit by missing `python` command from Python/makeopcodetargets.py, too. Is `make touch` the correct way?

Victor: Gnu Make has a special “.DELETE_ON_ERROR” target that will cause Objects/typeslots.inc to be removed if the “python” command fails, which is slightly better than truncating it. See delete-on-error.patch. It looks like you were not using Gnu Make, but maybe there is an equivalent for BSD’s Make.

Chi: Yes I think “make touch” may help you work around the problem. It should be safe to try anyway :)

Thanks! It's now working find with `make touch` :)

Right, I built Python on FreeBSD, it's likely that make was not GNU make.

Survey of different mechanisms for running Python to generate files:

Tools/scripts/generate_opcode_h.py and Parser/asdl_c.py: configure.ac tries python3.6, python3, python, etc commands. It either runs the scripts with what it finds, or substitutes an “echo” command that emits an explanation but does not fail. One disadvantage of this was it needed a script to be made compatible with Python 2.5 <https://bugs.python.org/issue17861#msg217409>.

Python/makeopcodetargets.py: Always run as a Unix command with shebang “#! /usr/bin/env python”.

Objects/typeslots.py: Always run via $(PYTHON) command, which defaults to “python$(EXE)”. No magic configure stuff, but could be manually adjusted by using “make PYTHON=[. . .]”. This is the only remaining use of the $(PYTHON) makefile variable in 3.6, but it is shared with other stuff in 2.7.

An idea I had a while ago was to only run these scripts with a special Make command, eliminating the need for “make touch”. Going over a discussion earlier this year, Nick Coghlan <http://article.gmane.org/gmane.comp.python.devel/156682> said a disadvantage of this is that you have to remember the special Make command to use in the rare event that your work affects a generated. Maybe this could be overcome with documentation, warning messages, etc. Argument Clinic code and the configure script are not automatically regenerated.

Another potential idea is a flag (either to configure, or to Make) to manually disable or enable all running of Python scripts. When disabled, it would use the existing generated files, and perhaps warn if they are out of date. Or by default perhaps the warning could be an error, with the resolution to either tell it to use a pre-installed Python, or explicitly run the build with the existing generated files.

This patch follows the first mechanism listed by Martin. The change in configure.ac fixes the problem described by Victor, it does not generate an empty Objects/typeslots.inc file when python is not found that would cause the cryptic " <class 'RuntimeError'>: invalid slot offset" errors on subsequent builds.

The change in Objects/typeslots.py is not strictly necessary to fix the problem but prevents typeslots.py to create an invalid typeslots.inc file through the previous stdout redirection mechanism when, for example, typeslots.py is modified inadvertently with a change that is not python2 compatible and typeslots.py is run by python2 and fails.

Forgot to say that with the patch and no python in $PATH, the following error message is output:
    $ touch Include/typeslots.h
    $ make
    Cannot generate Objects/typeslots.inc, python not found !
    Re-run configure with python in PATH.
    make: *** [Makefile:908: Objects/typeslots.inc] Error 1

The PYTHON_FOR_GEN scheme seems reasonable to me, as is changing the way typeslots script is run.

I do wonder about the advice in the message. If I ran into the problem, I would probably either override PYTHON_FOR_GEN when running Make, or try something like “make touch” instead. But that’t not a big deal.

Following (and thanks to :) Martin's review and comments, this new patch:
* does not use the not portable '-e' echo option
* uses single quotes (') to avoid the escaping
* improves the error message

“make touch” does not always fix the problem, for example when Objects/typeslots.py has been modified or when this file has been touched, so it is not mentioned in the error message.

The new patch looks good enough.

The main reason I complained about the error message is that it sounds like you need Python in order to build Python. Obviously you need Python to run a modified file like typeslots.py, but there is supposed to be an alternative if you don’t need to regenerate files. I think running “make touch” should fudge the timestamps so that Make does not run it. It works for me:

$ touch Objects/typeslots.py
$ make touch
cd .; \
	hg --config extensions.touch=Tools/hg/hgtouch.py touch -v
Touching Objects/typeslots.inc
$ make  # Does not run typeslots.py
gcc [. . .] Objects/typeobject.c
[. . .]

Unfortunately, I understand “make touch” requires Mercurial, which requires Python 2. That weakens my argument about bootstrapping Python, but it is still valid in some scenarios. Maybe we should recommend “make -t Objects/typeslots.inc” etc instead of “make touch”.

So the error message could be ('$@' being the target):

Cannot generate $@, python not found !
To skip re-generation of $@ run 'make touch' or 'make -t $@'.
Otherwise, set python in PATH and run configure or run make with PYTHON_FOR_GEN=python.

Martin, what do you think ?

Issue 24034 is a duplicate.

I think that error message would be okay.

Patch with a new error message.

New changeset 5aff28f33b2a by Xavier de Gaye in branch '3.5':
Issue #26662: Set PYTHON_FOR_GEN in configure
https://hg.python.org/cpython/rev/5aff28f33b2a

New changeset a290f992e69a by Xavier de Gaye in branch 'default':
(merge from 3.5) Issue #26662: Set PYTHON_FOR_GEN in configure
https://hg.python.org/cpython/rev/a290f992e69a