In Fedora, when building Python 3.11.0a2 with Python 3.11.0a2 for regen, we found that regen-frozen fails.

It can be reproduced either on the v3.11.0a2 tag or on the main branch.

On tag v3.11.0a2 with 3.11.0a2 installed in $PATH:

# in root of cpython local clone:
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make regen-all PYTHON_FOR_REGEN=python3.11
...
ERROR: missing _freeze_module

On the main branch:

# in root of cpython local clone:
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make
$ make regen-all PYTHON_FOR_REGEN=./python
(success)

However, the working tree is dirty:

$ git diff
diff --git a/Parser/parser.c b/Parser/parser.c
index b3aa35989ed..c26cd6eeb05 100644
--- a/Parser/parser.c
+++ b/Parser/parser.c
@@ -1,4 +1,4 @@
-// @generated by pegen from ./Grammar/python.gram
+// @generated by pegen from ../../Grammar/python.gram
 #include "pegen.h"
 
 #if defined(Py_DEBUG) && defined(Py_BUILD_CORE)
diff --git a/Tools/peg_generator/pegen/grammar_parser.py b/Tools/peg_generator/pegen/grammar_parser.py
index 6e9f7d3d11d..fbbbfad76b0 100644
--- a/Tools/peg_generator/pegen/grammar_parser.py
+++ b/Tools/peg_generator/pegen/grammar_parser.py
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3.8
-# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram
+# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram
 
 import ast
 import sys

And if we install Python somewhere:

$ make
$ make install DESTDIR=/tmp/python

And use that one again for regen:

$ cd ../..
$ git clean -fdx
$ mkdir -p build/optimized
$ cd build/optimized
$ ../../configure
$ make regen-all PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11
...
# Regenerate Lib/keyword.py from Grammar/python.gram and Grammar/Tokens
# using Tools/peg_generator/pegen
PYTHONPATH=../../Tools/peg_generator /tmp/python/usr/local/bin/python3.11 -m pegen.keywordgen \
	../../Grammar/python.gram \
	../../Grammar/Tokens \
	../../Lib/keyword.py.new
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/update_file.py ../../Lib/keyword.py ../../Lib/keyword.py.new
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

It fails. I've isolated the failure to:

$ make regen-frozen PYTHON_FOR_REGEN=/tmp/python/usr/local/bin/python3.11
/tmp/python/usr/local/bin/python3.11 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

git bisect says:

ff8859d is the first new commit
commit ff8859d
Author: Victor Stinner <vstinner@python.org>
Date: Thu Oct 7 21:19:13 2021 +0200

bpo-45402: Fix test_tools.test_sundry() (GH-28786)

Fix test_tools.test_sundry() when Python is built out of tree: fix
how the freeze_modules.py tool locates the _freeze_module program.

Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst | 3 +++
Tools/scripts/freeze_modules.py | 7 ++++---
2 files changed, 7 insertions(+), 3 deletions(-)
create mode 100644 Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst

It seems the fix for an out-of-tree build bug introduced another one.

It also seems that it is *not* necessary to use Python 3.11 for regen-frozen to get the failure:

$ make regen-frozen PYTHON_FOR_REGEN=python3.10
python3.10 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

$ make regen-frozen PYTHON_FOR_REGEN=python3.9
python3.9 ../../Tools/scripts/freeze_modules.py
ERROR: missing _freeze_module
make: *** [Makefile:1259: regen-frozen] Error 1

My guess is that the fix for test_tools.test_sundry() actually broke the real usage. It has:

        # When building out of the source tree, get the tool from directory
        # of the Python executable
        TOOL = os.path.dirname(sys.executable)
        TOOL = os.path.join(TOOL, 'Programs', '_freeze_module')

But sys.executable is *not* the Python we have built, but the Python we use for regen.
E.g. when Python for regen is /usr/bin/python3.X, it tries to use /usr/bin/Programs/_freeze_module which obviously is not there.

The assumption that sys.executable is the freshly built Python is only correct in the tests, but not in reality.

New changeset 253b7a0 by Victor Stinner in branch 'main':
bpo-45866: pegen strips directory of "generated from" header (GH-29777)
253b7a0

New changeset b6defde by Victor Stinner in branch '3.10':
bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792)
b6defde

New changeset 93a540d by Victor Stinner in branch '3.9':
bpo-45866: pegen strips directory of "generated from" header (GH-29777) (GH-29792) (GH-29797)
93a540d

It seems like all bugs reported in this issue has been fixed, so I close the issue. Thanks for the bug report ;-)

-# @generated by pegen from ./Tools/peg_generator/pegen/metagrammar.gram
+# @generated by pegen from ../../Tools/peg_generator/pegen/metagrammar.gram

This issue is now fixed in 3.9, 3.10 and main branches.

ERROR: missing _freeze_module

I'm no longer able to reproduce this error on an up-to-date main branch. The issue has been fixed by this change:

commit 5be98e5
Author: Guido van Rossum <guido@python.org>
Date: Tue Nov 23 08:56:06 2021 -0800

bpo-45873: Get rid of bootstrap_python (bpo-29717)

Instead we use $(PYTHON_FOR_REGEN) .../deepfreeze.py with the
frozen .h file as input, as we did for Windows in bpo-45850.

We also get rid of the code that generates the .h files
when make regen-frozen is run (i.e., .../make_frozen.py),
and the MANIFEST file.

Restore Python 3.8 and 3.9 as Windows host Python again

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

New changeset 9a7611a by Victor Stinner in branch 'main':
bpo-45866: Fix typo in the NEWS entry (GH-29798)
9a7611a

New changeset fc16ea9 by Miss Islington (bot) in branch '3.10':
bpo-45866: Fix typo in the NEWS entry (GH-29798)
fc16ea9

New changeset b3f14da by Miss Islington (bot) in branch '3.9':
bpo-45866: Fix typo in the NEWS entry (GH-29798)
b3f14da