Dear core developers, I noticed that for many recent commits, the s390x Fedora LTO + PGO 3.x buildbot often fails. Here's an error log::


gcc -pthread   -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g  -Xlinker -export-dynamic -o python Programs/python.o libpython3.10.a -lcrypt -lpthread -ldl  -lutil -lm   -lm 
gcc -pthread   -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g  -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o libpython3.10.a -lcrypt -lpthread -ldl  -lutil -lm   -lm 
/usr/bin/ld: python.lto.o: in function `run_mod':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1230: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `builtin_compile':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Python/bltinmodule.c:808: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `Py_CompileStringObject':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Python/pythonrun.c:1307: undefined reference to `PyAST_CompileObject'
/usr/bin/ld: python.lto.o: in function `symtable_lookup':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1004: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `symtable_record_directive':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1161: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `type_new':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Objects/typeobject.c:2544: undefined reference to `_Py_Mangle'
/usr/bin/ld: python.lto.o: in function `symtable_add_def_helper':
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Python/symtable.c:1018: undefined reference to `_Py_Mangle'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:592: python] Error 1
make[1]: *** Waiting for unfinished jobs....

[.............. I truncated the rest]

I have a hunch the error is caused by a lack of the '-fprofile-generate' flag for the first two gcc commands. The only issue is that it's sometimes there, and sometimes not, and I'm not familiar enough with the buildbot code to find out why. Sorry.

Eg. this commit https://github.com/python/cpython/commit/cc3467a57b61b0e7ef254b36790a1c44b13f2228 has s390x Fedora LTO + PGO 3.x succeeding, the 2 gcc lines also have the '-fprofile-generate' flag.

This commit https://github.com/python/cpython/commit/c6c43b28746b0642cc3c49dd8138b896bed3028f has s390x Fedora LTO + PGO 3.x failing, the 2 gcc lines have a blank space in place of the '-fprofile-generate' flag.

I could also be completely wrong and off track, so please feel free to correct me :), thanks.