This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: FreeBSD buildbots cannot compile Python
Type: compile error Stage: resolved
Components: Build Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, eric.snow, erlendaasland, gvanrossum, koobs, lukasz.langa, miss-islington, ned.deily, pablogsal, steve.dower
Priority: Keywords: patch

Created on 2022-01-04 23:09 by pablogsal, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 30410 merged christian.heimes, 2022-01-05 01:59
PR 30413 merged miss-islington, 2022-01-05 09:54
PR 30414 merged miss-islington, 2022-01-05 09:54
PR 30434 merged christian.heimes, 2022-01-06 14:49
PR 30436 merged miss-islington, 2022-01-06 15:15
PR 30437 merged miss-islington, 2022-01-06 15:15
PR 30438 merged eric.snow, 2022-01-06 18:00
PR 30440 merged christian.heimes, 2022-01-06 19:22
PR 30442 merged miss-islington, 2022-01-06 19:49
PR 30443 merged miss-islington, 2022-01-06 19:49
Messages (41)
msg409719 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-04 23:09
Seems that the FreeBSD buildbots cannot compile Python 3.11 anymore:

LD_LIBRARY_PATH=/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build CC='cc -pthread' LDSHARED='cc -pthread -shared    ' OPT='-g -O0 -Wall'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/./setup.py", line 49, in <module>
    from distutils.command.build_ext import build_ext
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/distutils/command/build_ext.py", line 13, in <module>
    from distutils.sysconfig import customize_compiler, get_python_version
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/distutils/sysconfig.py", line 53, in <module>
    _config_vars = get_config_vars()
                   ^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/sysconfig.py", line 621, in get_config_vars
    _init_posix(_CONFIG_VARS)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/sysconfig.py", line 482, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_freebsd14_x86_64-unknown-freebsd14'
*** [sharedmods] Error code 1
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-564d/build
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-564d/build

https://buildbot.python.org/all/#/builders/483/builds/1408
msg409722 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-04 23:17
There is also this other builder:

https://buildbot.python.org/all/#/builders/172/builds/1093

failing what what seems is a freeze problem:

-  'use_frozen_modules': 1,
+  'use_frozen_modules': -1,
?                        +
   'use_hash_seed': 0,
   'user_site_directory': 1,
   'verbose': 0,
   'warn_default_encoding': 0,
   'warnoptions': [],
   'write_bytecode': 1,
   'xoptions': []}
----------------------------------------------------------------------
Ran 1 test in 0.152s
FAILED (failures=1)
test test_embed failed
1 test failed again:
    test_embed

Adding Eric and Guido for this particular problem to the nosy list
msg409727 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2022-01-05 00:02
Not to point out the obvious but this buildbot uses --enable-shared which is often a source of build / bootstrap issues, such as the compiler executable being built ends up dynamically loading a libpython from a previous installation or build.
msg409731 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-05 01:45
On my FreeBSD 13 VM the sysconfig data module is

   build/lib.freebsd-13.0-RELEASE-amd64-3.11-pydebug/_sysconfigdata_d_freebsd13_.py 

It does not have any multiarch suffix. The multiarch suffix is detected by configure.ac. Did FreeBSD 14.0's CC get support for "cc --print-multiarch"? I found a related FreeBSD bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258377
msg409741 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-05 09:33
I have tested my PR on a FreeBSD VM. Python fails to compile on main and builds correctly with my patch.
msg409744 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-05 09:54
New changeset cae55542d23e606dde9819d5dadd7430085fcc77 by Christian Heimes in branch 'main':
bpo-46263: Don't use MULTIARCH on FreeBSD (#30410)
https://github.com/python/cpython/commit/cae55542d23e606dde9819d5dadd7430085fcc77
msg409746 - (view) Author: miss-islington (miss-islington) Date: 2022-01-05 10:17
New changeset 7e951f356ec76a5a5fdb851d71df5d120014bf3f by Miss Islington (bot) in branch '3.10':
bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410)
https://github.com/python/cpython/commit/7e951f356ec76a5a5fdb851d71df5d120014bf3f
msg409747 - (view) Author: miss-islington (miss-islington) Date: 2022-01-05 10:20
New changeset 64199e9235275a795097ee0c53b2c560e21c70d0 by Miss Islington (bot) in branch '3.9':
bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410)
https://github.com/python/cpython/commit/64199e9235275a795097ee0c53b2c560e21c70d0
msg409765 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-05 13:27
I have backported the patch to all versions that receive regular fixes.
msg409768 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-05 13:49
I closed the bug without intention to do so. Sorry for the noise.
msg409775 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-05 15:29
FWIW: from test_embed.test_init_setpythonhome:

    if not config['executable']:
        config['use_frozen_modules'] = -1

From the buildbot test stdout (but not pythoninfo) I see that config["executable"] is set to "".
msg409824 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 10:07
Marked as a release blocker. See python-committers post:

https://mail.python.org/archives/list/python-committers@python.org/thread/7OWGAL4UL5HNKKQFUGH6N6L4JDVPEN7R/
msg409828 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-06 11:26
Unfortunately the buildbots are still red:

https://buildbot.python.org/all/#/builders/483/builds/1416/steps/5/logs/stdio

https://buildbot.python.org/all/#/builders/172/builds/1102

One is the test_embed problem with frozen modules:

======================================================================
FAIL: test_init_setpythonhome (test.test_embed.InitConfigTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/test_embed.py", line 1253, in test_init_setpythonhome
    self.check_all_configs("test_init_setpythonhome", config,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/test_embed.py", line 736, in check_all_configs
    self.check_config(configs, expected_config)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib/test/test_embed.py", line 662, in check_config
    self.assertEqual(config, expected)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: {'_co[346 chars]'], 'xoptions': [], 'warnoptions': [], 'python[1945 chars]': 1} != {'_co[346 chars]'], 'orig_argv': [], 'xoptions': [], 'warnopti[1946 chars]: -1}
  {'_config_init': 1,
   '_init_main': 1,
   '_install_importlib': 1,
   '_isolated_interpreter': 0,
   'argv': [''],
   'base_exec_prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build',
   'base_executable': '',
   'base_prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build',
   'buffered_stdio': 1,
   'bytes_warning': 0,
   'check_hash_pycs_mode': 'default',
   'code_debug_ranges': 1,
   'configure_c_stdio': 0,
   'dev_mode': 0,
   'dump_refs': 0,
   'exec_prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build',
   'executable': '',
   'faulthandler': 0,
   'hash_seed': 0,
   'home': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build',
   'import_time': 0,
   'inspect': 0,
   'install_signal_handlers': 1,
   'interactive': 0,
   'isolated': 0,
   'malloc_stats': 0,
   'module_search_paths': ['/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/target/lib/python311.zip',
                           '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib',
                           '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/build/lib.freebsd-12.3-RELEASE-amd64-3.11',
                           '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python311.zip',
                           '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python3.11',
                           '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python3.11/lib-dynload'],
   'module_search_paths_set': 1,
   'optimization_level': 0,
   'orig_argv': [],
   'parse_argv': 0,
   'parser_debug': 0,
   'pathconfig_warnings': 1,
   'platlibdir': 'lib',
   'prefix': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build',
   'program_name': 'python3',
   'pycache_prefix': None,
   'pythonpath_env': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/target/lib/python311.zip:/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/Lib:/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/build/lib.freebsd-12.3-RELEASE-amd64-3.11',
   'quiet': 0,
   'run_command': None,
   'run_filename': None,
   'run_module': None,
   'show_ref_count': 0,
   'site_import': 1,
   'skip_source_first_line': 0,
   'stdlib_dir': '/usr/home/buildbot/python/3.x.koobs-freebsd-9e36.nondebug/build/lib/python3.11',
   'tracemalloc': 0,
   'use_environment': 1,
-  'use_frozen_modules': 1,
+  'use_frozen_modules': -1,
?                        +
   'use_hash_seed': 0,
   'user_site_directory': 1,
   'verbose': 0,
   'warn_default_encoding': 0,
   'warnoptions': [],
   'write_bytecode': 1,
   'xoptions': []}
----------------------------------------------------------------------
Ran 1 test in 0.131s
FAILED (failures=1)
test test_embed failed
0:27:39 load avg: 1.11 Re-running test_logging in verbose mode (matching: test_udp_reconnection)
test_udp_reconnection (test.test_logging.IPv6SysLogHandlerTest) ... ok
test_udp_reconnection (test.test_logging.SysLogHandlerTest) ... ok
test_udp_reconnection (test.test_logging.UnixSysLogHandlerTest) ... ok
----------------------------------------------------------------------



The other is a capi problem:

======================================================================
FAIL: test_pyobject_freed_is_freed (test.test_capi.PyMemMallocDebugTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_capi.py", line 943, in test_pyobject_freed_is_freed
    self.check_pyobject_is_freed('check_pyobject_freed_is_freed')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_capi.py", line 931, in check_pyobject_is_freed
    assert_python_ok('-c', code, PYTHONMALLOC=self.PYTHONMALLOC)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 145, in _assert_python
    res.fail(cmd_line)
    ^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/support/script_helper.py", line 72, in fail
    raise AssertionError("Process return code is %d\n"
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Process return code is 1
command line: ['/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/python', '-X', 'faulthandler', '-c', '\nimport gc, os, sys, _testcapi\n# Disable the GC to avoid crash on GC collection\ngc.disable()\ntry:\n    _testcapi.check_pyobject_freed_is_freed()\n    # Exit immediately to avoid a crash while deallocating\n    # the invalid object\n    os._exit(0)\nexcept _testcapi.error:\n    os._exit(1)\n']
stdout:
---
---
stderr:
---
---
msg409831 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 11:42
Here's a strange observation from my FreeBSD 14.0 VM:

With system supplied python3.8 (dependency of the git package), test_embed succeeds. If I explicitly install python3.9 (pkg install python3.9 installs Python 3.9.9), rebuild and test, test_embed fails.
msg409832 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 11:44
FYI, the buildbot has Python 3.9.9 installed, as seen from the configure step.

(nit: it's pkg install python39, not pkg install python3.9)
msg409833 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 11:52
Wait a minute. It is not tied to the Python version installed; it seems to be an issue with in-tree vs. out-of-tree builds. The failed build was in-tree. The build that succeeded was out-of-tree. I did a new build with Python 3.9 installed, which succeeded out-of-tree.

To clear things up:

- out-of-tree build with system Python 3.8 => succeeds
- out-of-tree build with system Python 3.9 => succeeds
- in-tree build with system Python 3.9 => fails
msg409834 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 12:02
FTR:

- in-tree build with system Python 3.8 => fails
msg409850 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-06 14:27
The test_capi check for check_pyobject_freed_is_freed() is failing because FreeBSD 14.0 uses jemalloc, which performs its own dead memory cleaning. jemalloc's free() fills freed memory with byte 0x5a, which overrides Python's 0xdd marker.

$ PYTHONMALLOC=malloc_debug gdb ./python
(gdb) b check_pyobject_freed_is_freed
(gdb) b _PyMem_DebugRawFree
(gdb) disable 2
(gdb) run -X faulthandler t.py
...
Breakpoint 1, check_pyobject_freed_is_freed (self=0x800f16fe0, _unused_args=0x0) at Modules/_testcapimodule.c:5032
5032        PyObject *op = PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type);
(gdb) enable 2
(gdb) c
Continuing.

Breakpoint 2, _PyMem_DebugRawFree (ctx=0x5f8ad8 <_PyMem_Debug+96>, p=0x800e1a0a0) at Objects/obmalloc.c:2544
2544        if (p == NULL) {
(gdb) n
2549        uint8_t *q = (uint8_t *)p - 2*SST;  /* address returned from malloc */
(gdb) n
2552        _PyMem_DebugCheckAddress(__func__, api->api_id, p);
(gdb) n
2553        nbytes = read_size_t(q);
(gdb) n
2554        nbytes += PYMEM_DEBUG_EXTRA_BYTES;
(gdb) n
2555        memset(q, PYMEM_DEADBYTE, nbytes);
(gdb) n
2556        api->alloc.free(api->alloc.ctx, q);
(gdb) p *q@20
$18 = '\335' <repeats 19 times>, <incomplete sequence \335>
(gdb) n
_PyMem_RawFree (ctx=0x0, ptr=0x800e1a090) at Objects/obmalloc.c:127
127     {
(gdb) n
128         free(ptr);
(gdb) p free
$19 = {void (void *)} 0x8006002c0 <free>
(gdb) s
__free (ptr=0x800e1a090) at jemalloc_jemalloc.c:2848
2848    jemalloc_jemalloc.c: No such file or directory.
(gdb) n
2851    in jemalloc_jemalloc.c
(gdb) 
2852    in jemalloc_jemalloc.c
...
(gdb) 
check_pyobject_freed_is_freed (self=0x800f16fe0, _unused_args=0x0) at Modules/_testcapimodule.c:5038
5038        Py_SET_REFCNT(op, 1);
(gdb) p *(char*)op@20
$23 = 'Z' <repeats 20 times>
msg409851 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-06 14:40
$ PYTHONMALLOC=malloc_debug ./python t.py
Traceback (most recent call last):
  File "/usr/home/heimes/dev/python/cpython/t.py", line 4, in <module>
    _testcapi.check_pyobject_freed_is_freed()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_testcapi.error: check_pyobject_freed_is_freed: object is not seen as freed
$ echo $?
1

$ MALLOC_CONF="junk:false" PYTHONMALLOC=malloc_debug ./python t.py
$ echo $?
0
msg409858 - (view) Author: miss-islington (miss-islington) Date: 2022-01-06 15:14
New changeset a4aa52dc2801d25b6343fe2ef8de7f40ea3bc883 by Christian Heimes in branch 'main':
bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434)
https://github.com/python/cpython/commit/a4aa52dc2801d25b6343fe2ef8de7f40ea3bc883
msg409863 - (view) Author: miss-islington (miss-islington) Date: 2022-01-06 15:37
New changeset b951dec4418326c06d493020dadedf85a3b29a82 by Miss Islington (bot) in branch '3.10':
bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434)
https://github.com/python/cpython/commit/b951dec4418326c06d493020dadedf85a3b29a82
msg409865 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-06 15:52
New changeset b259015c1079744dbe3d58bd4c27757a6df9d1c6 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) (GH-30437)
https://github.com/python/cpython/commit/b259015c1079744dbe3d58bd4c27757a6df9d1c6
msg409873 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-01-06 16:35
Is this under control, or do you still need help with some part? (Are there
two separate issues or not?)--
--Guido (mobile)
msg409875 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-06 16:44
test_capi is fixed now. test_embed is still an issue.
msg409876 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-06 16:47
In summary, regarding the test_embed failure:

> FWIW: from test_embed.test_init_setpythonhome:
> 
>     if not config['executable']:
>         config['use_frozen_modules'] = -1
>
> From the buildbot test stdout (but not pythoninfo) I see that
> config["executable"] is set to "".

...

> ======================================================================
> FAIL: test_init_setpythonhome (test.test_embed.InitConfigTests)
> ----------------------------------------------------------------------
> ...
> -  'use_frozen_modules': 1,
> +  'use_frozen_modules': -1,
> ?                        +
> ...

...

> - out-of-tree build with system Python 3.8 => succeeds
> - out-of-tree build with system Python 3.9 => succeeds
> - in-tree build with system Python 3.9 => fails
> - in-tree build with system Python 3.8 => fails

Note that in-tree builds are working find on other platforms, AFAICS.
msg409877 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-06 16:48
> Is this under control, or do you still need help with some part?

As Christian mentions, we still have the test_embed issue
msg409878 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 16:50
> Note that in-tree builds are working find on other platforms, AFAICS.

Yes, almost all of the buildbots are using in-tree builds. IIRC, buildbots for out-of-tree builds were added just recently.
msg409880 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 16:57
AFAICS, the test_embed failure first appeared after GH-29041 (bpo-45582) was merged. After GH-29041, there has been numerous fixes for test_embed:

    $ git log --oneline | grep bpo-45582
    3f398a77d3 bpo-45582: Fix test_embed failure during a PGO build on Windows (GH-30014)
    b0b3086279 bpo-45582: Write empty pybuilddir.txt on Windows to allow relocatable build directories (GH-29979)
    06c4ae8b13 bpo-45582: Fix framework path and bootstrap build (GH-29954)
    b7ef27bc08 bpo-45582: Ensure PYTHONHOME still overrides detected build prefixes (GH-29948)
    f16f93e527 bpo-45582: framework build: modPath must not be const (GH-29944)
    af1db4eb55 bpo-45582: Fix getpath_isxfile() and test_embed on Windows (GH-29930)
    628abe4463 bpo-45582: Fix signature of _Py_Get_Getpath_CodeObject (GH-29921)
    0ae4e0c959 bpo-45582 Fix prototype of _Py_Get_Getpath_CodeObject. (GH-29907)
    7d7c91a8e8 bpo-45582: Add a NOT operator to the condition in getpath_isxfile (GH-29906)
    ccb73a0d50 bpo-45582: Fix out-of-tree build issues with new getpath (GH-29902)
    99fcf15052 bpo-45582: Port getpath[p].c to Python (GH-29041)
msg409882 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 16:59
> the test_embed failure first appeared after GH-29041 (bpo-45582) was merged.

Make that "first appeared on FreeBSD".

Also, numerous was perhaps an exaggeration. Make that a few :)
msg409885 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-06 17:19
Just to be clear, the test_embed failure is on AMD64 FreeBSD Non-Debug 3.x (https://buildbot.python.org/all/#/builders/172).

I agree that 99fcf15052 (bpo-45582: Port getpath[p].c to Python (GH-29041)) is likely to be the cause.
msg409888 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-01-06 17:38
IIUC Steve is still on vacation, so we shouldn't wait for him to fix this.
msg409889 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-06 17:41
A FreeBSD builder where it's working:

AMD64 FreeBSD Shared 3.x (koobs-freebsd-564d)
OS: FreeBSD 14.0-CURRENT main-n244626-cb7cc72c546: GENERIC-NODEBUG (amd64) System: 2-core, 8Gb, VirtualBox (6.x) Guest Features: - Compiler: FreeBSD clang version 11.0.1 (llvmorg-11.0.1-0-g43ff75f2c3fe) - Clang "is" cc (default system compiler) - OpenSSL 1.1.1i-freebsd 8 Dec 2020

Where it isn't:

AMD64 FreeBSD Non-Debug 3.x (koobs-freebsd-9e36)
Environment: * OS: FreeBSD 12.3-RELEASE r371126 GENERIC amd64 * System: 2-core, 2Gb, VirtualBox (6.x) Guest Features: * cc = FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) * DTRACE * ZFS * Posix Semaphores

The main difference I see is the FreeBSD version.
msg409891 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 17:51
Also, AMD64 FreeBSD Shared 3.x is configured --with-pydebug and --enable-shared. I'd be surprised if those were the root cause of this though. Both buildbots are in-tree builds.

FWIW, my FreeBSD 14.0-CURRENT main-n250453-7ac82c96fe7 VM also fails, as noted in previous posts. It's built with just ./configure && make.
msg409892 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-06 18:01
I just put up a PR that should fix the failure.
msg409896 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-06 18:56
New changeset 68c76d9766cccb5fd992b0ac4b39645d9665dbe2 by Eric Snow in branch 'main':
bpo-46263: Do not ever expect "use_frozen_modules" to be -1. (gh-30438)
https://github.com/python/cpython/commit/68c76d9766cccb5fd992b0ac4b39645d9665dbe2
msg409897 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 19:01
Great. I tried the same trick on my VM earlier today, but I was unsure if it was the correct fix :) Thanks!
msg409900 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-06 19:09
test_pyobject_freed_is_freed is still failing on AMD64 FreeBSD Shared 3.x.
msg409906 - (view) Author: miss-islington (miss-islington) Date: 2022-01-06 19:49
New changeset c9137d4b638c0699b904011cafe68895d28dd80b by Christian Heimes in branch 'main':
bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440)
https://github.com/python/cpython/commit/c9137d4b638c0699b904011cafe68895d28dd80b
msg409910 - (view) Author: miss-islington (miss-islington) Date: 2022-01-06 20:12
New changeset 3cb64ede57bba260228864150123014fc682167e by Miss Islington (bot) in branch '3.10':
bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440)
https://github.com/python/cpython/commit/3cb64ede57bba260228864150123014fc682167e
msg409913 - (view) Author: miss-islington (miss-islington) Date: 2022-01-06 20:16
New changeset b98730c5165467c6053e7274ae094d733aea804d by Miss Islington (bot) in branch '3.9':
bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440)
https://github.com/python/cpython/commit/b98730c5165467c6053e7274ae094d733aea804d
msg409924 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2022-01-06 20:49
Both FreeBSD bots are green again. Thank you Christian and Eric!
History
Date User Action Args
2022-04-11 14:59:54adminsetgithub: 90421
2022-04-08 10:04:22koobssetnosy: + koobs
2022-01-06 20:49:48erlendaaslandsetstatus: open -> closed
priority: release blocker ->
messages: + msg409924

resolution: fixed
stage: patch review -> resolved
2022-01-06 20:16:03miss-islingtonsetmessages: + msg409913
2022-01-06 20:12:17miss-islingtonsetmessages: + msg409910
2022-01-06 19:49:27miss-islingtonsetpull_requests: + pull_request28649
2022-01-06 19:49:22miss-islingtonsetpull_requests: + pull_request28648
2022-01-06 19:49:12miss-islingtonsetmessages: + msg409906
2022-01-06 19:22:01christian.heimessetstage: patch review
pull_requests: + pull_request28647
2022-01-06 19:09:47eric.snowsetstatus: pending -> open

messages: + msg409900
stage: patch review -> (no value)
2022-01-06 19:02:26erlendaaslandsetstatus: open -> pending
2022-01-06 19:01:52erlendaaslandsetstatus: pending -> open

messages: + msg409897
2022-01-06 18:57:50eric.snowsetstatus: open -> pending
2022-01-06 18:56:16eric.snowsetmessages: + msg409896
2022-01-06 18:01:27eric.snowsetmessages: + msg409892
2022-01-06 18:00:47eric.snowsetpull_requests: + pull_request28645
2022-01-06 17:51:43erlendaaslandsetmessages: + msg409891
2022-01-06 17:41:28eric.snowsetmessages: + msg409889
2022-01-06 17:38:04gvanrossumsetmessages: + msg409888
2022-01-06 17:19:05eric.snowsetnosy: + steve.dower
messages: + msg409885
2022-01-06 16:59:56erlendaaslandsetmessages: + msg409882
2022-01-06 16:57:08erlendaaslandsetmessages: + msg409880
2022-01-06 16:50:42erlendaaslandsetmessages: + msg409878
2022-01-06 16:48:38pablogsalsetmessages: + msg409877
2022-01-06 16:47:02eric.snowsetmessages: + msg409876
2022-01-06 16:44:24christian.heimessetmessages: + msg409875
2022-01-06 16:35:36gvanrossumsetmessages: + msg409873
2022-01-06 15:52:24christian.heimessetmessages: + msg409865
2022-01-06 15:37:05miss-islingtonsetmessages: + msg409863
2022-01-06 15:15:06miss-islingtonsetpull_requests: + pull_request28644
2022-01-06 15:15:00miss-islingtonsetpull_requests: + pull_request28643
2022-01-06 15:14:57miss-islingtonsetmessages: + msg409858
2022-01-06 14:49:49christian.heimessetstage: resolved -> patch review
pull_requests: + pull_request28641
2022-01-06 14:40:00christian.heimessetmessages: + msg409851
2022-01-06 14:27:50christian.heimessetmessages: + msg409850
2022-01-06 12:02:25erlendaaslandsetmessages: + msg409834
2022-01-06 11:52:32erlendaaslandsetmessages: + msg409833
2022-01-06 11:44:38erlendaaslandsetmessages: + msg409832
2022-01-06 11:42:19erlendaaslandsetmessages: + msg409831
2022-01-06 11:26:51pablogsalsetmessages: + msg409828
2022-01-06 10:07:44erlendaaslandsetpriority: normal -> release blocker

messages: + msg409824
2022-01-05 15:29:52erlendaaslandsetmessages: + msg409775
2022-01-05 15:29:41erlendaaslandsetmessages: - msg409774
2022-01-05 15:27:59erlendaaslandsetmessages: + msg409774
2022-01-05 13:49:30erlendaaslandsetstatus: closed -> open

nosy: + erlendaasland
messages: + msg409768

resolution: fixed -> (no value)
2022-01-05 13:27:31erlendaaslandlinkissue46268 superseder
2022-01-05 13:27:12christian.heimessetstatus: open -> closed
type: compile error
messages: + msg409765

resolution: fixed
stage: patch review -> resolved
2022-01-05 10:20:13miss-islingtonsetmessages: + msg409747
2022-01-05 10:17:45miss-islingtonsetmessages: + msg409746
2022-01-05 09:58:41christian.heimessetpriority: release blocker -> normal
2022-01-05 09:54:33miss-islingtonsetpull_requests: + pull_request28620
2022-01-05 09:54:28miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request28619
2022-01-05 09:54:25christian.heimessetmessages: + msg409744
2022-01-05 09:33:07christian.heimessetmessages: + msg409741
components: + Build
versions: + Python 3.9, Python 3.10
2022-01-05 01:59:50christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request28616
2022-01-05 01:45:24christian.heimessetnosy: + christian.heimes
messages: + msg409731
2022-01-05 00:18:07gvanrossumsettitle: FreeBSL buiildbots cannot compile Python -> FreeBSD buildbots cannot compile Python
2022-01-05 00:02:44ned.deilysetnosy: + ned.deily
messages: + msg409727
2022-01-04 23:17:08pablogsalsetnosy: + gvanrossum, eric.snow
messages: + msg409722
2022-01-04 23:09:03pablogsalcreate