FreeBSD buildbots cannot compile Python #90421

pablogsal · 2022-01-04T23:09:04Z

BPO	46263
Nosy	@gvanrossum, @tiran, @ned-deily, @ambv, @ericsnowcurrently, @koobs, @zooba, @pablogsal, @miss-islington, @erlend-aasland
PRs	bpo-46263: Don't use MULTIARCH on FreeBSD #30410 [3.10] bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410) #30413 [3.9] bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410) #30414 bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory #30434 [3.10] bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) #30436 [3.9] bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) (GH-30437) #30437 bpo-46263: Do not ever expect "use_frozen_modules" to be -1. #30438 bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD #30440 [3.10] bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440) #30442 [3.9] bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440) #30443

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-06.20:49:48.873>
created_at = <Date 2022-01-04.23:09:03.585>
labels = ['build', '3.9', '3.10', '3.11']
title = 'FreeBSD buildbots cannot compile Python'
updated_at = <Date 2022-04-08.10:04:22.749>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2022-04-08.10:04:22.749>
actor = 'koobs'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-06.20:49:48.873>
closer = 'erlendaasland'
components = ['Build']
creation = <Date 2022-01-04.23:09:03.585>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 46263
keywords = ['patch']
message_count = 41.0
messages = ['409719', '409722', '409727', '409731', '409741', '409744', '409746', '409747', '409765', '409768', '409775', '409824', '409828', '409831', '409832', '409833', '409834', '409850', '409851', '409858', '409863', '409865', '409873', '409875', '409876', '409877', '409878', '409880', '409882', '409885', '409888', '409889', '409891', '409892', '409896', '409897', '409900', '409906', '409910', '409913', '409924']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'christian.heimes', 'ned.deily', 'lukasz.langa', 'eric.snow', 'koobs', 'steve.dower', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['30410', '30413', '30414', '30434', '30436', '30437', '30438', '30440', '30442', '30443']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue46263'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

pablogsal · 2022-01-04T23:09:03Z

Seems that the FreeBSD buildbots cannot compile Python 3.11 anymore:

LD_LIBRARY_PATH=/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build CC='cc -pthread' LDSHARED='cc -pthread -shared    ' OPT='-g -O0 -Wall'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/./setup.py", line 49, in <module>
    from distutils.command.build_ext import build_ext
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/distutils/command/build_ext.py", line 13, in <module>
    from distutils.sysconfig import customize_compiler, get_python_version
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/distutils/sysconfig.py", line 53, in <module>
    _config_vars = get_config_vars()
                   ^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/sysconfig.py", line 621, in get_config_vars
    _init_posix(_CONFIG_VARS)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/sysconfig.py", line 482, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_sysconfigdata_d_freebsd14_x86_64-unknown-freebsd14'
*** [sharedmods] Error code 1
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-564d/build
make: stopped in /usr/home/buildbot/python/3.x.koobs-freebsd-564d/build

https://buildbot.python.org/all/#/builders/483/builds/1408

pablogsal · 2022-01-04T23:17:08Z

There is also this other builder:

https://buildbot.python.org/all/#/builders/172/builds/1093

failing what what seems is a freeze problem:

-  'use_frozen_modules': 1,
+  'use_frozen_modules': -1,
?                        +
   'use_hash_seed': 0,
   'user_site_directory': 1,
   'verbose': 0,
   'warn_default_encoding': 0,
   'warnoptions': [],
   'write_bytecode': 1,
   'xoptions': []}

Ran 1 test in 0.152s
FAILED (failures=1)
test test_embed failed
1 test failed again:
test_embed

Adding Eric and Guido for this particular problem to the nosy list

ned-deily · 2022-01-05T00:02:45Z

Not to point out the obvious but this buildbot uses --enable-shared which is often a source of build / bootstrap issues, such as the compiler executable being built ends up dynamically loading a libpython from a previous installation or build.

tiran · 2022-01-05T01:45:24Z

On my FreeBSD 13 VM the sysconfig data module is

build/lib.freebsd-13.0-RELEASE-amd64-3.11-pydebug/sysconfigdata_d_freebsd13.py

It does not have any multiarch suffix. The multiarch suffix is detected by configure.ac. Did FreeBSD 14.0's CC get support for "cc --print-multiarch"? I found a related FreeBSD bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258377

tiran · 2022-01-05T09:33:08Z

I have tested my PR on a FreeBSD VM. Python fails to compile on main and builds correctly with my patch.

tiran · 2022-01-05T09:54:25Z

New changeset cae5554 by Christian Heimes in branch 'main':
bpo-46263: Don't use MULTIARCH on FreeBSD (bpo-30410)
cae5554

miss-islington · 2022-01-05T10:17:46Z

New changeset 7e951f3 by Miss Islington (bot) in branch '3.10':
bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410)
7e951f3

miss-islington · 2022-01-05T10:20:13Z

New changeset 64199e9 by Miss Islington (bot) in branch '3.9':
bpo-46263: Don't use MULTIARCH on FreeBSD (GH-30410)
64199e9

tiran · 2022-01-05T13:27:12Z

I have backported the patch to all versions that receive regular fixes.

erlend-aasland · 2022-01-05T13:49:30Z

I closed the bug without intention to do so. Sorry for the noise.

erlend-aasland · 2022-01-06T12:02:26Z

FTR:

in-tree build with system Python 3.8 => fails

tiran · 2022-01-06T14:27:50Z

The test_capi check for check_pyobject_freed_is_freed() is failing because FreeBSD 14.0 uses jemalloc, which performs its own dead memory cleaning. jemalloc's free() fills freed memory with byte 0x5a, which overrides Python's 0xdd marker.

$ PYTHONMALLOC=malloc_debug gdb ./python
(gdb) b check_pyobject_freed_is_freed
(gdb) b _PyMem_DebugRawFree
(gdb) disable 2
(gdb) run -X faulthandler t.py
...
Breakpoint 1, check_pyobject_freed_is_freed (self=0x800f16fe0, _unused_args=0x0) at Modules/_testcapimodule.c:5032
5032        PyObject *op = PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type);
(gdb) enable 2
(gdb) c
Continuing.

Breakpoint 2, _PyMem_DebugRawFree (ctx=0x5f8ad8 <_PyMem_Debug+96>, p=0x800e1a0a0) at Objects/obmalloc.c:2544
2544 if (p == NULL) {
(gdb) n
2549 uint8_t *q = (uint8_t *)p - 2*SST; /* address returned from malloc */
(gdb) n
2552 _PyMem_DebugCheckAddress(func, api->api_id, p);
(gdb) n
2553 nbytes = read_size_t(q);
(gdb) n
2554 nbytes += PYMEM_DEBUG_EXTRA_BYTES;
(gdb) n
2555 memset(q, PYMEM_DEADBYTE, nbytes);
(gdb) n
2556 api->alloc.free(api->alloc.ctx, q);
(gdb) p *q@20
$18 = '\335' <repeats 19 times>, <incomplete sequence \335>
(gdb) n
_PyMem_RawFree (ctx=0x0, ptr=0x800e1a090) at Objects/obmalloc.c:127
127 {
(gdb) n
128 free(ptr);
(gdb) p free
$19 = {void (void *)} 0x8006002c0 <free>
(gdb) s
__free (ptr=0x800e1a090) at jemalloc_jemalloc.c:2848
2848 jemalloc_jemalloc.c: No such file or directory.
(gdb) n
2851 in jemalloc_jemalloc.c
(gdb)
2852 in jemalloc_jemalloc.c
...
(gdb)
check_pyobject_freed_is_freed (self=0x800f16fe0, _unused_args=0x0) at Modules/_testcapimodule.c:5038
5038 Py_SET_REFCNT(op, 1);
(gdb) p *(char*)op@20
$23 = 'Z' <repeats 20 times>

tiran · 2022-01-06T14:40:01Z

$ PYTHONMALLOC=malloc_debug ./python t.py
Traceback (most recent call last):
  File "/usr/home/heimes/dev/python/cpython/t.py", line 4, in <module>
    _testcapi.check_pyobject_freed_is_freed()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_testcapi.error: check_pyobject_freed_is_freed: object is not seen as freed
$ echo $?
1

$ MALLOC_CONF="junk:false" PYTHONMALLOC=malloc_debug ./python t.py
$ echo $?
0

miss-islington · 2022-01-06T15:14:57Z

New changeset a4aa52d by Christian Heimes in branch 'main':
bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434)
a4aa52d

miss-islington · 2022-01-06T15:37:05Z

New changeset b951dec by Miss Islington (bot) in branch '3.10':
bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434)
b951dec

tiran · 2022-01-06T15:52:25Z

New changeset b259015 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-46263: FreeBSD 14.0 jemalloc workaround for junk bytes of freed memory (GH-30434) (GH-30437)
b259015

gvanrossum · 2022-01-06T16:35:36Z

Is this under control, or do you still need help with some part? (Are there
two separate issues or not?)--
--Guido (mobile)

tiran · 2022-01-06T16:44:24Z

test_capi is fixed now. test_embed is still an issue.

ericsnowcurrently · 2022-01-06T16:47:02Z

In summary, regarding the test_embed failure:

FWIW: from test_embed.test_init_setpythonhome:
if not config['executable']:
    config['use_frozen_modules'] = -1
From the buildbot test stdout (but not pythoninfo) I see that
config["executable"] is set to "".

...

======================================================================
FAIL: test_init_setpythonhome (test.test_embed.InitConfigTests)
----------------------------------------------------------------------
...

'use_frozen_modules': 1,

'use_frozen_modules': -1,
? +
...

...

out-of-tree build with system Python 3.8 => succeeds

out-of-tree build with system Python 3.9 => succeeds

in-tree build with system Python 3.9 => fails

in-tree build with system Python 3.8 => fails

Note that in-tree builds are working find on other platforms, AFAICS.

pablogsal · 2022-01-06T16:48:38Z

Is this under control, or do you still need help with some part?

As Christian mentions, we still have the test_embed issue

erlend-aasland · 2022-01-06T16:50:42Z

Note that in-tree builds are working find on other platforms, AFAICS.

Yes, almost all of the buildbots are using in-tree builds. IIRC, buildbots for out-of-tree builds were added just recently.

erlend-aasland · 2022-01-06T16:57:08Z

AFAICS, the test_embed failure first appeared after #73227 (bpo-45582) was merged. After #73227, there has been numerous fixes for test_embed:

    $ git log --oneline | grep bpo-45582
    3f398a77d3 bpo-45582: Fix test_embed failure during a PGO build on Windows (GH-30014)
    b0b3086279 bpo-45582: Write empty pybuilddir.txt on Windows to allow relocatable build directories (GH-29979)
    06c4ae8b13 bpo-45582: Fix framework path and bootstrap build (GH-29954)
    b7ef27bc08 bpo-45582: Ensure PYTHONHOME still overrides detected build prefixes (GH-29948)
    f16f93e527 bpo-45582: framework build: modPath must not be const (GH-29944)
    af1db4eb55 bpo-45582: Fix getpath_isxfile() and test_embed on Windows (GH-29930)
    628abe4463 bpo-45582: Fix signature of _Py_Get_Getpath_CodeObject (GH-29921)
    0ae4e0c959 bpo-45582 Fix prototype of _Py_Get_Getpath_CodeObject. (GH-29907)
    7d7c91a8e8 bpo-45582: Add a NOT operator to the condition in getpath_isxfile (GH-29906)
    ccb73a0d50 bpo-45582: Fix out-of-tree build issues with new getpath (GH-29902)
    99fcf15052 bpo-45582: Port getpath[p].c to Python (GH-29041)

erlend-aasland · 2022-01-06T16:59:56Z

the test_embed failure first appeared after #73227 (bpo-45582) was merged.

Make that "first appeared on FreeBSD".

Also, numerous was perhaps an exaggeration. Make that a few :)

ericsnowcurrently · 2022-01-06T17:19:06Z

Just to be clear, the test_embed failure is on AMD64 FreeBSD Non-Debug 3.x (https://buildbot.python.org/all/#/builders/172).

I agree that 99fcf15 (bpo-45582: Port getpath[p].c to Python (GH-29041)) is likely to be the cause.

gvanrossum · 2022-01-06T17:38:05Z

IIUC Steve is still on vacation, so we shouldn't wait for him to fix this.

ericsnowcurrently · 2022-01-06T17:41:28Z

A FreeBSD builder where it's working:

AMD64 FreeBSD Shared 3.x (koobs-freebsd-564d)
OS: FreeBSD 14.0-CURRENT main-n244626-cb7cc72c546: GENERIC-NODEBUG (amd64) System: 2-core, 8Gb, VirtualBox (6.x) Guest Features: - Compiler: FreeBSD clang version 11.0.1 (llvmorg-11.0.1-0-g43ff75f2c3fe) - Clang "is" cc (default system compiler) - OpenSSL 1.1.1i-freebsd 8 Dec 2020

Where it isn't:

AMD64 FreeBSD Non-Debug 3.x (koobs-freebsd-9e36)
Environment: * OS: FreeBSD 12.3-RELEASE r371126 GENERIC amd64 * System: 2-core, 2Gb, VirtualBox (6.x) Guest Features: * cc = FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) * DTRACE * ZFS * Posix Semaphores

The main difference I see is the FreeBSD version.

erlend-aasland · 2022-01-06T17:51:43Z

Also, AMD64 FreeBSD Shared 3.x is configured --with-pydebug and --enable-shared. I'd be surprised if those were the root cause of this though. Both buildbots are in-tree builds.

FWIW, my FreeBSD 14.0-CURRENT main-n250453-7ac82c96fe7 VM also fails, as noted in previous posts. It's built with just ./configure && make.

ericsnowcurrently · 2022-01-06T18:01:28Z

I just put up a PR that should fix the failure.

ericsnowcurrently · 2022-01-06T18:56:17Z

New changeset 68c76d9 by Eric Snow in branch 'main':
bpo-46263: Do not ever expect "use_frozen_modules" to be -1. (gh-30438)
68c76d9

erlend-aasland · 2022-01-06T19:01:52Z

Great. I tried the same trick on my VM earlier today, but I was unsure if it was the correct fix :) Thanks!

ericsnowcurrently · 2022-01-06T19:09:47Z

test_pyobject_freed_is_freed is still failing on AMD64 FreeBSD Shared 3.x.

miss-islington · 2022-01-06T19:49:13Z

New changeset c9137d4 by Christian Heimes in branch 'main':
bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440)
c9137d4

miss-islington · 2022-01-06T20:12:17Z

New changeset 3cb64ed by Miss Islington (bot) in branch '3.10':
bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440)
3cb64ed

miss-islington · 2022-01-06T20:16:03Z

New changeset b98730c by Miss Islington (bot) in branch '3.9':
bpo-46263: Fix second location that needs MALLOC_CONF on FreeBSD (GH-30440)
b98730c

erlend-aasland · 2022-01-06T20:49:49Z

Both FreeBSD bots are green again. Thank you Christian and Eric!

pablogsal added release-blocker 3.11 only security fixes labels Jan 4, 2022

gvanrossum changed the title ~~FreeBSL buiildbots cannot compile Python~~ FreeBSD buildbots cannot compile Python Jan 5, 2022

tiran added build The build process and cross-build 3.9 only security fixes 3.10 only security fixes labels Jan 5, 2022

tiran removed release-blocker labels Jan 5, 2022

tiran closed this as completed Jan 5, 2022

tiran added the build The build process and cross-build label Jan 5, 2022

tiran closed this as completed Jan 5, 2022

tiran added the build The build process and cross-build label Jan 5, 2022

erlend-aasland reopened this Jan 5, 2022

erlend-aasland closed this as completed Jan 6, 2022

erlend-aasland removed the release-blocker label Jan 6, 2022

erlend-aasland closed this as completed Jan 6, 2022

erlend-aasland removed the release-blocker label Jan 6, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

FreeBSD buildbots cannot compile Python #90421

FreeBSD buildbots cannot compile Python #90421

Comments

pablogsal commented Jan 4, 2022

pablogsal commented Jan 4, 2022

pablogsal commented Jan 4, 2022

ned-deily commented Jan 5, 2022

tiran commented Jan 5, 2022

tiran commented Jan 5, 2022

tiran commented Jan 5, 2022

miss-islington commented Jan 5, 2022

miss-islington commented Jan 5, 2022

tiran commented Jan 5, 2022

erlend-aasland commented Jan 5, 2022

erlend-aasland commented Jan 6, 2022

tiran commented Jan 6, 2022

tiran commented Jan 6, 2022

miss-islington commented Jan 6, 2022

miss-islington commented Jan 6, 2022

tiran commented Jan 6, 2022

gvanrossum commented Jan 6, 2022

tiran commented Jan 6, 2022

ericsnowcurrently commented Jan 6, 2022

pablogsal commented Jan 6, 2022

erlend-aasland commented Jan 6, 2022

erlend-aasland commented Jan 6, 2022

erlend-aasland commented Jan 6, 2022

ericsnowcurrently commented Jan 6, 2022

gvanrossum commented Jan 6, 2022

ericsnowcurrently commented Jan 6, 2022

erlend-aasland commented Jan 6, 2022

ericsnowcurrently commented Jan 6, 2022

ericsnowcurrently commented Jan 6, 2022

erlend-aasland commented Jan 6, 2022

ericsnowcurrently commented Jan 6, 2022

miss-islington commented Jan 6, 2022

miss-islington commented Jan 6, 2022

miss-islington commented Jan 6, 2022

erlend-aasland commented Jan 6, 2022