Title: Redefinition of malloc(3) family of functions at build time
Type: crash Stage: resolved
Components: ctypes Versions: Python 3.4, Python 3.3, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, belopolsky, christian.heimes, cvs-src, koobs, meador.inge, python-dev
Priority: normal Keywords:

Created on 2013-06-10 08:11 by cvs-src, last changed 2013-08-18 13:14 by christian.heimes. This issue is now closed.

File name Uploaded Description Edit cvs-src, 2013-06-10 08:11
Messages (7)
msg190892 - (view) Author: rm (cvs-src) Date: 2013-06-10 08:11

Marcel Moolenaar ( pointed this out after committing FreeBSD revision 250991 [1], that makes the malloc(3) family of functions weak symbols. I'm citing him, because (silly me) I don't understand all of this completely:

After my commit to head that makes the malloc(3) family of functions weak symbols, a bug in what appears to be the Python build was exposed. The net effect of the bug is that contains (strong) definitions of the malloc(3) family of functions and resulting in unintended symbol resolution. incorporates the libffi functionality for what I presume is
the basis for Python bindings. libffi includes dlmalloc.c, an open source allocator. dlmalloc.c is incuded by closures.c and closures.c defines USE_DL_PREFIX. On top of that closures.c makes all allocator functions static. This, therefore adds a memory allocator to libffi without exposure of standard symbols In short: dlmalloc.c never gets compiler separately or independently for this reason.

The python build however compiles dlmalloc.c separately/independently. As such, dlmalloc.c now defines and exports the malloc(3) family of functions and it also get linked into Thus when libffi is built as part of the Python build, the set of symbols exported is different from when libffi is compiled as a port.

The simplest test case is this (on a -current machines):

1.  Build & install lang/python27 (unmodified; if needed)
2.  try and build databases/py-sqlite3

The build of databases/py-sqlite3 fails because this:
running config
*** Signal 11

On amd64 I observed an assertion failure of FreeBSD's malloc, with
the same effect.

Manually triggering this:

fbsdvm% pwd

fbsdvm% /usr/local/bin/python2.7 config
running config
Segmentation fault

But if symbols are resolved non-lazily, things change:

fbsdvm% setenv LD_BIND_NOW yes
fbsdvm% /usr/local/bin/python2.7 config
running config

This demonstrates how, after loading, malloc(3) and friends
get resolved differently and thus inconsistently from before.

The publicly visible symbols in, also show the problem:

fbsdvm% nm /usr/local/lib/python2.7/lib-dynload/ | grep ' T ' | egrep '(alloc)|(free)'
0000c3f0 T _ctypes_alloc_callback
00005250 T _ctypes_alloc_format_string
00016d10 T calloc
000121e0 T ffi_closure_alloc
00013760 T ffi_closure_free
00016050 T free
000170c0 T independent_calloc
000172d0 T independent_comalloc
000148e0 T malloc
00017460 T malloc_footprint
00017480 T malloc_max_footprint
000175c0 T malloc_stats
00017440 T malloc_trim
000176e0 T malloc_usable_size
000173a0 T pvalloc
00016d90 T realloc
00017310 T valloc

There are definitions of malloc(3) and friends that shouldn't be there.

We have similar reports in our bug-tracker [2] and [3]. And the patch attached fixes this for post-r250991 versions of FreeBSD, and noop for earlier versions. The same patch applied to python2.7, python3.2, python3.3 and patched in FreeBSD ports tree locally.

msg190897 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-10 10:13
Said simpler: dlmalloc.c code is indeed compiled twice:
- once as part of closures.c, which #include "dlmalloc.c"; this is done carefully (with 'static' and 'USE_DL_PREFIX') to not clash with standard malloc functions.
- once as a separate target; without USE_DL_PREFIX it will define functions named malloc() and free()...

The second one is not necessary, of course. The patch looks good to me.
msg190926 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2013-06-10 18:04
I don't have a system affected by the change, but the explanation and the patch look right to me.

FWIW, the patch applies cleanly to 3.4 head and passes 'make test'.
msg192068 - (view) Author: koobs (koobs) Date: 2013-06-30 13:22
I've added a new FreeBSD 10.0-CURRENT buildbot to the pool (thanks antoine) that reproduces the issue and should provide sufficient coverage for testing the proposed patch:

I'll upgrade the FreeBSD 9-STABLE buildbot once this is resolved
msg195462 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-17 13:03
New changeset f09ca52747a6 by Christian Heimes in branch '3.3':
Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.

New changeset bea2f12e899e by Christian Heimes in branch 'default':
Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.

New changeset d4ac6eee7061 by Christian Heimes in branch '2.7':
Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
msg195493 - (view) Author: koobs (koobs) Date: 2013-08-17 16:29
Commit looks good, confirming test suite passing for 3.x, 3.3 and 2.7.on

Thank you for picking this up and finishing it off Christian.
msg195557 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-08-18 13:14
Thx for the fix!
Date User Action Args
2013-08-18 13:14:26christian.heimessetstatus: open -> closed

nosy: + christian.heimes
messages: + msg195557

resolution: fixed
stage: commit review -> resolved
2013-08-17 16:29:23koobssetmessages: + msg195493
2013-08-17 13:03:10python-devsetnosy: + python-dev
messages: + msg195462
2013-06-30 13:22:47koobssetnosy: + koobs
messages: + msg192068
2013-06-10 18:04:31belopolskysetstage: commit review
messages: + msg190926
versions: + Python 3.3, Python 3.4
2013-06-10 10:13:25amaury.forgeotdarcsetmessages: + msg190897
2013-06-10 09:07:08serhiy.storchakasetnosy: + amaury.forgeotdarc, belopolsky, meador.inge
2013-06-10 08:11:18cvs-srccreate