classification
Title: Redefinition of malloc(3) family of functions at build time
Type: crash Stage: resolved
Components: ctypes Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, belopolsky, christian.heimes, cvs-src, koobs, meador.inge, python-dev
Priority: normal Keywords:

Created on 2013-06-10 08:11 by cvs-src, last changed 2013-08-18 13:14 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
patch-Modules-_ctypes-libffi_fficonfig.py.in cvs-src, 2013-06-10 08:11
Messages (7)
msg190892 - (view) Author: rm (cvs-src) Date: 2013-06-10 08:11
Hello.

Marcel Moolenaar (marcel@FreeBSD.org) pointed this out after committing FreeBSD revision 250991 [1], that makes the malloc(3) family of functions weak symbols. I'm citing him, because (silly me) I don't understand all of this completely:

"""
After my commit to head that makes the malloc(3) family of functions weak symbols, a bug in what appears to be the Python build was exposed. The net effect of the bug is that _ctypes.so contains (strong) definitions of the malloc(3) family of functions and resulting in unintended symbol resolution.

_ctypes.so incorporates the libffi functionality for what I presume is
the basis for Python bindings. libffi includes dlmalloc.c, an open source allocator. dlmalloc.c is incuded by closures.c and closures.c defines USE_DL_PREFIX. On top of that closures.c makes all allocator functions static. This, therefore adds a memory allocator to libffi without exposure of standard symbols In short: dlmalloc.c never gets compiler separately or independently for this reason.

The python build however compiles dlmalloc.c separately/independently. As such, dlmalloc.c now defines and exports the malloc(3) family of functions and it also get linked into _ctypes.so. Thus when libffi is built as part of the Python build, the set of symbols exported is different from when libffi is compiled as a port.


The simplest test case is this (on a -current machines):

1.  Build & install lang/python27 (unmodified; if needed)
2.  try and build databases/py-sqlite3

The build of databases/py-sqlite3 fails because this:
running config
*** Signal 11

On amd64 I observed an assertion failure of FreeBSD's malloc, with
the same effect.


Manually triggering this:

fbsdvm% pwd
/usr/ports/databases/py-sqlite3/work/Python-2.7.5/Modules

fbsdvm% /usr/local/bin/python2.7 setup.py config
running config
Segmentation fault

But if symbols are resolved non-lazily, things change:

fbsdvm% setenv LD_BIND_NOW yes
fbsdvm% /usr/local/bin/python2.7 setup.py config
running config

This demonstrates how, after loading _ctypes.so, malloc(3) and friends
get resolved differently and thus inconsistently from before.

The publicly visible symbols in _ctypes.so, also show the problem:

fbsdvm% nm /usr/local/lib/python2.7/lib-dynload/_ctypes.so | grep ' T ' | egrep '(alloc)|(free)'
0000c3f0 T _ctypes_alloc_callback
00005250 T _ctypes_alloc_format_string
00016d10 T calloc
000121e0 T ffi_closure_alloc
00013760 T ffi_closure_free
00016050 T free
000170c0 T independent_calloc
000172d0 T independent_comalloc
000148e0 T malloc
00017460 T malloc_footprint
00017480 T malloc_max_footprint
000175c0 T malloc_stats
00017440 T malloc_trim
000176e0 T malloc_usable_size
000173a0 T pvalloc
00016d90 T realloc
00017310 T valloc

There are definitions of malloc(3) and friends that shouldn't be there.
"""

We have similar reports in our bug-tracker [2] and [3]. And the patch attached fixes this for post-r250991 versions of FreeBSD, and noop for earlier versions. The same patch applied to python2.7, python3.2, python3.3 and patched in FreeBSD ports tree locally.

[1] http://svnweb.freebsd.org/changeset/base/250991
[2] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179102
[3] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/179413
msg190897 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-10 10:13
Said simpler: dlmalloc.c code is indeed compiled twice:
- once as part of closures.c, which #include "dlmalloc.c"; this is done carefully (with 'static' and 'USE_DL_PREFIX') to not clash with standard malloc functions.
- once as a separate target; without USE_DL_PREFIX it will define functions named malloc() and free()...

The second one is not necessary, of course. The patch looks good to me.
msg190926 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2013-06-10 18:04
I don't have a system affected by the change, but the explanation and the patch look right to me.

FWIW, the patch applies cleanly to 3.4 head and passes 'make test'.
msg192068 - (view) Author: koobs (koobs) Date: 2013-06-30 13:22
I've added a new FreeBSD 10.0-CURRENT buildbot to the pool (thanks antoine) that reproduces the issue and should provide sufficient coverage for testing the proposed patch:

http://buildbot.python.org/all/buildslaves/koobs-freebsd10

I'll upgrade the FreeBSD 9-STABLE buildbot once this is resolved
msg195462 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-17 13:03
New changeset f09ca52747a6 by Christian Heimes in branch '3.3':
Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
http://hg.python.org/cpython/rev/f09ca52747a6

New changeset bea2f12e899e by Christian Heimes in branch 'default':
Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
http://hg.python.org/cpython/rev/bea2f12e899e

New changeset d4ac6eee7061 by Christian Heimes in branch '2.7':
Issue #18178: Fix ctypes on BSD. dlmalloc.c was compiled twice which broke malloc weak symbols.
http://hg.python.org/cpython/rev/d4ac6eee7061
msg195493 - (view) Author: koobs (koobs) Date: 2013-08-17 16:29
Commit looks good, confirming test suite passing for 3.x, 3.3 and 2.7.on http://buildbot.python.org/all/buildslaves/koobs-freebsd10

Thank you for picking this up and finishing it off Christian.
msg195557 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-08-18 13:14
Thx for the fix!
History
Date User Action Args
2013-08-18 13:14:26christian.heimessetstatus: open -> closed

nosy: + christian.heimes
messages: + msg195557

resolution: fixed
stage: commit review -> resolved
2013-08-17 16:29:23koobssetmessages: + msg195493
2013-08-17 13:03:10python-devsetnosy: + python-dev
messages: + msg195462
2013-06-30 13:22:47koobssetnosy: + koobs
messages: + msg192068
2013-06-10 18:04:31belopolskysetstage: commit review
messages: + msg190926
versions: + Python 3.3, Python 3.4
2013-06-10 10:13:25amaury.forgeotdarcsetmessages: + msg190897
2013-06-10 09:07:08serhiy.storchakasetnosy: + amaury.forgeotdarc, belopolsky, meador.inge
2013-06-10 08:11:18cvs-srccreate