classification
Title: _decimal on Android requires linking with libm
Type: behavior Stage: resolved
Components: Cross-Build Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: skrah Nosy List: Alex.Willmer, python-dev, skrah, xdegaye, yan12125
Priority: normal Keywords: patch

Created on 2017-02-04 05:07 by yan12125, last changed 2017-02-07 14:02 by yan12125. This issue is now closed.

Files
File name Uploaded Description Edit
decimal.patch yan12125, 2017-02-04 05:07
Messages (10)
msg286906 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2017-02-04 05:07
Just like issue21668, _decimal requires -lm on Android. This wasn't fixed because _decimal didn't build before issue26846 lands. More specificially, log10 is called https://hg.python.org/cpython/file/tip/Modules/_decimal/libmpdec/mpdecimal.c#l7862

Added _decimal and Android experts
msg286960 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-02-04 13:41
Thanks.  Strange that on other systems the compilers don't complain (usually they do).
msg286961 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-04 13:59
New changeset b60b46ad8751 by Stefan Krah in branch '3.6':
Issue29439: _decimal on Android requires linking with libm.
https://hg.python.org/cpython/rev/b60b46ad8751
msg286962 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-04 14:00
New changeset aa26a5712a80849e968171b71b6bc8d9da3ac163 by Stefan Krah in branch '3.6':
Issue29439: _decimal on Android requires linking with libm.
https://github.com/python/cpython/commit/aa26a5712a80849e968171b71b6bc8d9da3ac163
msg286963 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-04 14:00
New changeset aa26a5712a80849e968171b71b6bc8d9da3ac163 by Stefan Krah in branch 'master':
Issue29439: _decimal on Android requires linking with libm.
https://github.com/python/cpython/commit/aa26a5712a80849e968171b71b6bc8d9da3ac163
msg286975 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-02-04 16:42
> This wasn't fixed because _decimal didn't build before issue26846 lands.

This is misleading, the problem went unnoticed when not linking with '-lm' because:
* The _decimal extension module builds without any warning.
* test_decimal does not fail.

One way to demonstrate the problem in the interactive interpreter on Android:
>>> import decimal
>>> import _decimal
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen failed: cannot locate symbol "log10" referenced by "_decimal.cpython-37dm.so"...
msg286978 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-02-04 17:10
Perhaps test_decimal should fail for CPython if _decimal can't be imported.

All compilers that I tested seem to link fine without -lm in this case.
msg287167 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-02-06 19:41
With the following module named dlsym.py, the command 'python dlsym.py log10' produces the same successful output on linux and on Android API 21 (when _decimal is not explicitly linked with libm.so, i.e.without changeset b60b46ad8751):
  'The symbol "log10" is resolved.'

----------------------     dlsym.py     ---------------------------
import sys, os
from ctypes import *

if len(sys.argv) != 2:
    sys.exit('Error: the symbol name is required.')
symbol = sys.argv[1]
libdl = CDLL('libdl.so')
libdl.dlopen.restype = c_void_p
libdl.dlsym.restype = c_void_p
hdle = libdl.dlopen(None, os.RTLD_NOW | RTLD_GLOBAL)
if hdle is None:
    print('Cannot get a handle on the global symbol object.')
else:
    v = libdl.dlsym(c_void_p(hdle), symbol.encode())
    _not = 'not ' if v is None else ''
    print('The symbol "%s" is %sresolved.' % (symbol, _not))
-------------------------------------------------------------------

This is as expected since the python executable is linked with the math library. However on Android API 21 importing _decimal fails with the error message reported earlier:
  'dlopen failed: cannot locate symbol "log10"'.

So this seems to be a bug with Android dlopen() at API 21 since the 'log10' symbol does exist in the process image relocation tables as shown by dlsym.py. This is confirmed by the fact that the import does not fail anymore at API 24 (still without changeset b60b46ad8751). This change of behavior may be a side effect of the changes reported in this bionic (Android libc) changelog entry:
    https://android.googlesource.com/platform/bionic/+show/refs/heads/master/android-changes-for-ndk-developers.md#36

For completness, my (possibly wrong) interpretation of why changeset b60b46ad8751 fixes the problem at API 21:
Changeset b60b46ad8751 adds the '-lm' command line option to the linker and this adds [1] a new DT_NEEDED entry to the .dynamic section of the shared library ELF file as shown by the command 'readelf -d build/lib.linux-x86_64-3.7-pydebug/_decimal.cpython-37dm-x86_64-linux-gnu.so':

Dynamic section at offset 0x54d30 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 ...

The Android loader lookup for 'log10' is now successful at API 21 and the symbol is found in the libm.so.6 library listed in the DT_NEEDED lists of needed libraries.

[1] See option '--as-needed' in the 'ld' man pages or see the ELF specification. Note that the Android toolchains use the clang compiler but continue to use the GNU binutils tools including the linker 'ld'.
msg287168 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-02-06 20:00
> Perhaps test_decimal should fail for CPython if _decimal can't be imported.
This is not a problem specific to _decimal, the same problem exists for third-party extension modules, for example pyephem (https://github.com/brandon-rhodes/pyephem/issues/114) and existed with some other Python extension modules (issue 21668).
msg287233 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2017-02-07 14:02
Actually my device is 6.0. Seems there's nothing interesting between API 23 and 24 on android-changes-for-ndk-developers.md :)

Anyway, _decimal should not depend on the whether the python binary references to libm.so or not. Instead, _decimal should explicitly have an explicit DT_NEEDED entry to libm.so if it uses symbols from libm.

> Strange that on other systems the compilers don't complain (usually they do).

I guess on most system python is built with --enable-shared, so _decimal.so has an entry to libpython3.7m.so, and the latter references libm.so. If a linker supports recursive search, it can find the symbol.

For future references, here's the partial output of logcat when I run python with `LD_DEBUG=3 python3.7m` and import _decimal:

02-07 19:43:25.697  10266    10266                 linker  D  DEBUG: /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: looking up log10 in python3.7m (from global group)
02-07 19:43:25.697  10266    10266                 linker  I  SEARCH log10 in python3.7m@0x55634f3000 h=735a40(elf) 452
02-07 19:43:25.697  10266    10266                 linker  I  NOT FOUND log10 in python3.7m@0x55634f3000 735a40 452
02-07 19:43:25.698  10266    10266                 linker  D  DEBUG: /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: looking up log10 in /system/vendor/lib64/libNimsWrap.so (from global group)
02-07 19:43:25.698  10266    10266                 linker  I  SEARCH log10 in /system/vendor/lib64/libNimsWrap.so@0x7f7c5fb000 (gnu)
02-07 19:43:25.698  10266    10266                 linker  I  NOT FOUND log10 in /system/vendor/lib64/libNimsWrap.so@0x7f7c5fb000
02-07 19:43:25.698  10266    10266                 linker  D  DEBUG: /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: looking up log10 in /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so (from local group)
02-07 19:43:25.698  10266    10266                 linker  I  SEARCH log10 in /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so@0x7f7c157000 h=735a40(elf) 49
02-07 19:43:25.698  10266    10266                 linker  I  NOT FOUND log10 in /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so@0x7f7c157000 735a40 49
02-07 19:43:25.698  10266    10266                 linker  D  DEBUG: /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: looking up log10 in libdl.so (from local group)
02-07 19:43:25.698  10266    10266                 linker  I  SEARCH log10 in libdl.so@0x0 h=735a40(elf) 0
02-07 19:43:25.698  10266    10266                 linker  I  NOT FOUND log10 in libdl.so@0x0 735a40 0
02-07 19:43:25.698  10266    10266                 linker  D  DEBUG: /data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so: looking up log10 in /system/lib64/libc.so (from local group)
02-07 19:43:25.698  10266    10266                 linker  I  SEARCH log10 in /system/lib64/libc.so@0x7f7c4f6000 (gnu)
02-07 19:43:25.698  10266    10266                 linker  I  NOT FOUND log10 in /system/lib64/libc.so@0x7f7c4f6000
02-07 19:43:25.698  10266    10266                 linker  D  DEBUG: cannot locate symbol "log10" referenced by "/data/local/tmp/python3/usr/lib/python3.7/lib-dynload/_decimal.cpython-37m.so"...

And BioniC's linker only checks HASH table of ELF: https://android.googlesource.com/platform/bionic/+/master/linker/linker_soinfo.cpp#277, so log10 is not found in python3.7m

> This is misleading

Maybe it is. I mean "I didn't write a patch for _decimal at issue21668 as _decimal didn't build then"

> Perhaps test_decimal should fail for CPython if _decimal can't be imported.

I create a simple script to ensure all installed dynamic modules can be imported: https://github.com/yan12125/python3-android/blob/master/devscripts/import_all.py. It may be worth to transform it into a proper unittest and put it into Lib/test/
History
Date User Action Args
2017-02-07 14:02:42yan12125setmessages: + msg287233
2017-02-06 20:00:20xdegayesetmessages: + msg287168
2017-02-06 19:41:23xdegayesetmessages: + msg287167
2017-02-04 17:10:26skrahsetmessages: + msg286978
2017-02-04 16:42:22xdegayesetmessages: + msg286975
2017-02-04 14:04:35skrahsetstatus: open -> closed
stage: resolved
resolution: fixed
versions: + Python 3.6
2017-02-04 14:00:27python-devsetmessages: + msg286963
2017-02-04 14:00:25python-devsetmessages: + msg286962
2017-02-04 13:59:42python-devsetnosy: + python-dev
messages: + msg286961
2017-02-04 13:41:00skrahsetassignee: skrah
messages: + msg286960
2017-02-04 05:07:30yan12125create