classification
Title: multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Type: crash Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: hongxu
Priority: normal Keywords:

Created on 2017-08-10 07:21 by hongxu, last changed 2017-08-10 07:37 by hongxu.

Pull Requests
URL Status Linked Edit
PR 3054 open hongxu, 2017-08-10 07:37
Messages (3)
msg300052 - (view) Author: Hongxu Jia (hongxu) * Date: 2017-08-10 07:21
To build python for embedded Linux systems,
(http://www.yoctoproject.org/docs/2.3.1/yocto-project-qs/yocto-project-qs.html)
The 32-bit python's multiprocessing.BoundedSemaphore could not work.

1. Prerequisite
- Build 32bit python on 64bit host, add '-m32' to gcc

- Cross compling on linux platform, not have `-pthread'
  In configure.ac, ac_cv_pthread=no while cross compiling
  ...
  2001 AC_MSG_CHECKING(whether $CC accepts -pthread)
  2002 AC_CACHE_VAL(ac_cv_pthread,
  2003 [ac_save_cc="$CC"
  2004 CC="$CC -pthread"
  2005 AC_RUN_IFELSE([AC_LANG_SOURCE([[
  [snip]
  2018 ]])],[ac_cv_pthread=yes],[ac_cv_pthread=no],[ac_cv_pthread=no])
  ...

  — Macro: AC_RUN_IFELSE (input, [action-if-true], [action-if-false], [action-if-cross-compiling])

2. Reproduce with simplifying the prerequisite

$ git clone https://github.com/python/cpython.git; cd ./cpython

$ ac_cv_pthread=no CC="gcc -m32" ./configure

$ make -j 10

$ ./python -c "import multiprocessing; pool_sema = multiprocessing.BoundedSemaphore(value=1); pool_sema.acquire(); pool_sema.release()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: semaphore or lock released too many times

And the issue also caused invoking put of 'multiprocessing.Queue' hung.
$ ./python -c "import multiprocessing; queue_instance = multiprocessing.Queue(); queue_instance.put(('install'))"
msg300053 - (view) Author: Hongxu Jia (hongxu) * Date: 2017-08-10 07:22
3. Analysis
1) The multiprocessing invokes named semaphore in C library
   (sem_open/sem_post/sem_getvalue/sem_unlink in
   ./Modules/_multiprocessing/semaphore.c)

2) The glibc defines two different sem_getvalue since the following commit.
https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/sem_getvalue.c;h=1432cc795ece84d5bf31c7e5cafe01cc1a09d98d;hb=042e1521c794a945edc43b5bfa7e69ad70420524

The `__new_sem_getvalue' is the sem_getvalue@@GLIBC_2.1
which work with named semaphore `sem_open'

The `__old_sem_getvalue' is the old version sem_getvalue@GLIBC_2.0
which work with unamed semaphore `sem_init'

In 32-bit C library, it provides both of them.
$ nm -g /lib/i386-linux-gnu/libpthread-2.23.so
0000df30 T sem_getvalue@GLIBC_2.0
0000df10 T sem_getvalue@@GLIBC_2.1

3) In multiprocessing, it invokes sem_open, so sem_getvalue@@GLIBC_2.1
is supposed.

If `-pthread' or `-lpthread' not passed to gcc, the compiled
_multiprocessing dynamic library could not explicitly linked to
sem_getvalue@@GLIBC_2.1
$ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so
         U sem_getvalue
         U sem_open

If `-pthread' or `-lpthread' passed to gcc:
$ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so
         U sem_getvalue@@GLIBC_2.1
         U sem_open@@GLIBC_2.1.1

4) On 32-bit OS, the multiprocessing was incorrectly linked to
sem_getvalue@GLIBC_2.0 which caused the issue.
msg300054 - (view) Author: Hongxu Jia (hongxu) * Date: 2017-08-10 07:23
4. Solution

For cross compiling, there is no `-pthread', so we should explicitly add
`-lpthread' to build multiprocessing.

Peterson tried to do it in the following commit:
...
commit e711cafab13efc9c1fe6c5cd75826401445eb585
Author: Benjamin Peterson <benjamin@python.org>
Date:   Wed Jun 11 16:44:04 2008 +0000

    Merged revisions 64104,64117 via svnmerge from
    svn+ssh://pythondev@svn.python.org/python/trunk
...
git show e711cafab13efc9c1fe6c5cd75826401445eb585 -- setup.py

--- a/setup.py
+++ b/setup.py
@@ -1110,6 +1110,56 @@ class PyBuildExt(build_ext):
 
         # _fileio -- supposedly cross platform
         exts.append(Extension('_fileio', ['_fileio.c']))
+        # Richard Oudkerk's multiprocessing module
+        if platform == 'win32':             # Windows
+            macros = dict()
+            libraries = ['ws2_32']
+
+        elif platform == 'darwin':          # Mac OSX
+            macros = dict(
+                HAVE_SEM_OPEN=1,
+                HAVE_SEM_TIMEDWAIT=0,
+                HAVE_FD_TRANSFER=1,
+                HAVE_BROKEN_SEM_GETVALUE=1
+                )
+            libraries = []
+
+        elif platform == 'cygwin':          # Cygwin
+            macros = dict(
+                HAVE_SEM_OPEN=1,
+                HAVE_SEM_TIMEDWAIT=1,
+                HAVE_FD_TRANSFER=0,
+                HAVE_BROKEN_SEM_UNLINK=1
+                )
+            libraries = []
+        else:                                   # Linux and other unices
+            macros = dict(
+                HAVE_SEM_OPEN=1,
+                HAVE_SEM_TIMEDWAIT=1,
+                HAVE_FD_TRANSFER=1
+                )
+            libraries = ['rt']
+
+        if platform == 'win32':
+            multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+                                     '_multiprocessing/semaphore.c',
+                                     '_multiprocessing/pipe_connection.c',
+                                     '_multiprocessing/socket_connection.c',
+                                     '_multiprocessing/win32_functions.c'
+                                   ]
+
+        else:
+            multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+                                     '_multiprocessing/socket_connection.c'
+                                   ]
+
+            if macros.get('HAVE_SEM_OPEN', False):
+                multiprocessing_srcs.append('_multiprocessing/semaphore.c')
+
+        exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+                                 define_macros=list(macros.items()),
+                                 include_dirs=["Modules/_multiprocessing"]))
+        # End multiprocessing

It defined variable `libraries' and assigned it according to
host_platform, but forgot to pass it to Extension.

So we should correct it, and add `-lpthread' for linux.
History
Date User Action Args
2017-08-10 07:37:01hongxusetpull_requests: + pull_request3087
2017-08-10 07:23:39hongxusetmessages: + msg300054
2017-08-10 07:22:33hongxusetmessages: + msg300053
2017-08-10 07:21:56hongxucreate