classification
Title: multiprocessing.BoundedSemaphore of 32-bit python could not work while cross compiling on linux platform
Type: crash Stage: patch review
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Barry Davis, benjamin.peterson, hongxu, pitrou
Priority: normal Keywords: patch

Created on 2017-08-10 07:21 by hongxu, last changed 2019-05-16 03:15 by hongxu.

Pull Requests
URL Status Linked Edit
PR 10563 open hongxu, 2018-11-16 07:31
PR 3054 hongxu, 2018-11-16 07:39
PR 13353 open hongxu, 2019-05-16 03:15
Messages (5)
msg300052 - (view) Author: Hongxu Jia (hongxu) * Date: 2017-08-10 07:21
To build python for embedded Linux systems,
(http://www.yoctoproject.org/docs/2.3.1/yocto-project-qs/yocto-project-qs.html)
The 32-bit python's multiprocessing.BoundedSemaphore could not work.

1. Prerequisite
- Build 32bit python on 64bit host, add '-m32' to gcc

- Cross compling on linux platform, not have `-pthread'
  In configure.ac, ac_cv_pthread=no while cross compiling
  ...
  2001 AC_MSG_CHECKING(whether $CC accepts -pthread)
  2002 AC_CACHE_VAL(ac_cv_pthread,
  2003 [ac_save_cc="$CC"
  2004 CC="$CC -pthread"
  2005 AC_RUN_IFELSE([AC_LANG_SOURCE([[
  [snip]
  2018 ]])],[ac_cv_pthread=yes],[ac_cv_pthread=no],[ac_cv_pthread=no])
  ...

  — Macro: AC_RUN_IFELSE (input, [action-if-true], [action-if-false], [action-if-cross-compiling])

2. Reproduce with simplifying the prerequisite

$ git clone https://github.com/python/cpython.git; cd ./cpython

$ ac_cv_pthread=no CC="gcc -m32" ./configure

$ make -j 10

$ ./python -c "import multiprocessing; pool_sema = multiprocessing.BoundedSemaphore(value=1); pool_sema.acquire(); pool_sema.release()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: semaphore or lock released too many times

And the issue also caused invoking put of 'multiprocessing.Queue' hung.
$ ./python -c "import multiprocessing; queue_instance = multiprocessing.Queue(); queue_instance.put(('install'))"
msg300053 - (view) Author: Hongxu Jia (hongxu) * Date: 2017-08-10 07:22
3. Analysis
1) The multiprocessing invokes named semaphore in C library
   (sem_open/sem_post/sem_getvalue/sem_unlink in
   ./Modules/_multiprocessing/semaphore.c)

2) The glibc defines two different sem_getvalue since the following commit.
https://sourceware.org/git/?p=glibc.git;a=blob;f=nptl/sem_getvalue.c;h=1432cc795ece84d5bf31c7e5cafe01cc1a09d98d;hb=042e1521c794a945edc43b5bfa7e69ad70420524

The `__new_sem_getvalue' is the sem_getvalue@@GLIBC_2.1
which work with named semaphore `sem_open'

The `__old_sem_getvalue' is the old version sem_getvalue@GLIBC_2.0
which work with unamed semaphore `sem_init'

In 32-bit C library, it provides both of them.
$ nm -g /lib/i386-linux-gnu/libpthread-2.23.so
0000df30 T sem_getvalue@GLIBC_2.0
0000df10 T sem_getvalue@@GLIBC_2.1

3) In multiprocessing, it invokes sem_open, so sem_getvalue@@GLIBC_2.1
is supposed.

If `-pthread' or `-lpthread' not passed to gcc, the compiled
_multiprocessing dynamic library could not explicitly linked to
sem_getvalue@@GLIBC_2.1
$ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so
         U sem_getvalue
         U sem_open

If `-pthread' or `-lpthread' passed to gcc:
$ nm -g ./build/lib.linux-x86_64-3.7/_multiprocessing.cpython-37m-i386-linux-gnu.so
         U sem_getvalue@@GLIBC_2.1
         U sem_open@@GLIBC_2.1.1

4) On 32-bit OS, the multiprocessing was incorrectly linked to
sem_getvalue@GLIBC_2.0 which caused the issue.
msg300054 - (view) Author: Hongxu Jia (hongxu) * Date: 2017-08-10 07:23
4. Solution

For cross compiling, there is no `-pthread', so we should explicitly add
`-lpthread' to build multiprocessing.

Peterson tried to do it in the following commit:
...
commit e711cafab13efc9c1fe6c5cd75826401445eb585
Author: Benjamin Peterson <benjamin@python.org>
Date:   Wed Jun 11 16:44:04 2008 +0000

    Merged revisions 64104,64117 via svnmerge from
    svn+ssh://pythondev@svn.python.org/python/trunk
...
git show e711cafab13efc9c1fe6c5cd75826401445eb585 -- setup.py

--- a/setup.py
+++ b/setup.py
@@ -1110,6 +1110,56 @@ class PyBuildExt(build_ext):
 
         # _fileio -- supposedly cross platform
         exts.append(Extension('_fileio', ['_fileio.c']))
+        # Richard Oudkerk's multiprocessing module
+        if platform == 'win32':             # Windows
+            macros = dict()
+            libraries = ['ws2_32']
+
+        elif platform == 'darwin':          # Mac OSX
+            macros = dict(
+                HAVE_SEM_OPEN=1,
+                HAVE_SEM_TIMEDWAIT=0,
+                HAVE_FD_TRANSFER=1,
+                HAVE_BROKEN_SEM_GETVALUE=1
+                )
+            libraries = []
+
+        elif platform == 'cygwin':          # Cygwin
+            macros = dict(
+                HAVE_SEM_OPEN=1,
+                HAVE_SEM_TIMEDWAIT=1,
+                HAVE_FD_TRANSFER=0,
+                HAVE_BROKEN_SEM_UNLINK=1
+                )
+            libraries = []
+        else:                                   # Linux and other unices
+            macros = dict(
+                HAVE_SEM_OPEN=1,
+                HAVE_SEM_TIMEDWAIT=1,
+                HAVE_FD_TRANSFER=1
+                )
+            libraries = ['rt']
+
+        if platform == 'win32':
+            multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+                                     '_multiprocessing/semaphore.c',
+                                     '_multiprocessing/pipe_connection.c',
+                                     '_multiprocessing/socket_connection.c',
+                                     '_multiprocessing/win32_functions.c'
+                                   ]
+
+        else:
+            multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
+                                     '_multiprocessing/socket_connection.c'
+                                   ]
+
+            if macros.get('HAVE_SEM_OPEN', False):
+                multiprocessing_srcs.append('_multiprocessing/semaphore.c')
+
+        exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+                                 define_macros=list(macros.items()),
+                                 include_dirs=["Modules/_multiprocessing"]))
+        # End multiprocessing

It defined variable `libraries' and assigned it according to
host_platform, but forgot to pass it to Extension.

So we should correct it, and add `-lpthread' for linux.
msg334351 - (view) Author: Barry Davis (Barry Davis) Date: 2019-01-25 10:50
I've just hit this issue using Python-2.7.9, gcc-8.1.0, glibc-2.23.

The patch I made to fix the issue based on comments in this issue:

--- Python-2.7.9/setup.py       2019-01-25 09:30:39.049501423 +0000
+++ Python-2.7.9/setup.py       2019-01-25 09:30:55.369609646 +0000
@@ -1670,7 +1670,7 @@

         else:                                   # Linux and other unices
             macros = dict()
-            libraries = ['rt']
+            libraries = ['rt', 'pthread']

         if host_platform == 'win32':
             multiprocessing_srcs = [ '_multiprocessing/multiprocessing.c',
@@ -1690,6 +1690,7 @@

         if sysconfig.get_config_var('WITH_THREAD'):
             exts.append ( Extension('_multiprocessing', multiprocessing_srcs,
+                                    libraries = libraries,
                                     define_macros=macros.items(),
                                     include_dirs=["Modules/_multiprocessing"]))
         else:
msg334352 - (view) Author: Barry Davis (Barry Davis) Date: 2019-01-25 10:58
The behaviour I saw (32-bit only) was a python process getting stuck.

I got this from strace:
...
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
futex(0xb5acc000, FUTEX_WAIT, 0, NULL)  = -1 EAGAIN (Resource temporarily unavailable)
...

And this from gdb:
(gdb) bt
#0  0xb76fbc5d in __kernel_vsyscall ()
#1  0xb74af2a4 in sem_wait () from /lib/libpthread.so.0
#2  0xb69a5429 in ?? () from /usr/lib/python2.7/lib-dynload/_multiprocessing.so
#3  0xb75a262e in call_function (oparg=<optimized out>, pp_stack=0xbfb7f038) at Python/ceval.c:4033
#4  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#5  0xb75a2f7d in PyEval_EvalCodeEx (co=<optimized out>, globals=
    {'Queue': <type at remote 0xb69c43dc>, 'atexit': <module at remote 0xb6d369bc>, 'Semaphore': <type at remote 0xb69b978c>, 'Empty': <type at remote 0xb69b902c>, 'Full': <type at remote 0xb69b9204>, 'SimpleQueue': <type at remote 0xb69c478c>, 'assert_spawning': <function at remote 0xb69ba684>, '__all__': ['Queue', 'SimpleQueue', 'JoinableQueue'], '_multiprocessing': <module at remote 0xb6c16944>, '_sentinel': <object at remote 0xb726f4d0>, '__package__': 'multiprocessing', 'collections': <module at remote 0xb71bce3c>, '__doc__': None, 'Condition': <type at remote 0xb69c402c>, 'JoinableQueue': <type at remote 0xb69c45b4>, '__builtins__': {'bytearray': <type at remote 0xb76c27e0>, 'IndexError': <type at remote 0xb76c6900>, 'all': <built-in function all>, 'help': <_Helper at remote 0xb71f824c>, 'vars': <built-in function vars>, 'SyntaxError': <type at remote 0xb76c6c80>, 'unicode': <type at remote 0xb76d6c00>, 'UnicodeDecodeError': <type at remote 0xb76c6420>, 'memoryview': <type at remote 0xb76cdda0>, 'isinstance...(truncated), locals=0x0, args=0xb535f5cc, argcount=2, kws=0xb535f5d4, kwcount=0, defs=0xb69b6058, defcount=2, closure=0x0) at Python/ceval.c:3265
#6  0xb75a0f3d in fast_function (nk=0, na=<optimized out>, n=2, pp_stack=0xbfb7f178, func=<function at remote 0xb69c51b4>) at Python/ceval.c:4129
#7  call_function (oparg=<optimized out>, pp_stack=0xbfb7f178) at Python/ceval.c:4054
#8  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
...
(gdb) py-bt
#4 Frame 0xb536602c, for file /usr/lib/python2.7/multiprocessing/queues.py, line 101, in put (self=<Queue(_writer=<_multiprocessing.Connection at remote 0x96878c8>, _recv=<built-in method recv of _multiprocessing.Connection object at remote 0x967ddc8>, _thread=None, _sem=<BoundedSemaphore(release=<built-in method release of _multiprocessing.SemLock object at remote 0xb53427c0>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0xb53427c0>, _semlock=<_multiprocessing.SemLock at remote 0xb53427c0>) at remote 0xb53425cc>, _buffer=<collections.deque at remote 0xb533f844>, _closed=False, _send=<built-in method send of _multiprocessing.Connection object at remote 0x96878c8>, _jointhread=None, _reader=None, _opid=3752, _rlock=<Lock(release=<built-in method release of _multiprocessing.SemLock object at remote 0xb5342420>, acquire=<built-in method acquire of _multiprocessing.SemLock object at remote 0xb5342420>, _semlock=<_multiprocessing.SemLock at remote 0xb5342420>) at remote 0xb534240c>, _...(truncated)
...
History
Date User Action Args
2019-05-16 03:15:00hongxusetpull_requests: + pull_request13264
2019-01-25 10:58:47Barry Davissetmessages: + msg334352
2019-01-25 10:50:04Barry Davissetnosy: + Barry Davis
messages: + msg334351
2018-11-26 21:18:03pablogsalsetnosy: + pitrou, benjamin.peterson
2018-11-16 07:39:25hongxusetpull_requests: + pull_request9812
2018-11-16 07:33:04hongxusetpull_requests: - pull_request9808
2018-11-16 07:32:56hongxusetpull_requests: - pull_request9809
2018-11-16 07:31:54hongxusetpull_requests: + pull_request9810
2018-11-16 07:20:43hongxusetpull_requests: + pull_request9809
2018-11-16 07:19:23hongxusetpull_requests: - pull_request3087
2018-11-16 07:19:05hongxusetkeywords: + patch
stage: patch review
pull_requests: + pull_request9808
2017-08-10 07:37:01hongxusetpull_requests: + pull_request3087
2017-08-10 07:23:39hongxusetmessages: + msg300054
2017-08-10 07:22:33hongxusetmessages: + msg300053
2017-08-10 07:21:56hongxucreate