classification
Title: Problem compiling the multiprocessing module on sunos5
Type: compile error Stage: resolved
Components: Build Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jnoller Nosy List: BreamoreBoy, SebaM6, fosterremy, jnoller, jr244, neologix, python-dev
Priority: normal Keywords: patch

Created on 2008-10-03 13:13 by jr244, last changed 2011-12-14 17:45 by neologix. This issue is now closed.

Files
File name Uploaded Description Edit
multiprocessing_sendfd.diff neologix, 2011-08-28 17:16 review
Messages (8)
msg74248 - (view) Author: JB Robertson (jr244) Date: 2008-10-03 13:13
Hello,

there's some issues compiling the multiprocessing module on the SunOS I
have here, where CMSG_LEN, CMSG_ALIGN, CMSG_SPACE and sem_timedwait are
absent.

$ uname -av
SunOS xxxxxxx 5.9 Generic_117171-15 sun4u sparc SUNW,Sun-Fire-V440

it looks like simply defining the first three macros like this works:
#ifndef CMSG_SPACE
#   define CMSG_SPACE(len) (CMSG_ALIGN(sizeof(struct
cmsghdr))+CMSG_ALIGN(len))
#endif

#ifndef CMSG_ALIGN
#   ifdef __sun__
#       define CMSG_ALIGN _CMSG_DATA_ALIGN
#   else
#       define CMSG_ALIGN(len) (((len)+sizeof(long)-1) & ~(sizeof(long)-1))
#   endif
#endif

#ifndef CMSG_LEN
#   define CMSG_LEN(len) (CMSG_ALIGN(sizeof(struct cmsghdr))+(len))
#endif

(from http://mailman.videolan.org/pipermail/vlc-devel/2006-May/024400.html )

and setting HAVE_SEM_TIMEDWAIT=0 in setup.py does the trick for the last
one.  Presumably, adding something like 

    elif platform.startswith('sunos5'):
        macros = dict(
                HAVE_SEM_OPEN=1,
                HAVE_SEM_TIMEDWAIT=0,
                HAVE_FD_TRANSFER=1
                )
        libraries = ['rt']

in setup.py should work, but I have no other SunOS machines to test this on.

thanks
msg109590 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-08 20:19
Is this still an issue with more recent versions of Python?
msg140226 - (view) Author: Sebastian M (SebaM6) Date: 2011-07-13 07:47
Yes, it is. I encountered it at Solaris9 with python 2.7.1.
msg140245 - (view) Author: Sebastian M (SebaM6) Date: 2011-07-13 11:14
One more thing, as I tried to rebuild whole python I've encountered on following problem:

building '_multiprocessing' extension
gcc -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_multiprocessing -I. -IInclude -I./Include -I/usr/local/include -I/home/malyska/bld/python_sol10/Python-2.7.1/Include -I/home/malyska/bld/python_sol10/Python-2.7.1 -c /home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/multiprocessing.c -o build/temp.solaris-2.9-sun4u-2.7/home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/multiprocessing.o
gcc -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_multiprocessing -I. -IInclude -I./Include -I/usr/local/include -I/home/malyska/bld/python_sol10/Python-2.7.1/Include -I/home/malyska/bld/python_sol10/Python-2.7.1 -c /home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/socket_connection.c -o build/temp.solaris-2.9-sun4u-2.7/home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/socket_connection.o
gcc -fPIC -fno-strict-aliasing -g -O2 -DNDEBUG -DHAVE_SEM_OPEN=1 -DHAVE_FD_TRANSFER=1 -DHAVE_SEM_TIMEDWAIT=0 -IModules/_multiprocessing -I. -IInclude -I./Include -I/usr/local/include -I/home/malyska/bld/python_sol10/Python-2.7.1/Include -I/home/malyska/bld/python_sol10/Python-2.7.1 -c /home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/semaphore.c -o build/temp.solaris-2.9-sun4u-2.7/home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/semaphore.o
gcc -shared build/temp.solaris-2.9-sun4u-2.7/home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/multiprocessing.o build/temp.solaris-2.9-sun4u-2.7/home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/socket_connection.o build/temp.solaris-2.9-sun4u-2.7/home/malyska/bld/python_sol10/Python-2.7.1/Modules/_multiprocessing/semaphore.o -L/home/malyska/bld/python_sol10/install/lib -L/usr/local/lib -o build/lib.solaris-2.9-sun4u-2.7/_multiprocessing.so
*** WARNING: renaming "_multiprocessing" since importing it failed: ld.so.1: python: fatal: relocation error: file build/lib.solaris-2.9-sun4u-2.7/_multiprocessing.so: symbol sem_timedwait: referenced symbol not found

so I had to commented out HAVE_SEM_TIMEDWAIT from setup.py, see:
             elif platform.startswith('sunos5'):
                  macros = dict(
                      HAVE_SEM_OPEN=1,
                      HAVE_FD_TRANSFER=1
                      )
                      #HAVE_SEM_TIMEDWAIT=0,
                  libraries = ['rt']
thanks
msg143107 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-08-28 17:16
Hello,

> there's some issues compiling the multiprocessing module on the SunOS
> I have here, where CMSG_LEN, CMSG_ALIGN, CMSG_SPACE and sem_timedwait
> are absent.

CMSG_LEN and friends should be defined by <sys/socket.h> (as required by POSIX). SunOS 5.10 man page lists them:

http://download.oracle.com/docs/cd/E19253-01/816-5173/socket.h-3head/index.html

But not the SunOS 5.9 version:
http://ewp.rpi.edu/hartford/webgen/sysdocs/C/solaris_9/SUNWaman/hman3head/socket.3head.html

> it looks like simply defining the first three macros like this works

It works, but it's probably not a good idea: if the headers don't define CMSG_LEN and friends, then FD passing will probably not work.
It'd be better to not compile multiprocessing_(sendfd|recvfd) if CMSG_LEN is not defined (patch attached).

> sem_timedwait are absent.

Hmmm.
Do you have the compilation's log?
Normally, if sem_timedwait isn't available, HAVE_SEM_TIMEDWAIT shouldn't be defined, and we should fallback to sem_trywait (by the way, calling sem_trywait multiple times until the timeout expires is not the same has calling sem_timedwait: this will fail in case of heavy contention).
So this should build correctly.

And this seems even stranger when I read Sebastian's message:
"""
so I had to commented out HAVE_SEM_TIMEDWAIT from setup.py, see:
             elif platform.startswith('sunos5'):
                  macros = dict(
                      HAVE_SEM_OPEN=1,
                      HAVE_FD_TRANSFER=1
                      )
                      #HAVE_SEM_TIMEDWAIT=0,
                  libraries = ['rt']
"""

Makes sense.
If we define HAVE_SEMTIMEDWAIT=0, then code guarded by
#ifdef HAVE_SEMTIMEDWAIT

will be compiled, and the linker won't be able to resolve sem_timedwait.
The preprocessor just checks that the symbol is defined, not that it has a non-zero value.
To sum up: could someone with a SunOS box test the attached patch, and post the compilation logs if it still fails?
msg149401 - (view) Author: Craig Foster (fosterremy) Date: 2011-12-13 18:48
I can confirm that the compile completes with a multiprocessing module where it hasn't before--at least in SunOS 5.9.
msg149460 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-12-14 17:41
New changeset 8c658b625475 by Charles-François Natali in branch '2.7':
Issue #4028: Make multiprocessing build on SunOS.
http://hg.python.org/cpython/rev/8c658b625475

New changeset 49e82c885d6b by Charles-François Natali in branch '3.2':
Issue #4028: Make multiprocessing build on SunOS.
http://hg.python.org/cpython/rev/49e82c885d6b

New changeset 4a27b5ab2710 by Charles-François Natali in branch 'default':
Null merge - Issue #4028: Make multiprocessing build on SunOS.
http://hg.python.org/cpython/rev/4a27b5ab2710
msg149461 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2011-12-14 17:45
Thanks Craig.
History
Date User Action Args
2011-12-14 17:45:09neologixsetstatus: open -> closed
versions: - Python 3.1
messages: + msg149461

resolution: fixed
stage: resolved
2011-12-14 17:41:23python-devsetnosy: + python-dev
messages: + msg149460
2011-12-13 18:48:40fosterremysetnosy: + fosterremy
messages: + msg149401
2011-08-28 17:16:27neologixsetfiles: + multiprocessing_sendfd.diff

nosy: + neologix
messages: + msg143107

keywords: + patch
2011-07-13 11:14:34SebaM6setmessages: + msg140245
2011-07-13 07:47:31SebaM6setnosy: + SebaM6
messages: + msg140226
2010-07-08 20:19:38BreamoreBoysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6, Python 3.0
nosy: + BreamoreBoy

messages: + msg109590

components: + Build, - Extension Modules
2008-10-06 15:00:22jnollersetpriority: normal
assignee: jnoller
nosy: + jnoller
versions: + Python 3.0
2008-10-03 13:13:19jr244create