classification
Title: clean up Solaris port and allow C99 extension modules
Type: Stage: resolved
Components: Build Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: loewis Nosy List: avl, drkirkby, jcea, loewis, movement, pitrou, tflecht, trs80, zooko
Priority: normal Keywords: patch

Created on 2007-07-23 20:24 by zooko, last changed 2011-04-29 17:14 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
pypatch.diff zooko, 2007-07-23 20:24
Messages (23)
msg52938 - (view) Author: Zooko O'Whielacronx (zooko) Date: 2007-07-23 20:24
I wanted to build a Python extension module written in C99, but Solaris's headers ("sys/feature_test.h") declare that you can't have an XPG 4.2 application using C99.

Investigating led me eventually to Solaris hacker John Levon, who kindly pointed out to me that in the Sun-maintained port of Python, they use a perl script to strip out _XOPEN_SOURCE, _XOPEN_SOURCE_EXTENDED, and _POSIX_SOURCE from pyconfig.h:

http://src.opensolaris.org/source/xref/jds/spec-files/trunk/SUNWPython.spec

(See lines 123-125.)

The attached patch accomplishes the same goal more cleanly by changing the contents of configure.in.  With this patch, I can now build python trunk and release25-maint and also build a C99 extension module:

http://cheeseshop.python.org/pypi/zfec

This patch might also impact "[ 1306253 ] Python 2.4.2c1 fails to build on 64-bit Solaris 10":

https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1306253&group_id=5470

I think perhaps this patch should supercede the patch mentioned in "[ 1116722 ] Solaris 10 fails to compile complexobject.c [FIX incl.]":

https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1116722&group_id=5470
msg52939 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2007-07-24 22:02
Looks like a patch for you, Martin...
msg55588 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-09-02 19:14
I don't understand the problem. What Solaris version are you using?

Why do you want to remove _XOPEN_SOURCE? Solaris considers this as a
request for XPG 4.2 only _XOPEN_SOURCE_EXTENDED is defined, which it
shouldn't be on Solaris 10.
msg73709 - (view) Author: Zooko O'Whielacronx (zooko) Date: 2008-09-24 12:23
Sorry I didn't get back to this ticket, MvL.

Recently someone trying to build the Tahoe distributed filesystem on
Solaris had a problem:

http://allmydata.org/pipermail/tahoe-dev/2008-September/000789.html

They had built Python 2.5 themselves from source.

It turned out to be this issue:

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6395191

Which is that g++ on Solaris fails when _XOPEN_SOURCE_EXTENDED is defined.

This reminded me of this ticket, so here I am again.

So, here is the port of Python to Solaris that is maintained by the
Solaris engineers:

http://src.opensolaris.org/source/xref/jds/spec-files/trunk/SUNWPython.spec

It contains the following stanza:

    146 # These #define's break the build with gcc and cause problems when
    147 # building c99 compliant python modules
    148 perl -pi -e 's/^(#define _POSIX_C_SOURCE.*)/\/* $1 *\//' pyconfig.h
    149 perl -pi -e 's/^(#define _XOPEN_SOURCE.*)/\/* $1 *\//' pyconfig.h
    150 perl -pi -e 's/^(#define _XOPEN_SOURCE_EXTENDED.*)/\/* $1 *\//'
pyconfig.h

This shows that the Python that comes with Solaris has, for a long time
now, had these #define's stripped out of the pyconfig.h.  Why?  Well, I
asked the Solaris engineers about it on IRC, and they were very nice and
helpful, and they said, if I am remembering correctly, that Solaris
doesn't use these #define's in the way that we Python people think.

#defining _XOPEN_SOURCE_EXTENDED on Solaris is *not* understood by the
Solaris runtime as a request for XPG4v2 (UNIX98) features to be
supported, but more as a declaration that the code we are compiling was
written for XPG4v2.  So, for example, this disables C99, because after
all C99 wasn't available in XPG4v2.  It also apparently breaks g++, but
that is by accident rather than design.  In any case, the Solaris
engineers assured me that rather than figuring out which unixy features
are turned on by which #defines, we should instead #define
__EXTENSIONS__ and then assume all the features we can eat.

Whether this is true in general remains to be seen, but apparently
_POSIX_C_SOURCE, _XOPEN_SOURCE, and _XOPEN_SOURCE_EXTENDED, at least,
are proven to be unnecessary on Solaris by the fact that the version of
Python that has been shipping with Solaris for who knows how long has
them stripped out.

Therefore, I propose to apply my patch so that we do not turn on
_XOPEN_SOURCE but we do turn on __EXTENSIONS__, when building for Solaris.
msg80949 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-02-02 16:35
If the Sun guy only commented on _XOPEN_SOURCE_EXTENDED, I'm not sure
that this also applies to _XOPEN_SOURCE. For example, on my Solaris 9,
I have (in signal.h) conditionally

#pragma redefine_extname sigwait __posix_sigwait 

and in sys/socket.h

#pragma redefine_extname connect __xnet_connect

So these feature selection macros not only select what API is declared
in the header files, but also what specific semantics some of the
functions have (e.g. if the traditional BSD semantics deviates from the
POSIX semantics).

In cases where there are alternative implementations of some system
calls, we *want* to select the POSIX definition. So I think we should
continue to define _XOPEN_SOURCE. I don't mind also defining __EXTENSIONS__.
msg82201 - (view) Author: John Levon (movement) Date: 2009-02-16 01:22
Just came across this bug.

_XOPEN_SOURCE_EXTENDED isn't the only sticking point here: neither XPG3
nor XPG4 have C99 either: _XOPEN_SOURCE (at least with a value below
600) is definitely wrong. To quote the header:

"346  * It is invalid to compile an XPG3, XPG4, XPG4v2, or XPG5 application
347  * using c99.  The same is true for POSIX.1-1990, POSIX.2-1992,
POSIX.1b,
348  * and POSIX.1c applications. Likewise, it is invalid to compile an XPG6
349  * or a POSIX.1-2001 application with anything other than a c99 or later
350  * compiler.  Therefore, we force an error in both cases.
"

Martin, you are correct that these macros choose particular semantics
in some cases.

However, this does not change my point. It is a fact that specifying
these macros mean what I describe, and that doing so causes significant
problems in compiling Python extensions. We may wish it to be different,
but it's not.

As Solaris has only ever moved towards the POSIX spec, __EXTENSIONS__ is
the right thing to always specify here: it enables everything "modern"
that you need (*including* POSIX semantics - check that signal.h on
Solaris 9 for __EXTENSIONS__), without making any claims about
conformity of Python extensions (which it it is not pyconfig.h's place
to say!).

I hope that's a clearer explanation of why Python is doing the wrong
thing both theoretically and practically for Solaris.
msg82203 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-02-16 02:06
> As Solaris has only ever moved towards the POSIX spec, __EXTENSIONS__ is
> the right thing to always specify here

And indeed, Python does define __EXTENSIONS__.

> I hope that's a clearer explanation of why Python is doing the wrong
> thing both theoretically and practically for Solaris.

No. You are arguing that Python should define __EXTENSIONS__ when it
already does. This doesn't help to clarify.
msg82212 - (view) Author: John Levon (movement) Date: 2009-02-16 02:59
Python should define *only* __EXTENSIONS__, not the others, as Zooko
already mentioned, for the reasons he mentioned. I'm not sure how you
got a different impression from what I or he said.
msg82216 - (view) Author: Zooko O'Whielacronx (zooko) Date: 2009-02-16 05:02
Martin (sometimes called "MvL") mentioned some specific issues about e.g.

#pragma redefine_extname sigwait __posix_sigwait

I didn't understand exactly what MvL's concern was, and I don't know off
the top of my head how Solaris sigwait behavior differs with
_XOPEN_SOURCE, and I haven't taken the time to investigate yet, so I
haven't replied.

(Of course, last time I was in that state and I dropped the ball, this
ticket stayed quiet for a full year...)

Perhaps John Levon could inquire what semantics MvL is concerned about
and let him know what the Solaris behavior is.  I think it would be cool
if the python executable that people use on Solaris were closer to
upstream python.  Currently, as I've mentioned, the Solaris folks
maintain some kind of tool (I think it is a perl script) which searches
and replaces "#define _XOPEN_SOURCE" to empty as well as applying a few
patches.  It would be great if the Solaris and Python folks would
cooperate to close that gap.

Speaking of which, if John Levon or other Solaris folk are reading --
where are those patches?  This file mentions a bunch of patches,
starting with:

http://src.opensolaris.org/source/xref/jds/spec-files/trunk/SUNWPython.spec

     28 Patch1:                  Python-01-solaris-lib-dirs.diff

But I can't seem to find "Python-01-solaris-lib-dirs.diff".

Thanks!

--Zooko
msg82223 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-02-16 08:17
> Python should define *only* __EXTENSIONS__, not the others, as Zooko
> already mentioned, for the reasons he mentioned. I'm not sure how you
> got a different impression from what I or he said.

Ok, so we can ignore __EXTENSIONS__ for the discussion. Then you were
arguing that defining _XOPEN_SOURCE is wrong if the value is less than
600. What about setting the value to 600 or larger? Is that also wrong,
and if so, why?

[Aside: I think the Solaris comment on using C99 with XOPEN applications
is misguided. Compiling such applications with C99 is *not* invalid.
While POSIX specified that it is based on C89, the original POSIX spec
is silent on what the semantics is if a different C implementation is
used - so at worst, the behavior of such applications is undefined.
However, it is actually better than that: The C99 spec defines clearly
what the semantics of all the functions are, and thus what the
difference in behavior is. So for a specific POSIX application, the
effect of compiling it with a C99 compiler is well-defined, and most
applications should work just fine (if they compile at all).

In a sense, Solaris is doing the same thing to its applications what
you are criticizing here as Python doing to its extension modules:
making claims of how compatible they are wrt. certain standards.]
msg82240 - (view) Author: John Levon (movement) Date: 2009-02-16 14:26
Zooko, the patches are available here:

http://src.opensolaris.org/source/xref/jds/spec-files/trunk/patches/

> Then you were arguing that defining _XOPEN_SOURCE is wrong if the value

Actually, I was saying that below that value simply won't work for many
extensions.

> What about setting the value to 600 or larger? Is that also wrong,
> and if so, why?

Practically: I don't know for sure about a larger value. As it is
presumably very close to POSIX, it shouldn't significantly harm the
ability to compile stuff. I can't say for sure. However, specifying
_XOPEN_SOURCE is still unnecessary I believe.

Theoretically: yes, it's wrong for Python to claim that all extensions
are compliant to a particular XPG environment.

If it helps, Martin, I can put you in touch with our standards expert.
I'm far from such, so this might help assuage your doubts?
msg82274 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-02-16 21:43
> Theoretically: yes, it's wrong for Python to claim that all extensions
> are compliant to a particular XPG environment.

It's somewhat unfortunate that Python makes such claims for extensions;
primarily, these macros are defined for the compilation of Python
itself. It would be possible to suppress them when compiling non-core
extensions.

However, experience tells that systems can break in surprising ways
if the system headers are compiled with different defines. On some
systems, the sizes and layouts of some types change under such macros,
causing crashes when the extension sees a different size or layout than
the Python core.

> If it helps, Martin, I can put you in touch with our standards expert.
> I'm far from such, so this might help assuage your doubts?

I do feel this restrictiveness of the header files (wrt. C99) is
arbitrary, and has no use. I think system vendors often interpret
"feature selection macro" in a different way than it should be
interpreted (i.e. instead of "make sure any program complying with
the selected features compiles fine", the macros are interpreted
as "reject any program that uses features beyond the selected ones").

So if there is a chance that Sun might reconsider this specific design
choice, I'd love to talk with somebody - setting _XOPEN_SOURCE had
worked fine since Solaris 7.

Independent of such a discussion, I can grudgingly accept removal
of _XOPEN_SOURCE on Solaris 5.{9,10,11} (?), to work-around this
system limitation.
msg82338 - (view) Author: John Levon (movement) Date: 2009-02-17 14:30
> However, experience tells that systems can break in surprising ways
> if the system headers are compiled with different defines.

This is indeed a reasonable concern (for which the best solution is
dropping the defines in the Python compile).

> I do feel this restrictiveness of the header files (wrt. C99) is
> arbitrary, and has no use.

Unfortunately, neither you, I, nor Sun can do anything at all about this
fact. I've filed an RFE for you:

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6806390
feature_tests.h could be laxer about C99

(It'll take a while to be visible externally.)

Even if this happens, though, it's strictly a workaround: it is still a
bug that Python sets these defines on Solaris: they do not mean what the
Python build thinks they mean. It's effectively a Linux-ism.

> setting _XOPEN_SOURCE had worked fine since Solaris 7.

The particular change we're talking about (C99) went back in 2003:

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=4760794
4760794 UNIX03: sys/feature_tests.h updates required to support SUSv3
and c99

so Python has been broken by this for all of the Solaris 10 lifecycle.
msg86934 - (view) Author: James Andrewartha (trs80) Date: 2009-05-02 08:38
I'm jhbuilding GNOME on Solaris, and the attached patch fixes the
problem for me, having compiled Python with it I can now compile
dbus-python, pycairo and pyorbit against it.
msg93300 - (view) Author: Tim Flechtner (tflecht) Date: 2009-09-29 15:58
Hi.  I think I'm running into the problem discussed in this ticket.  I'm
trying to build pycairo-1.8.8 on Solaris 10 (x86) against python-2.6.2
using gcc-4.4.1.  I'm afraid I'm out of my depth in understanding really
what's going on, but the error I get looks like it fits the symptoms
described here.  Can anyone point me towards the patch to apply?  I
don't see it as available here (but I wouldn't be shocked if I'm just
being blind).

Thanks!
-tim

gmake[2]: Entering directory
`/export/home/src/ThirdParty/gnu/pycairo-1.8.8/src'
/bin/bash ../libtool  --tag=CC   --mode=compile
/opt/app/g++lib6/gcc-4.4/bin/gcc -DHAVE_CONFIG_H -I. 
-I/opt/app/g++lib6/python-2.6/include/python2.6
-I/opt/app/g++lib6/python-2.6/include/python2.6
-I/opt/app/nonc++/release-1/include/cairo   -DNDEBUG  -g -O2 -std=c99
-Wall -fno-strict-aliasing -MT _cairo_la-cairomodule.lo -MD -MP -MF
.deps/_cairo_la-cairomodule.Tpo -c -o _cairo_la-cairomodule.lo `test -f
'cairomodule.c' || echo './'`cairomodule.c
libtool: compile:  /opt/app/g++lib6/gcc-4.4/bin/gcc -DHAVE_CONFIG_H -I.
-I/opt/app/g++lib6/python-2.6/include/python2.6
-I/opt/app/g++lib6/python-2.6/include/python2.6
-I/opt/app/nonc++/release-1/include/cairo -DNDEBUG -g -O2 -std=c99 -Wall
-fno-strict-aliasing -MT _cairo_la-cairomodule.lo -MD -MP -MF
.deps/_cairo_la-cairomodule.Tpo -c cairomodule.c  -fPIC -DPIC -o
.libs/_cairo_la-cairomodule.o
In file included from /usr/include/limits.h:18,
                 from
/opt/app/g++lib6/gcc-4.4/lib/gcc/i386-pc-solaris2.10/4.4.1/include-fixed/limits.h:122,
                 from
/opt/app/g++lib6/gcc-4.4/lib/gcc/i386-pc-solaris2.10/4.4.1/include-fixed/syslimits.h:7,
                 from
/opt/app/g++lib6/gcc-4.4/lib/gcc/i386-pc-solaris2.10/4.4.1/include-fixed/limits.h:11,
                 from
/opt/app/g++lib6/python-2.6/include/python2.6/Python.h:19,
                 from cairomodule.c:32:
/opt/app/g++lib6/gcc-4.4/lib/gcc/i386-pc-solaris2.10/4.4.1/include-fixed/sys/feature_tests.h:341:2:
error: #error "Compiler or options invalid for pre-UNIX 03 X/Open
applications       and pre-2001 POSIX applications"
msg93329 - (view) Author: Zooko O'Whielacronx (zooko) Date: 2009-09-29 20:43
In msg82274 MvL wrote: "I can grudgingly accept removal of _XOPEN_SOURCE
on Solaris 5.{9,10,11} (?), to work-around this system limitation."

Great!  That's what the attached patch does.  So please apply that patch
and then people can stop coming to this ticket saying that they couldn't
compile their extension modules until they applied the patch.

Also then the Solaris folks who build an installation of Python for
Solaris can remove the perl script that searches for "#define
_XOPEN_SOURCE" and replaces it with the empty string when building Python.
msg93400 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-01 11:06
We're in the middle of a release so this will have to wait until we are
done.
Apart from that, is there a reason *not* to apply the patch?
(it only seems to affect Solaris after all, and Solaris users here seem
consensual that the patch should be applied)
msg94117 - (view) Author: Zooko O'Whielacronx (zooko) Date: 2009-10-16 00:39
I just tried to port a library of mine -- zfec -- to Nexenta (the Ubuntu
variant built on top of OpenSolaris).  I hit this bug because the
Nexenta folks don't use all the patches that are applied to Python by
the Solaris folks.  My port of my library is blocked until the Nexenta
folks apply this patch to their package of Python:

https://bugs.launchpad.net/python/+bug/452667 # can't compile Python
modules that use C99
msg97129 - (view) Author: anders musikka (avl) Date: 2010-01-02 14:28
Just wanted to chip in my $.02: 

Defining _XOPEN_SOURCE in the python headers causes problems for
Solaris. It also causes problems for Ubuntu Linux.

Because _XOPEN_SOURCE is defined, Python.h must included first in any
program under Ubuntu.

Perhaps the right fix is to just not define _XOPEN_SOURCE at all? Which
platforms disable important features when you don't define _XOPEN_SOURCE?

Sometimes there are standards which nobody follows. In such cases it can
be advantageous to "just do what everyone expects".
msg105223 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-05-07 19:21
A question about the patch: in configure.in, after "if test $define_xopen_source = yes" (lines 395 and following), there's a bunch of Solaris-specific stuff. Shouldn't it be removed, given that _XOPEN_SOURCE isn't defined anymore under Solaris?
msg106675 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-05-28 15:46
Thanks for the patch. Committed as r81582 and r81583.

Antoine was right: subsequent references to Solaris needed to be removed also.
msg134790 - (view) Author: David Kirkby (drkirkby) Date: 2011-04-29 17:09
Is there any progress on this? I see it is marked as 

Status: closed
Resolution: accepted
Stage: patch review
    

That apparently means:

''There is a patch, but it needs reviewing or is in the process of being reviewed. This can be done by any triager as well as a core developer.''

So is there any movement on this, which will allow it to be fixed? 

I'm somewhat puzzled the Status is marked as "closed" when apparently the patch is still awaiting review. 

Dave
msg134791 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-04-29 17:14
As noted by Martin above (he quoted the Subversion revision numbers), this was actually fixed.
History
Date User Action Args
2011-04-29 17:14:32pitrousetresolution: accepted -> fixed
messages: + msg134791
stage: patch review -> resolved
2011-04-29 17:09:19drkirkbysetnosy: + drkirkby
messages: + msg134790
2010-11-25 15:36:58jceasetnosy: + jcea
2010-05-28 15:46:51loewissetstatus: open -> closed
resolution: accepted
messages: + msg106675
2010-05-20 20:39:51skip.montanarosetnosy: - skip.montanaro
2010-05-07 19:21:47pitrousetstage: patch review
messages: + msg105223
versions: + Python 3.1, Python 2.7, Python 3.2
2010-01-02 14:28:34avlsetnosy: + avl
messages: + msg97129
2009-10-16 00:39:04zookosetmessages: + msg94117
2009-10-01 11:06:42pitrousetnosy: + pitrou
messages: + msg93400
2009-09-29 20:43:00zookosetmessages: + msg93329
2009-09-29 20:02:08skip.montanarosetnosy: + skip.montanaro
2009-09-29 15:58:42tflechtsetnosy: + tflecht
messages: + msg93300
2009-05-02 08:38:44trs80setnosy: + trs80
messages: + msg86934
2009-02-17 14:30:33movementsetmessages: + msg82338
2009-02-16 21:43:35loewissetmessages: + msg82274
2009-02-16 14:26:59movementsetmessages: + msg82240
2009-02-16 08:18:04georg.brandlsetnosy: - georg.brandl
2009-02-16 08:17:03loewissetmessages: + msg82223
2009-02-16 05:02:18zookosetmessages: + msg82216
2009-02-16 02:59:26movementsetmessages: + msg82212
2009-02-16 02:06:39loewissetmessages: + msg82203
2009-02-16 01:22:57movementsetnosy: + movement
messages: + msg82201
2009-02-02 16:35:02loewissetmessages: + msg80949
2008-09-24 12:23:13zookosetmessages: + msg73709
2007-09-02 19:14:52loewissetmessages: + msg55588
2007-07-23 20:24:10zookocreate