classification
Title: Build failure for 3.6 on Centos 5.11
Type: compile error Stage:
Components: Build Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, lukasz.langa, martin.panter, ncoghlan, ned.deily, njs, python-dev, rgbecker, steven.daprano, tds333, thewtex, vstinner
Priority: Keywords:

Created on 2016-09-12 03:16 by steven.daprano, last changed 2016-10-21 05:41 by benjamin.peterson. This issue is now closed.

Messages (37)
msg275935 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-09-12 03:16
On Centos 5.11, building fails with:

Python/dtrace_stubs.o: In function `PyDTrace_LINE':
/home/steve/python/python-dev/cpython/Include/pydtrace.h:28: multiple definition of `PyDTrace_LINE'
Python/ceval.o:/home/steve/python/python-dev/cpython/Include/pydtrace.h:28: first defined here
Python/dtrace_stubs.o: In function `PyDTrace_FUNCTION_ENTRY':
/home/steve/python/python-dev/cpython/Include/pydtrace.h:29: multiple definition of `PyDTrace_FUNCTION_ENTRY'
Python/ceval.o:/home/steve/python/python-dev/cpython/Include/pydtrace.h:29: first defined here
 [... many, many more duplicate definitions ...]
collect2: ld returned 1 exit status
make: *** [Programs/_freeze_importlib] Error 1


I've tried the usual `make distclean` and `hg update -C default` tricks, and the error remains.

See also discussion here:

https://mail.python.org/pipermail/python-list/2016-September/713888.html
msg275939 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-12 03:26
This is like the OS X Tiger buildbot failure.
http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/11323/steps/compile/logs/stdio
/usr/bin/ld: multiple definitions of symbol _PyDTrace_FUNCTION_ENTRY
Python/ceval.o definition of _PyDTrace_FUNCTION_ENTRY in section (__TEXT,__text)
Python/dtrace_stubs.o definition of _PyDTrace_FUNCTION_ENTRY in section (__TEXT,__text)

My stab in the dark is that the compiler (GCC?) being used is interpreting “inline” functions differently to C99.
msg275953 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-09-12 04:33
(The OS X Tiger buildbot uses a version of GCC 4.0.)
msg275966 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-09-12 05:38
Benjamin changed PEP 7 to allow static inline functions directly in Python 3.6. But later he added program-wide, linkable inline functions in the Python 3.6 code:

2f77a9f0b9d6: add plain “inline” to header file
63ae310b60ff: add “extern inline” stubs in a new file

It seems GCC does not support the C99 syntax for these kind of linkable inline functions until 4.3. Some possible fixes or workarounds:

1. Revert the offending functions back to macros

2. Clarify in PEP 7 that we intentionally use linkable C99 (extern) inline functions, therefore compilers like Steven’s and the buildbot’s aren’t supported for 3.6

3. Add some preprocessor magic based on __GNUC_STDC_INLINE__ to detect when the GCC 4.0 reversed meanings of “inline” and “extern inline” have to be used.
msg275971 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-09-12 06:18
GCC 4.3 was released in March 2008, so I think we are within our rights to drop support for older toolchains.
msg275984 - (view) Author: Wolfgang Langner (tds333) * Date: 2016-09-12 07:35
This can be related to building manylinux1 binaries where the Centos 5.11 is used.
There can be a possibility to break manaylinux1 builds for Python 3.6, they are used to distribute packages for every Linux distribution.

https://www.python.org/dev/peps/pep-0513/
msg275997 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-09-12 08:35
Do we know for sure that the manylinux1 builds are affected by this, i.e. can someone try building manylinux1 for 3.6?

On the one hand, it is clear that many core developers would like to use these newer features of C, features that have been available for many years.  On the other, we may be breaking the "manylinux1" proposed standard and force a move to a newer one, as well as directly impacting some users (such as Steven and one of our buildbots) using old setups.

Nick, as BDFL-Delegate for PEP 513, what do you think?  We need to resolve this issue before we can release b1.
msg276012 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-09-12 10:02
While manylinux1 requires that compliant binaries *run* when linked against the GCC 4.2 binaries, the actual recommended compiler version (and the one used in the reference Docker image) is 4.8.2: https://www.python.org/dev/peps/pep-0513/#compilation-of-compliant-wheels

That's the compiler version in Red Hat's Developer Toolset 2, the last version to support RHEL/CentOS 5.

Unfortunately, softwarecollections.org only goes back to RHEL/CentOS 6, so the devtoolset RPMs are available via Tru Huynh's personal account on the CentOS file server: https://people.centos.org/tru/devtools-2/readme

So if 3.6 compiles and runs on CentOS 5 with GCC 4.8.2, that's sufficiently compatible for manylinux1 purposes.
msg276018 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-09-12 10:33
Thanks for the info, Nick.  With that in mind, I"m removing this as a "release blocker" for now.  Steven, would it be possible to upgrade gcc on your system?  We'll have to deal with the Tiger buildbot separately.
msg276020 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-12 10:51
I created the issue #28099 "Drop Mac OS X Tiger support in Python 3.6.
msg276079 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2016-09-12 17:34
Benjamin, what's the rationale behind switching those to inline functions? Does it improve runtime performance or build speed? If not, I don't understand why the additional complexity.
msg276089 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-09-12 18:57
For the specific case of Dtrace, #define are just fine. But for
unicodeobject.h, it would be nice to be able to use inline functions.
msg276144 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2016-09-13 00:37
Ned, I know my system is old so I understand if 3.6 no longer supports 
gcc 4.1. Tell me what minimum version I should use (4.8?) and I'll find 
a way to upgrade or use another system.

Shouldn't the build system explicitly report that the compiler is too 
old, and suggest a minimum version, rather than just dump some random 
C-level error in the user's lap? There's a big gap between "able to run 
configure/make" and "able to diagnose configure/make errors" where users 
are currently not well provided for. It is very frustrating to deal with 
these sorts of errors especially when most of the help you get from the 
internet is "run make distclean and try again".
msg276150 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-09-13 02:26
[discussion with Steven about compiler versions taken off-list]
msg276164 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-09-13 04:32
On Mon, Sep 12, 2016, at 10:34, Łukasz Langa wrote:
> 
> Łukasz Langa added the comment:
> 
> Benjamin, what's the rationale behind switching those to inline
> functions? Does it improve runtime performance or build speed? If not, I
> don't understand why the additional complexity.

Macros for the stub case of dtrace are "fine" because they're so simple.
I mainly converted them to see if it would cause problems (e.g., this
bug report) before seeking further expansion.

Many of CPython's uglier macros could be replaced by inline funtions.
Inline functions provide the same performance benefit of macros while
improving type safety, debugability, and avoiding notorious macro quirks
like double evaluation.
msg278392 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-10-09 21:41
For replacing macros, I think “static inline” may be fine, even with older compilers. Maybe these PyDTrace_ functions could also be static inline, since they were originally macros. Or do they really need to be linkable inline functions?
msg278556 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-13 11:05
Don't want to add too much noise, but this issue also affects the manylinux project build compiler (gcc 4.8.2).
msg278568 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-13 12:24
> Don't want to add too much noise, but this issue also affects the manylinux project build compiler (gcc 4.8.2).

Would it be possible to upgrade the "manylinux" compiler (take a more recent GCC version)?
msg278584 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2016-10-13 17:45
> Would it be possible to upgrade the "manylinux" compiler (take a more recent GCC version)?

No, it's possible :-(. 4.8.2 is the very most modern version of GCC you can use if you want to build binaries to run on CentOS/RHEL 5. (And "binaries should run on CentOS/RHEL 5" is the definition of manylinux1.)

I am a bit confused that gcc 4.8.2 is having trouble with cpython 3.6.0b2, though -- supposedly anything newer than gcc 4.3 should be fine with it. And yet. One possibility is that something funny is going on inside the build scripts Robin's using and that Python's ./configure is somehow finding and using the platform compiler (gcc 4.1) even though the first "gcc" in $PATH is 4.8.2, which would make this a false alarm.
msg278599 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-13 20:05
I executed gcc --version (&cc --version) in the do_cpython_build function immediately prior to the make -j2 that builds python noth show 4.8.2.

I see the exact same errors as in the initial report. If the makefile or the configure is doing something special then I guess I have to work around that. A possibility is that the CFLAGS="-Wformat" in the environment or the configure argument --disable-shared is having some effect.

I have made very few changes to the build scripts.
msg278602 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2016-10-13 20:16
@rgbecker: Are you able to pull out the config.log generated by running python's ./configure script, and post that somewhere?
msg278611 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-13 20:44
"Don't want to add too much noise, but this issue also affects the manylinux project build compiler (gcc 4.8.2)."

Can you elaborate on these issues? Are you getting the same errors than the error described in the initial message with GCC 4.2? If not, you may open a new issue to track compilation issues of Python 3.6 on GCC 4.8.
msg278639 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-14 09:20
Hi njs,

my manylinux diffs
https://www.reportlab.com/media/manylinux-diff.txt

full output of the docker command
docker build -f Dockerfile-x86_64 -t rl/manylinux-x86_64 . &> ~/tmp/ttt

https://www.reportlab.com/media/manylinux-docker-run-output.txt

the end showing the 3.6b2 config and the failure
https://www.reportlab.com/media/manylinux-docker-run-python-3.6b2.txt
msg278643 - (view) Author: Wolfgang Langner (tds333) * Date: 2016-10-14 11:46
Please check if you have enabled the compiler as default by enabling the devtoolset on CentOS 5.

I have compiled Python 3.6b2 on Ubuntu 14.04 with gcc 4.8.4 without any problems.
Therefore gcc 4.8.2 should not be that different.

Also keep in mind the default gcc for CentOS 5 will fail because it is to old.
msg278645 - (view) Author: Wolfgang Langner (tds333) * Date: 2016-10-14 12:18
Also verified on CentOS 6.8 with devtoolset2 installed, gcc 4.8.2
Python 3.6b2 builds fine, all unittest ok.
This is the same devtoolset as used on CentOS 5 manylinux1.
Have no CentOS 5 VM available to do more tests. But gcc 4.8 is able to build Python 3.6.
msg278647 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-14 12:41
tds333, the config says that 4.8.2 is being used, 

configure:3902: gcc --version >&5
gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

but perhaps the make is doing something else or the build goes wrong because shared is disabled.
msg278655 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2016-10-14 16:47
Yeah, the config.log there clearly shows that configure is using gcc 4.8.2 on the failed builds. Very odd.

Stepping back for a moment, is there any point in continuing to debug this? Given Benjamin's comment up-thread:

> I mainly converted them to see if it would cause problems

My impression is that restricting inline functions to 'static inline' only is the best plan for 3.6 anyway, since AFAIK that really does work on all the compilers that people are likely to run into, while non-static inline has problems with the default compilers on both CentOS 5 and OS X.
msg278659 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-14 17:22
"while non-static inline has problems with the default compilers on both CentOS 5 and OS X."

The changes introduced in 3.6 prevent compilation with gcc4.0 which was the default Apple-supplied compiler on Mac OS X 10.4 (Tiger).  3.6 currently compiles correctly with gcc4.2, the default on Mac OS X 10.5 (Leopard).
msg278661 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2016-10-14 18:25
> 3.6 currently compiles correctly with gcc4.2, the default on Mac OS X 10.5 (Leopard).

So to summarize our current understanding: gcc 4.3 added support for C99 inline, which explains why gcc 4.2 works and gcc 4.8 doesn't.

WTF is going on.
msg278696 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-10-15 06:26
Any different results with CFLAGS="-fno-gnu89-inline"?
msg278701 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-15 07:22
I'm not able to access my work computer, I'll try later today at home.
msg278715 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-15 10:43
benjamin.peterson: I tried adding that option -fno-gnu89-inline conditionally for the 3.6.0b2 build, but it goes wrong in config with an error

configure:3913: $? = 4
configure:3933: checking whether the C compiler works
configure:3955: gcc -Wformat -fno-gnu89-inline   conftest.c  >&5
cc1: error: -fno-gnu89-inline is only supported in GNU99 or C99 mode

The full docker output is here

https://www.reportlab.com/media/manylinux-docker-run-output-2.txt
msg278721 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-15 12:09
After some searching I tried adding -std=gnu99 and the config goes through OK, but running make produces


[d3cce9786c2e">root@d3cce9786c2e Python-3.6.0b2]# make -j2
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wformat -fno-gnu89-inline -std=gnu99   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers   -I. -I./Include    -DPy_BUILD_CORE -o Programs/python.o ./Programs/python.c
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wformat -fno-gnu89-inline -std=gnu99   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers   -I. -I./Include    -DPy_BUILD_CORE -o Parser/acceler.o Parser/acceler.c
/tmp/cc3Mfloe.s: Assembler messages:
/tmp/cc3Mfloe.s:1575: Error: symbol `stat64' is already defined
/tmp/cc3Mfloe.s:1595: Error: symbol `lstat64' is already defined
/tmp/cc3Mfloe.s:1615: Error: symbol `fstat64' is already defined
/tmp/cc3Mfloe.s:1635: Error: symbol `fstatat64' is already defined
make: *** [Programs/python.o] Error 1
make: *** Waiting for unfinished jobs....
/tmp/ccOoXPte.s: Assembler messages:
/tmp/ccOoXPte.s:1575: Error: symbol `stat64' is already defined
/tmp/ccOoXPte.s:1595: Error: symbol `lstat64' is already defined
/tmp/ccOoXPte.s:1615: Error: symbol `fstat64' is already defined
/tmp/ccOoXPte.s:1635: Error: symbol `fstatat64' is already defined
make: *** [Parser/acceler.o] Error 1
[d3cce9786c2e">root@d3cce9786c2e Python-3.6.0b2]#
msg279040 - (view) Author: Robin Becker (rgbecker) Date: 2016-10-20 15:34
for what it's worth I tried reverse patching

https://hg.python.org/cpython/rev/63ae310b60ff/

and

https://hg.python.org/cpython/rev/2f77a9f0b9d6/

in the manylinux docker and the make then proceeds fine with one warning at the end

*** WARNING: renaming "_sqlite3" since importing it failed: build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sqlite3_stmt_readonly

Following modules built successfully but were removed because they could not be imported:
_sqlite3
msg279087 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-10-20 22:09
"*** WARNING: renaming "_sqlite3" since importing it failed: build/lib.linux-x86_64-3.6/_sqlite3.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sqlite3_stmt_readonly"

That's a different issue: most likely you are building with an old version of libsqlite3.  See, for example: https://github.com/ghaering/pysqlite/issues/85
msg279110 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-21 05:39
New changeset fd9a4bd16587 by Benjamin Peterson in branch '3.6':
mark dtrace stubs as static inline; remove stubs
https://hg.python.org/cpython/rev/fd9a4bd16587
msg279111 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2016-10-21 05:41
I changed the dtrace stubs to static inline. Probably should reopen an investigation for 3.7. I would like to have exportable inlines.
History
Date User Action Args
2016-10-21 05:41:06benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg279111
2016-10-21 05:39:51python-devsetnosy: + python-dev
messages: + msg279110
2016-10-20 22:09:41ned.deilysetmessages: + msg279087
2016-10-20 15:34:55rgbeckersetmessages: + msg279040
2016-10-15 12:09:39rgbeckersetmessages: + msg278721
2016-10-15 10:43:15rgbeckersetmessages: + msg278715
2016-10-15 07:22:39rgbeckersetmessages: + msg278701
2016-10-15 06:26:01benjamin.petersonsetmessages: + msg278696
2016-10-14 18:25:39njssetmessages: + msg278661
2016-10-14 17:22:08ned.deilysetmessages: + msg278659
2016-10-14 16:47:48njssetmessages: + msg278655
2016-10-14 12:41:16rgbeckersetmessages: + msg278647
2016-10-14 12:18:31tds333setmessages: + msg278645
2016-10-14 11:46:02tds333setmessages: + msg278643
2016-10-14 09:20:30rgbeckersetmessages: + msg278639
2016-10-13 20:44:05vstinnersetmessages: + msg278611
2016-10-13 20:16:05njssetmessages: + msg278602
2016-10-13 20:05:22rgbeckersetmessages: + msg278599
2016-10-13 18:15:42thewtexsetnosy: + thewtex
2016-10-13 17:45:59njssetnosy: + njs
messages: + msg278584
2016-10-13 12:24:54vstinnersetmessages: + msg278568
2016-10-13 11:05:17rgbeckersetnosy: + rgbecker
messages: + msg278556
2016-10-09 21:41:07martin.pantersetmessages: + msg278392
2016-09-16 05:44:51martin.panterlinkissue28177 superseder
2016-09-13 04:32:47benjamin.petersonsetmessages: + msg276164
2016-09-13 02:26:34ned.deilysetmessages: + msg276150
2016-09-13 00:37:12steven.dapranosetmessages: + msg276144
2016-09-12 18:57:05vstinnersetmessages: + msg276089
2016-09-12 17:34:59lukasz.langasetmessages: + msg276079
2016-09-12 10:51:57vstinnersetnosy: + vstinner
messages: + msg276020
2016-09-12 10:33:34ned.deilysetpriority: release blocker ->

messages: + msg276018
2016-09-12 10:02:27ncoghlansetmessages: + msg276012
2016-09-12 08:35:43ned.deilysetpriority: normal -> release blocker
nosy: + ncoghlan
messages: + msg275997

2016-09-12 07:35:22tds333setnosy: + tds333
messages: + msg275984
2016-09-12 06:18:55benjamin.petersonsetmessages: + msg275971
2016-09-12 05:38:05martin.pantersetmessages: + msg275966
2016-09-12 05:20:06r.david.murraysetnosy: + lukasz.langa
2016-09-12 04:33:30ned.deilysetmessages: + msg275953
2016-09-12 03:26:47martin.pantersetnosy: + benjamin.peterson, martin.panter
messages: + msg275939
2016-09-12 03:16:59steven.dapranocreate