classification
Title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
Type: compile error Stage: resolved
Components: Build Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, nascheme, vstinner
Priority: normal Keywords:

Created on 2017-11-06 21:57 by vstinner, last changed 2018-01-08 17:33 by nascheme.

Messages (10)
msg305682 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-06 21:57
http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
 };
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1


pythoninfo from the previous build:

* GCC 6.3.0 20170516
* os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
* platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio
msg305687 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-06 23:11
Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree.  A few options:

- remove the profile-run-stamp file after checking out new code

- call "make profile-removal" after checkout of new code

- I can try to modify the makefile so that profile-run-stamp depends
  on the source files (.h, .c, etc).  I think that would make it
  re-generate automatically if the sources change.

On 2017-11-06, STINNER Victor wrote:
> 
> New submission from STINNER Victor <victor.stinner@gmail.com>:
> 
> http://buildbot.python.org/all/#/builders/47/builds/127
> 
> (...)
> gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
> Objects/setobject.c: In function ‘PySet_Clear’:
> Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
>  };
>  ^
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
> Makefile:1584: recipe for target 'Objects/setobject.o' failed
> make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
> make[1]: *** [Objects/setobject.o] Error 1
> 
> 
> pythoninfo from the previous build:
> 
> * GCC 6.3.0 20170516
> * os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
> * platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1
> 
> http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio
> 
> ----------
> components: Build
> messages: 305682
> nosy: gregory.p.smith, haypo, nascheme
> priority: normal
> severity: normal
> status: open
> title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
> type: performance
> versions: Python 3.7
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue31963>
> _______________________________________
msg305707 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-11-07 01:14
On Mon, Nov 6, 2017 at 3:11 PM Neil Schemenauer <report@bugs.python.org>
wrote:

>
> Neil Schemenauer <nas-python@arctrix.com> added the comment:
>
> Hi Victor,
>
> My first guess is that the build bot is not cleaning the fprofile
> information after updating the source tree.  A few options:
>

But it is.  I actually looked at what the buildbots did before the PR to
change the Makefile.pre.in for profile_opt builds went in.

Here's what happens:

http://buildbot.python.org/all/#/builders/47/builds/127/steps/2/logs/stdio

Notice the "make clean profile-removal" on line 17.

-gps
msg305708 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-07 01:31
Oh I see.  I'm at a loss then as to why the build is failing.  A possible clue is the errors like:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/setobject.gcda:Merge mismatch for function 10

A Google search brings up this Stackoverflow page:

https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismatch-for-summaries

To me, that implies that there are compiler outputs (.o, .a, etc) that were build from a previous version source code.  However, the first part of the log shows that the build does "make clean" so I don't see how that is possible.

Is there a way to run "git show 4e38d71a2b7 | git apply -R" on the built host and then try to re-build?
msg305906 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-08 21:12
The current "master" branch seems to be building successfully on "AMD64 Debian PGO 3.x".  Can we close this issue?
msg306157 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-13 13:07
Same bug, again,  AMD64 Debian PGO 3.x, so I reopen the issue:

http://buildbot.python.org/all/#/builders/47/builds/159

Python/compile.c: In function ‘compiler_sync_comprehension_generator’:
Python/compile.c:5457:1: internal compiler error: in create_edge, at cgraph.c:850
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
msg306177 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-11-13 23:23
Given this is gcc exploding, I'm not sure there is anything we can do about it.  I _believe_ we're doing everything right.
msg306237 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-14 19:56
I'm willing to put some time into trying to fix this, in the case that it is caused by my Makefile changes.  However, it would be very helpful if I could login to the build-bot and try running with the Makefile change backed out.  Is that possible?  Having to setup a VM to match the compiler of the build-bot would take quite a bit a of extra time.

Looking at the "stdout" log file, I can't see that we are doing anything wrong.  Specifically, the ".gc??" files are being removed and so I think that rules out the theory of stale profile information files messing up GCC.
msg309643 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-07 22:42
Different error:

http://buildbot.python.org/all/#/builders/47/builds/428

Objects/bytearrayobject.c: In function ‘bytearray_startswith’:
Objects/bytearrayobject.c:2429:1: error: the control flow of function ‘bytearray_startswith’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
 }
 ^
cc1: some warnings being treated as errors
msg309680 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2018-01-08 17:33
That's mysterious.  I reviewed the 'stdout' log from the buildbot.  The removal of profile data is run early in the build:

find . -name '*.gc??' -exec rm -f {} ';'

Later, the bytearrayobject is compiled with profile generation enabled:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c

When the unit tests run, we get errors:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/bytearrayobject.gcda:Merge mismatch for function 35

The second build (with profile use) fails because of the missing profile information:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c
Objects/bytearrayobject.c: In function ‘bytearray_startswith’:
Objects/bytearrayobject.c:2429:1: error: the control flow of function ‘bytearray_startswith’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
 }
 ^
cc1: some warnings being treated as errors

Seems like a compiler bug to me.  Wild theory, could it be something to do with address randomization and forking?  Looks like maybe the merge mismatch errors come when the 'python' executable forks.
History
Date User Action Args
2018-01-08 17:33:30naschemesetmessages: + msg309680
2018-01-07 22:42:09vstinnersetmessages: + msg309643
2017-11-14 19:56:04naschemesetmessages: + msg306237
2017-11-13 23:23:36gregory.p.smithsetmessages: + msg306177
2017-11-13 13:07:44vstinnersetstatus: closed -> open
resolution: works for me ->
messages: + msg306157
2017-11-08 21:27:39gregory.p.smithsetstatus: open -> closed
type: performance -> compile error
resolution: works for me
stage: resolved
2017-11-08 21:12:58naschemesetmessages: + msg305906
2017-11-07 01:31:33naschemesetmessages: + msg305708
2017-11-07 01:14:47gregory.p.smithsetmessages: + msg305707
2017-11-06 23:11:06naschemesetmessages: + msg305687
2017-11-06 21:57:41vstinnercreate