classification
Title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
Type: compile error Stage: resolved
Components: Build Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, nascheme, vstinner
Priority: normal Keywords:

Created on 2017-11-06 21:57 by vstinner, last changed 2017-11-14 19:56 by nascheme.

Messages (8)
msg305682 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-06 21:57
http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
 };
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1


pythoninfo from the previous build:

* GCC 6.3.0 20170516
* os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
* platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio
msg305687 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-06 23:11
Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree.  A few options:

- remove the profile-run-stamp file after checking out new code

- call "make profile-removal" after checkout of new code

- I can try to modify the makefile so that profile-run-stamp depends
  on the source files (.h, .c, etc).  I think that would make it
  re-generate automatically if the sources change.

On 2017-11-06, STINNER Victor wrote:
> 
> New submission from STINNER Victor <victor.stinner@gmail.com>:
> 
> http://buildbot.python.org/all/#/builders/47/builds/127
> 
> (...)
> gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
> Objects/setobject.c: In function ‘PySet_Clear’:
> Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
>  };
>  ^
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
> Makefile:1584: recipe for target 'Objects/setobject.o' failed
> make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
> make[1]: *** [Objects/setobject.o] Error 1
> 
> 
> pythoninfo from the previous build:
> 
> * GCC 6.3.0 20170516
> * os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
> * platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1
> 
> http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio
> 
> ----------
> components: Build
> messages: 305682
> nosy: gregory.p.smith, haypo, nascheme
> priority: normal
> severity: normal
> status: open
> title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
> type: performance
> versions: Python 3.7
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue31963>
> _______________________________________
msg305707 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-11-07 01:14
On Mon, Nov 6, 2017 at 3:11 PM Neil Schemenauer <report@bugs.python.org>
wrote:

>
> Neil Schemenauer <nas-python@arctrix.com> added the comment:
>
> Hi Victor,
>
> My first guess is that the build bot is not cleaning the fprofile
> information after updating the source tree.  A few options:
>

But it is.  I actually looked at what the buildbots did before the PR to
change the Makefile.pre.in for profile_opt builds went in.

Here's what happens:

http://buildbot.python.org/all/#/builders/47/builds/127/steps/2/logs/stdio

Notice the "make clean profile-removal" on line 17.

-gps
msg305708 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-07 01:31
Oh I see.  I'm at a loss then as to why the build is failing.  A possible clue is the errors like:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/setobject.gcda:Merge mismatch for function 10

A Google search brings up this Stackoverflow page:

https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismatch-for-summaries

To me, that implies that there are compiler outputs (.o, .a, etc) that were build from a previous version source code.  However, the first part of the log shows that the build does "make clean" so I don't see how that is possible.

Is there a way to run "git show 4e38d71a2b7 | git apply -R" on the built host and then try to re-build?
msg305906 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-08 21:12
The current "master" branch seems to be building successfully on "AMD64 Debian PGO 3.x".  Can we close this issue?
msg306157 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-13 13:07
Same bug, again,  AMD64 Debian PGO 3.x, so I reopen the issue:

http://buildbot.python.org/all/#/builders/47/builds/159

Python/compile.c: In function ‘compiler_sync_comprehension_generator’:
Python/compile.c:5457:1: internal compiler error: in create_edge, at cgraph.c:850
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
msg306177 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-11-13 23:23
Given this is gcc exploding, I'm not sure there is anything we can do about it.  I _believe_ we're doing everything right.
msg306237 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-14 19:56
I'm willing to put some time into trying to fix this, in the case that it is caused by my Makefile changes.  However, it would be very helpful if I could login to the build-bot and try running with the Makefile change backed out.  Is that possible?  Having to setup a VM to match the compiler of the build-bot would take quite a bit a of extra time.

Looking at the "stdout" log file, I can't see that we are doing anything wrong.  Specifically, the ".gc??" files are being removed and so I think that rules out the theory of stale profile information files messing up GCC.
History
Date User Action Args
2017-11-14 19:56:04naschemesetmessages: + msg306237
2017-11-13 23:23:36gregory.p.smithsetmessages: + msg306177
2017-11-13 13:07:44vstinnersetstatus: closed -> open
resolution: works for me ->
messages: + msg306157
2017-11-08 21:27:39gregory.p.smithsetstatus: open -> closed
type: performance -> compile error
resolution: works for me
stage: resolved
2017-11-08 21:12:58naschemesetmessages: + msg305906
2017-11-07 01:31:33naschemesetmessages: + msg305708
2017-11-07 01:14:47gregory.p.smithsetmessages: + msg305707
2017-11-06 23:11:06naschemesetmessages: + msg305687
2017-11-06 21:57:41vstinnercreate