classification
Title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
Type: compile error Stage: resolved
Components: Build Versions: Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: albertchristianto1994, gregory.p.smith, nascheme, vstinner
Priority: normal Keywords:

Created on 2017-11-06 21:57 by vstinner, last changed 2020-05-24 14:31 by albertchristianto1994. This issue is now closed.

Messages (14)
msg305682 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-06 21:57
http://buildbot.python.org/all/#/builders/47/builds/127

(...)
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
Objects/setobject.c: In function ‘PySet_Clear’:
Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
 };
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
Makefile:1584: recipe for target 'Objects/setobject.o' failed
make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
make[1]: *** [Objects/setobject.o] Error 1


pythoninfo from the previous build:

* GCC 6.3.0 20170516
* os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
* platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1

http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio
msg305687 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-06 23:11
Hi Victor,

My first guess is that the build bot is not cleaning the fprofile
information after updating the source tree.  A few options:

- remove the profile-run-stamp file after checking out new code

- call "make profile-removal" after checkout of new code

- I can try to modify the makefile so that profile-run-stamp depends
  on the source files (.h, .c, etc).  I think that would make it
  re-generate automatically if the sources change.

On 2017-11-06, STINNER Victor wrote:
> 
> New submission from STINNER Victor <victor.stinner@gmail.com>:
> 
> http://buildbot.python.org/all/#/builders/47/builds/127
> 
> (...)
> gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/setobject.o Objects/setobject.c
> Objects/setobject.c: In function ‘PySet_Clear’:
> Objects/setobject.c:2579:1: internal compiler error: in create_edge, at cgraph.c:850
>  };
>  ^
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
> Makefile:1584: recipe for target 'Objects/setobject.o' failed
> make[1]: Leaving directory '/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build'
> make[1]: *** [Objects/setobject.o] Error 1
> 
> 
> pythoninfo from the previous build:
> 
> * GCC 6.3.0 20170516
> * os.uname: posix.uname_result(sysname='Linux', nodename='enable-optimizations-bot', release='4.9.0-3-amd64', version='#1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26)', machine='x86_64')
> * platform.platform: Linux-4.9.0-3-amd64-x86_64-with-debian-9.1
> 
> http://buildbot.python.org/all/#/builders/47/builds/126/steps/3/logs/stdio
> 
> ----------
> components: Build
> messages: 305682
> nosy: gregory.p.smith, haypo, nascheme
> priority: normal
> severity: normal
> status: open
> title: AMD64 Debian PGO 3.x buildbot: compilation failed with an internal compiler error in create_edge
> type: performance
> versions: Python 3.7
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue31963>
> _______________________________________
msg305707 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-11-07 01:14
On Mon, Nov 6, 2017 at 3:11 PM Neil Schemenauer <report@bugs.python.org>
wrote:

>
> Neil Schemenauer <nas-python@arctrix.com> added the comment:
>
> Hi Victor,
>
> My first guess is that the build bot is not cleaning the fprofile
> information after updating the source tree.  A few options:
>

But it is.  I actually looked at what the buildbots did before the PR to
change the Makefile.pre.in for profile_opt builds went in.

Here's what happens:

http://buildbot.python.org/all/#/builders/47/builds/127/steps/2/logs/stdio

Notice the "make clean profile-removal" on line 17.

-gps
msg305708 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-07 01:31
Oh I see.  I'm at a loss then as to why the build is failing.  A possible clue is the errors like:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/setobject.gcda:Merge mismatch for function 10

A Google search brings up this Stackoverflow page:

https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismatch-for-summaries

To me, that implies that there are compiler outputs (.o, .a, etc) that were build from a previous version source code.  However, the first part of the log shows that the build does "make clean" so I don't see how that is possible.

Is there a way to run "git show 4e38d71a2b7 | git apply -R" on the built host and then try to re-build?
msg305906 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-08 21:12
The current "master" branch seems to be building successfully on "AMD64 Debian PGO 3.x".  Can we close this issue?
msg306157 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-13 13:07
Same bug, again,  AMD64 Debian PGO 3.x, so I reopen the issue:

http://buildbot.python.org/all/#/builders/47/builds/159

Python/compile.c: In function ‘compiler_sync_comprehension_generator’:
Python/compile.c:5457:1: internal compiler error: in create_edge, at cgraph.c:850
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-6/README.Bugs> for instructions.
msg306177 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-11-13 23:23
Given this is gcc exploding, I'm not sure there is anything we can do about it.  I _believe_ we're doing everything right.
msg306237 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2017-11-14 19:56
I'm willing to put some time into trying to fix this, in the case that it is caused by my Makefile changes.  However, it would be very helpful if I could login to the build-bot and try running with the Makefile change backed out.  Is that possible?  Having to setup a VM to match the compiler of the build-bot would take quite a bit a of extra time.

Looking at the "stdout" log file, I can't see that we are doing anything wrong.  Specifically, the ".gc??" files are being removed and so I think that rules out the theory of stale profile information files messing up GCC.
msg309643 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-07 22:42
Different error:

http://buildbot.python.org/all/#/builders/47/builds/428

Objects/bytearrayobject.c: In function ‘bytearray_startswith’:
Objects/bytearrayobject.c:2429:1: error: the control flow of function ‘bytearray_startswith’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
 }
 ^
cc1: some warnings being treated as errors
msg309680 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2018-01-08 17:33
That's mysterious.  I reviewed the 'stdout' log from the buildbot.  The removal of profile data is run early in the build:

find . -name '*.gc??' -exec rm -f {} ';'

Later, the bytearrayobject is compiled with profile generation enabled:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-generate  -I. -I./Include    -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c

When the unit tests run, we get errors:

profiling:/var/lib/buildbot/slaves/enable-optimizations-bot/3.x.gps-debian-profile-opt.nondebug/build/Objects/bytearrayobject.gcda:Merge mismatch for function 35

The second build (with profile use) fails because of the missing profile information:

gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include    -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c
Objects/bytearrayobject.c: In function ‘bytearray_startswith’:
Objects/bytearrayobject.c:2429:1: error: the control flow of function ‘bytearray_startswith’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
 }
 ^
cc1: some warnings being treated as errors

Seems like a compiler bug to me.  Wild theory, could it be something to do with address randomization and forking?  Looks like maybe the merge mismatch errors come when the 'python' executable forks.
msg361756 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-10 23:56
I didn't see this failure recently, I close the issue.
msg369328 - (view) Author: Albert Christianto (albertchristianto1994) Date: 2020-05-19 09:05
Hi admins,

I need help for compiling and building Python-3.7.7. 
My system is Ubuntu 16.04 LTS, gcc 5.4.0 20160609

this is the configuration cmd for compling python
./configure --enable-optimizations --enable-shared

when i compiled it, i get these similar error

```
Parser/tokenizer.c: In function ‘PyTokenizer_FindEncodingFilename’:
Parser/tokenizer.c:1909:1: error: the control flow of function ‘PyTokenizer_FindEncodingFilename’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
 }
 ^
Parser/tokenizer.c:1909:1: error: the control flow of function ‘PyTokenizer_FindEncodingFilename’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Parser/tokenizer.c: In function ‘tok_get’:
Parser/tokenizer.c:1909:1: error: the control flow of function ‘tok_get’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Parser/tokenizer.c:1909:1: error: the control flow of function ‘tok_get’ does not match its profile data (counter ‘single’) [-Werror=coverage-mismatch]
Parser/tokenizer.c:1909:1: error: the control flow of function ‘tok_get’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include   -fPIC -DPy_BUILD_CORE -o Objects/bytes_methods.o Objects/bytes_methods.c
Objects/boolobject.c: In function ‘bool_xor’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_xor’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
 };
 ^
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_xor’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_xor’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/boolobject.c: In function ‘bool_or’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_or’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_or’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_or’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/boolobject.c: In function ‘bool_and’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_and’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_and’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_and’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/boolobject.c: In function ‘bool_new’:
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_new’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/boolobject.c:185:1: error: the control flow of function ‘bool_new’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fprofile-use -fprofile-correction  -I. -I./Include   -fPIC -DPy_BUILD_CORE -o Objects/bytearrayobject.o Objects/bytearrayobject.c
cc1: some warnings being treated as errors
Makefile:1652: recipe for target 'Objects/boolobject.o' failed
make[1]: *** [Objects/boolobject.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Objects/abstract.c: In function ‘PyObject_GetIter’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_GetIter’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
 }
 ^
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_GetIter’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_GetIter’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_InPlaceRepeat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceRepeat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceRepeat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceRepeat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_InPlaceConcat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceConcat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceConcat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_InPlaceConcat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_Repeat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Repeat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Repeat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Repeat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PySequence_Concat’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Concat’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Concat’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PySequence_Concat’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PyObject_CopyData’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_CopyData’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_CopyData’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
Objects/abstract.c: In function ‘PyObject_LengthHint’:
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_LengthHint’ does not match its profile data (counter ‘arcs’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_LengthHint’ does not match its profile data (counter ‘indirect_call’) [-Werror=coverage-mismatch]
Objects/abstract.c:2642:1: error: the control flow of function ‘PyObject_LengthHint’ does not match its profile data (counter ‘time_profiler’) [-Werror=coverage-mismatch]
cc1: some warnings being treated as errors
Makefile:1652: recipe for target 'Parser/tokenizer.o' failed
make[1]: *** [Parser/tokenizer.o] Error 1
cc1: some warnings being treated as errors
Makefile:1652: recipe for target 'Objects/abstract.o' failed
make[1]: *** [Objects/abstract.o] Error 1
make[1]: Leaving directory '/home/ion/Documents/Python-3.7.7'
Makefile:534: recipe for target 'profile-opt' failed
make: *** [profile-opt] Error 2
```

what should i do to fix this? thank you very much.

best regards,
Albert Christianto
msg369351 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-19 12:58
> I need help for compiling and building Python-3.7.7. My system is Ubuntu 16.04 LTS, gcc 5.4.0 20160609

You have different options:

* Upgrade Ubuntu to the newer LTS to get a more recent C compiler
* Don't use PGO

Also, you might use a pre-built package rather than build Python yourself? See https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa
msg369804 - (view) Author: Albert Christianto (albertchristianto1994) Date: 2020-05-24 14:31
Sorry for my late response. 
Well, thank you very much for your fast response to help me. 
Actually, I have solved the problem in 3 hours after I posted my question. hehehe.
I found this tip about uncleaned files after executing "make clean" (https://bugs.python.org/msg346553). So, I decided to wipe out all python3.7 files from my computer and repeat the installation process. It worked!!
Anyway, thank you again for your response and your tip. It really made my day like I am not alone while doing some python program debugging. (^v^)
best regards,
Albert Christianto
History
Date User Action Args
2020-05-24 14:31:35albertchristianto1994setmessages: + msg369804
2020-05-19 12:58:31vstinnersetmessages: + msg369351
2020-05-19 09:05:25albertchristianto1994setnosy: + albertchristianto1994
messages: + msg369328
2020-02-10 23:56:08vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg361756
2018-01-08 17:33:30naschemesetmessages: + msg309680
2018-01-07 22:42:09vstinnersetmessages: + msg309643
2017-11-14 19:56:04naschemesetmessages: + msg306237
2017-11-13 23:23:36gregory.p.smithsetmessages: + msg306177
2017-11-13 13:07:44vstinnersetstatus: closed -> open
resolution: works for me -> (no value)
messages: + msg306157
2017-11-08 21:27:39gregory.p.smithsetstatus: open -> closed
type: performance -> compile error
resolution: works for me
stage: resolved
2017-11-08 21:12:58naschemesetmessages: + msg305906
2017-11-07 01:31:33naschemesetmessages: + msg305708
2017-11-07 01:14:47gregory.p.smithsetmessages: + msg305707
2017-11-06 23:11:06naschemesetmessages: + msg305687
2017-11-06 21:57:41vstinnercreate