classification
Title: GCC 4.6 bug with -ftree-vectorize
Type: compile error Stage:
Components: Build Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Tom.Browder, amaury.forgeotdarc, barry, georg.brandl, pitrou, skrah, vstinner
Priority: normal Keywords:

Created on 2010-09-16 22:02 by Tom.Browder, last changed 2011-01-12 22:26 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
os.pyc Tom.Browder, 2010-09-16 23:01
Messages (20)
msg116623 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-16 22:02
I am trying to rebuild the 2.7 maintenance branch and get this error
on Ubuntu 10.04.1 LTS:

XXX lineno: 743, opcode: 0
Traceback (most recent call last):
 File "/usr/local/src/python-2.7-maint-svn/Lib/site.py", line 62, in <module>
  import os
 File "/usr/local/src/python-2.7-maint-svn/Lib/os.py", line 743, in <module>
  def urandom(n):
SystemError: unknown opcode

I installed it successfully once so I may be getting conflicts, but I can't figure out why.  There were some similar bugs reported in previous versions but I didn't see a clear solution.

I have done "make distclean" and "./configure".  I have unset my
PYTHONPATH and LD_LIBRARY_PATH, but python2.7 is my default python from a previous good build and installation.
msg116628 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-09-16 22:22
Maybe a corrupted .pyc file? Can you please attach the file
/usr/local/src/python-2.7-maint-svn/Lib/os.pyc
msg116637 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-16 23:01
File attached as requested.
msg116639 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2010-09-16 23:12
Note that I tried to reproduce this on my 10.04.1 system, with various states of Python 2.7 installed, etc. and was unable to reproduce it.  It's certainly odd that you're getting the SystemError on that line.
msg116647 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-09-17 07:09
For some reason your file os.pyc is corrupted:
I loaded it with a working python2.7:

import marshal, dis
pyc = open('bados.pyc', 'rb').read()
code = marshal.loads(pyc[8:])
dis.dis(code)

And it appears that all the "jump" instructions are wrong: the address is either 0, or something around 65536.

Which compiler do you use? I remember subtle bugs caused by some version of gcc.
msg116650 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-17 10:29
I'm using gcc-4.5.1.  I'll try an older version: gcc (Ubuntu 4.4.3-4ubuntu5) 4.4.3.
msg116651 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-17 10:44
The build succeeded with the older version of gcc.  I either have a mis-compiled gcc-4.5.1 (but the same version on another host worked okay) or gcc has a very subtle bug.  I think this issue can be considered closed; however, it may be worth a note in a FAQ or build instructions.  Good catch Amaury!
msg116652 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-17 10:51
Correction on the bad gcc compiler: the actual version was a non-released version off the gcc-4.6 branch:  gcc version 4.6.0 20100908 (experimental) (GCC).  I'm filing a bug with gcc.  Sorry for the wasted time.
msg116654 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-09-17 11:08
You're welcome.
Please join a link to the gcc bug when you have one.
msg116657 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-17 11:37
Here is a link to the thread I started on the gcc-help mailing list concerning the issue: 

  http://gcc.gnu.org/ml/gcc-help/2010-09/msg00170.html

If I don't get a successful build with the current gcc trunk, I imagine this thread will transfer to the gcc (dev) list--I'll try to keep the links up.
msg116872 - (view) Author: Tom Browder (Tom.Browder) Date: 2010-09-19 14:46
I'm getting no interest from the gcc group at the moment.  I would like to help track down the bug and will nose around the Python source to zero in on where the problem is for detailed debugging.  Can anyone point me to the key files that concern the byte compiler?
msg116873 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-19 14:52
You can look at Python/compile.c (the ast-to-bytecode compiler), Python/peephole.c (the bytecode optimizer), Python/marshal.c (the on-disk bytecode serializer/unserialize).

First step could be to actually disable the optimizer (see PyCode_Optimize in peephole.c).
msg125808 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2011-01-08 23:46
I narrowed this issue down to -ftree-vectorize, which is part of
-O3.
msg126091 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-12 13:08
With gcc-4.6 -O1, Python 3.2 works correctly. With gcc-4.6 -O1 -ftree-vectorize, the assertion "assert (new_line - last_line < 255);" fails in PyCode_Optimize().

I think that the problem is in the following loop:
-------------------------------------
#define NOP    9
#define HAVE_ARGUMENT  90  /* Opcodes from here have an argument: */
#define HAS_ARG(op) ((op) >= HAVE_ARGUMENT)
#define CODESIZE(op)  (HAS_ARG(op) ? 3 : 1)

unsigned char *codestr = NULL;
Py_ssize_t codelen;
int *addrmap = NULL;

/* Fixup linenotab */
for (i=0, nops=0 ; i<codelen ; i += CODESIZE(codestr[i])) {
    addrmap[i] = i - nops;
    if (codestr[i] == NOP)
        nops++;
}
-------------------------------------

gcc-4.6 -O1:
-------------------------------------
0x0000000000480991 <+5041>:  mov    %eax,%edx
0x0000000000480993 <+5043>:  sub    %esi,%edx
0x0000000000480995 <+5045>:  mov    %edx,(%r12,%rax,4)
0x0000000000480999 <+5049>:  movzbl 0x0(%rbp,%rax,1),%edx
0x000000000048099e <+5054>:  cmp    $0x9,%dl
0x00000000004809a1 <+5057>:  jne    0x4809a8 <PyCode_Optimize+5064>
0x00000000004809a3 <+5059>:  add    $0x1,%esi
0x00000000004809a6 <+5062>:  jmp    0x4809b2 <PyCode_Optimize+5074>
0x00000000004809a8 <+5064>:  mov    $0x3,%ecx
0x00000000004809ad <+5069>:  cmp    $0x59,%dl
0x00000000004809b0 <+5072>:  ja     0x4809b7 <PyCode_Optimize+5079>
0x00000000004809b2 <+5074>:  mov    $0x1,%ecx
0x00000000004809b7 <+5079>:  add    %rcx,%rax
0x00000000004809ba <+5082>:  cmp    %rax,%rdi
0x00000000004809bd <+5085>:  jg     0x480991 <PyCode_Optimize+5041>
-------------------------------------

gcc-4.6 -O1 -ftree-vectorize
-------------------------------------
0x0000000000480991 <+5041>:  mov    %eax,%ecx
0x0000000000480993 <+5043>:  sub    %edx,%ecx
0x0000000000480995 <+5045>:  mov    %ecx,(%r12,%rax,4)
0x0000000000480999 <+5049>:  movzbl 0x0(%rbp,%rax,1),%ecx
0x000000000048099e <+5054>:  lea    0x1(%rdx),%esi
0x00000000004809a1 <+5057>:  cmp    $0x9,%cl
0x00000000004809a4 <+5060>:  cmovne %edx,%esi
0x00000000004809a7 <+5063>:  cmove  %esi,%edx
0x00000000004809aa <+5066>:  setne  %cl
0x00000000004809ad <+5069>:  movzbl %cl,%ecx
0x00000000004809b0 <+5072>:  lea    0x1(%rax,%rcx,2),%rax
0x00000000004809b5 <+5077>:  cmp    %rax,%rdi
0x00000000004809b8 <+5080>:  jg     0x480991 <PyCode_Optimize+5041>
-------------------------------------
msg126092 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-12 13:11
More info about the loop:
----
Py_ssize_t i;
int nops;
----

My setup:
 * Intel(R) Pentium(R) 4 CPU 3.00GHz
 * Debian Sid
 * gcc (Debian 20110106-1) 4.6.0 20110106 (experimental) [trunk revision 168538]
msg126097 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-12 13:32
[...]
> 
> gcc-4.6 -O1 -ftree-vectorize
> -------------------------------------
> 0x0000000000480991 <+5041>:  mov    %eax,%ecx
> 0x0000000000480993 <+5043>:  sub    %edx,%ecx
> 0x0000000000480995 <+5045>:  mov    %ecx,(%r12,%rax,4)
> 0x0000000000480999 <+5049>:  movzbl 0x0(%rbp,%rax,1),%ecx
> 0x000000000048099e <+5054>:  lea    0x1(%rdx),%esi
> 0x00000000004809a1 <+5057>:  cmp    $0x9,%cl
> 0x00000000004809a4 <+5060>:  cmovne %edx,%esi
> 0x00000000004809a7 <+5063>:  cmove  %esi,%edx
> 0x00000000004809aa <+5066>:  setne  %cl
> 0x00000000004809ad <+5069>:  movzbl %cl,%ecx
> 0x00000000004809b0 <+5072>:  lea    0x1(%rax,%rcx,2),%rax
> 0x00000000004809b5 <+5077>:  cmp    %rax,%rdi
> 0x00000000004809b8 <+5080>:  jg     0x480991 <PyCode_Optimize+5041>
> -------------------------------------

Looks like code generation is borked.
(it lacks a compare to 0x59 or 0x5a (90), and "setne" wouldn't be the
right instruction anyway)
msg126098 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-12 13:38
> Looks like code generation is borked.

I sent an email to the mailing list:
http://gcc.gnu.org/ml/gcc-help/2011-01/msg00136.html
msg126121 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-12 18:38
I reported the bug to GCC:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47271
msg126146 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-01-12 22:24
Do we have to keep this open?
msg126147 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-01-12 22:26
> Do we have to keep this open?

Nope, let's close it.
History
Date User Action Args
2011-01-12 22:26:02vstinnersetstatus: open -> closed
nosy: barry, georg.brandl, amaury.forgeotdarc, pitrou, vstinner, skrah, Tom.Browder
messages: + msg126147
2011-01-12 22:24:02georg.brandlsetnosy: + georg.brandl
messages: + msg126146
2011-01-12 18:38:15vstinnersetnosy: barry, amaury.forgeotdarc, pitrou, vstinner, skrah, Tom.Browder
messages: + msg126121
title: Python 2.7 Won't Build: SystemError: unknown opcode -> GCC 4.6 bug with -ftree-vectorize
2011-01-12 13:38:17vstinnersetnosy: barry, amaury.forgeotdarc, pitrou, vstinner, skrah, Tom.Browder
messages: + msg126098
2011-01-12 13:32:07pitrousetnosy: barry, amaury.forgeotdarc, pitrou, vstinner, skrah, Tom.Browder
messages: + msg126097
2011-01-12 13:11:34vstinnersetnosy: barry, amaury.forgeotdarc, pitrou, vstinner, skrah, Tom.Browder
messages: + msg126092
2011-01-12 13:08:34vstinnersetnosy: + vstinner
messages: + msg126091
2011-01-08 23:46:14skrahsetnosy: barry, amaury.forgeotdarc, pitrou, skrah, Tom.Browder
messages: + msg125808
2011-01-08 22:14:52skrahsetnosy: + skrah
2010-09-19 14:52:48pitrousetnosy: + pitrou
messages: + msg116873
2010-09-19 14:46:17Tom.Browdersetmessages: + msg116872
2010-09-17 11:37:40Tom.Browdersetstatus: pending -> open

messages: + msg116657
2010-09-17 11:08:24amaury.forgeotdarcsetstatus: open -> pending
resolution: not a bug
messages: + msg116654
2010-09-17 10:51:34Tom.Browdersetmessages: + msg116652
2010-09-17 10:44:49Tom.Browdersetmessages: + msg116651
2010-09-17 10:29:10Tom.Browdersetmessages: + msg116650
2010-09-17 07:09:07amaury.forgeotdarcsetmessages: + msg116647
2010-09-16 23:12:25barrysetmessages: + msg116639
2010-09-16 23:01:39Tom.Browdersetfiles: + os.pyc

messages: + msg116637
2010-09-16 22:22:31amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg116628
2010-09-16 22:03:54Tom.Browdersetnosy: + barry
2010-09-16 22:02:06Tom.Browdercreate