classification
Title: double Endian problem and more on arm
Type: behavior Stage: resolved
Components: ctypes Versions: Python 2.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: mancausoft, mark.dickinson, meador.inge
Priority: normal Keywords:

Created on 2009-10-24 23:52 by mancausoft, last changed 2011-09-16 12:00 by mark.dickinson. This issue is now closed.

Messages (12)
msg94444 - (view) Author: Mancausoft (mancausoft) Date: 2009-10-24 23:52
I compile python for arm (on debian etch) but it don't  pass ctype
test:

======================================================================
FAIL: test_struct_return_2H
(ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase)
----------------------------------------------------------------------                    
Traceback (most recent call last):                                                        
  File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_as_parameter.py", line
171, in test_struct_return_2H
    self.failUnlessEqual((s2h.x, s2h.y), (99*2, 88*3))                                                     
AssertionError: (99, 88) != (198, 264)                                                                     

======================================================================
FAIL: test_struct_return_2H
(ctypes.test.test_as_parameter.AsParamWrapperTestCase)
----------------------------------------------------------------------            
Traceback (most recent call last):                                                
  File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_as_parameter.py", line
171, in test_struct_return_2H
    self.failUnlessEqual((s2h.x, s2h.y), (99*2, 88*3))                                                     
AssertionError: (99, 88) != (198, 264)                                                                     

======================================================================
FAIL: test_struct_return_2H
(ctypes.test.test_as_parameter.BasicWrapTestCase)
----------------------------------------------------------------------       
Traceback (most recent call last):                                           
  File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_as_parameter.py", line
171, in test_struct_return_2H
    self.failUnlessEqual((s2h.x, s2h.y), (99*2, 88*3))
AssertionError: (99, 88) != (198, 264)

======================================================================
FAIL: test_endian_double (ctypes.test.test_byteswap.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_byteswap.py",
line 137, in test_endian_double
    self.failUnlessEqual(bin(struct.pack("<d", math.pi)), bin(s))
AssertionError: '182D4454FB210940' != 'FB210940182D4454'

======================================================================
FAIL: test_unaligned_native_struct_fields
(ctypes.test.test_byteswap.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_byteswap.py",
line 277, in test_unaligned_native_struct_fields
    self.failUnlessEqual(bin(s1), bin(s2))
AssertionError: '123412007856341200B81E09401F85EB51' !=
'1234120078563412001F85EB51B81E0940'
msg94448 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-10-25 08:11
The 4th failure (test_endian_double) probably has nothing to do with 
ctypes.  See also issue #1762561.
msg94450 - (view) Author: Mancausoft (mancausoft) Date: 2009-10-25 12:40
Mark Dickinson <report@bugs.python.org> scrisse:

> The 4th failure (test_endian_double) probably has nothing to do with 
> ctypes.  See also issue #1762561.

I try to use the patch arm-float2.diff, but test result is the same: 

======================================================================
FAIL: test_struct_return_2H
(ctypes.test.test_as_parameter.AsParamPropertyWrapperTestCase)
----------------------------------------------------------------------
Traceback (most recent call last): File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_as_parameter.py", line
171, in test_struct_return_2H self.failUnlessEqual((s2h.x, s2h.y),
(99*2, 88*3)) AssertionError: (99, 88) != (198,
264)                                                                     

======================================================================
FAIL: test_struct_return_2H
(ctypes.test.test_as_parameter.AsParamWrapperTestCase)
----------------------------------------------------------------------
Traceback (most recent call last): File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_as_parameter.py", line
171, in test_struct_return_2H self.failUnlessEqual((s2h.x, s2h.y),
(99*2, 88*3)) AssertionError: (99, 88) != (198,
264)                                                                     

======================================================================
FAIL: test_struct_return_2H
(ctypes.test.test_as_parameter.BasicWrapTestCase)
----------------------------------------------------------------------
Traceback (most recent call last): File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_as_parameter.py", line
171, in test_struct_return_2H self.failUnlessEqual((s2h.x, s2h.y),
(99*2, 88*3)) AssertionError: (99, 88) != (198, 264)

======================================================================
FAIL: test_endian_double (ctypes.test.test_byteswap.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_byteswap.py",
line 137, in test_endian_double
self.failUnlessEqual(bin(struct.pack("<d", math.pi)), bin(s))
AssertionError: '182D4454FB210940' != 'FB210940182D4454'

======================================================================
FAIL: test_unaligned_native_struct_fields
(ctypes.test.test_byteswap.Test)
----------------------------------------------------------------------
Traceback (most recent call last): File
"/mnt/root/stackless-2.6.3/Lib/ctypes/test/test_byteswap.py", line 277,
in test_unaligned_native_struct_fields self.failUnlessEqual(bin(s1),
bin(s2)) AssertionError: '123412007856341200B81E09401F85EB51' !=
'1234120078563412001F85EB51B81E0940'

----------------------------------------------------------------------
Ran 324 tests in 8.525s

Mancausoft
msg94460 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-10-25 20:22
Hmm.  Okay, I take it back then. :)  Sorry for the noise.
msg143883 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-12 04:21
I ran the ctypes tests on Debian GNU/Linux 5.0.8 (lenny) on an ARMv5tejl 
Versatile kernel and everything passed.  Is anyone else still seeing 
errors?
msg143894 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2011-09-12 12:10
> I ran the ctypes tests on Debian GNU/Linux 5.0.8 (lenny) on an ARMv5tejl 
> Versatile kernel and everything passed.

I believe the problem is specific to machines still using the old ABI ('OABI').  Which ABI was being used on your test machine? (Not sure how to tell, but there's more information about the ABIs at:

http://wiki.debian.org/ArmEabiPort

)
msg143924 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-12 17:40
On Mon, Sep 12, 2011 at 7:10 AM, Mark Dickinson <report@bugs.python.org> wrote:

> Mark Dickinson <dickinsm@gmail.com> added the comment:
>
> I believe the problem is specific to machines still using the old ABI ('OABI').  Which ABI was being used on your test machine?

I tested the new ABI (armel).  I will try the old ABI.  However, after
reading over the ABI differences, the problems seem to be expected.
In particular:

"""
Struct packing and alignment

With the new ABI, default structure packing changes, as do some
default data sizes and alignment (which also have a knock-on effect on
structure packing). In particular the minimum size and alignment of a
structure was 4 bytes. Under the EABI there is no minimum and the
alignment is determined by the types of the components it contains.
This will break programs that know too much about the way structures
are packed and can break code that writes binary files by dumping and
reading structures.
"""

Once I get an OABI system up and running I will substantiate that
claim.  I don't think there is going to be a bug fix here as I don't
think it is practical to support both ABIs.  Just a "these tests are
expected to fail due to ABI differences x, y, z" kind of thing.
msg143956 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2011-09-13 11:52
> I don't think it is practical to support both ABIs.

I suspect you're right.
msg144032 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-14 17:29
OK, I got an OABI system setup.  I am seeing the 'test_struct_return_2H' 
failure, which actually segfaults in my setup.  The difference does, 
indeed, seem like an ABI mismatch.

The test code that is failing has a Python side like:

   def test_struct_return_2H(self):
        class S2H(Structure):
            _fields_ = [("x", c_short),
                        ("y", c_short)]
        dll.ret_2h_func.restype = S2H
        dll.ret_2h_func.argtypes = [S2H]
        inp = S2H(99, 88)
        s2h = dll.ret_2h_func(inp)
        self.assertEqual((s2h.x, s2h.y), (99*2, 88*3))

and a C code side that looks like:

   typedef struct {
      short x;
      short y;
   } S2H;

   S2H ret_2h_func(S2H inp)
   {
      inp.x *= 2;
      inp.y *= 3;
      return inp;
   }

The APCS Section 5.4 Result Return [1], says:

"""
A Composite Type not larger than 4 bytes is returned in r0.  The format 
is as if the result had been stored in memory at a word-aligned address 
and then loaded into r0 with an LDR instruction.  Any bits in r0 that 
lie outside the bounds of the result have unspecified values.
"""

The EABI implementation does exactly this and packs the structure into r0, where as the OABI implementation places the address of a structure in r0.  'ctypes' is assuming the former and on an OABI system the contents of r0 are treated as an address, where they are actually a value.  Boom goes the dynamite.

I am looking into 'test_endian_double' and 
'test_unaligned_native_struct_fields' now.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.ihi0042d/IHI0042D_aapcs.pdf
msg144033 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-14 18:06
The 'test_endian_double' test fails because the 'double' floating-point 
type for an interpreter built for OABI is unknown:

>>> float.__getformat__("float")
'IEEE, little-endian'
>>> float.__getformat__("double")
'unknown'

According to [1], the double format discrepancies seem to be expected.

[1] http://wiki.debian.org/ArmEabiPort#ARM_floating_points
msg144113 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-16 02:06
I think Mark's original pointer to issue1762561 was right on.  The last
two cases are failing due to the mixed-endian format (mentioned
in that issue) used in OABI.

You can see this in the output of 'test_endian_double':

   '182D4454FB210940' != 'FB210940182D4454'

Note that the values are the same except the two 32-bit words are
swapped.

Similarly, in 'test_unaligned_native_struct_fields':

   '123412007856341200B81E09401F85EB51' !=
   '1234120078563412001F85EB51B81E0940'

The first 8 bytes in each case are the same.   The last 8 bytes of
each (which represent floating-point doubles) are the same except,
again, the words are swapped.

I am going to close this out as "won't fix".  As mentioned in issue1762561, supporting OABI will involve taking on another
host platform.  EABI is surely more predominant these days anyway.
msg144127 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2011-09-16 12:00
"won't fix" seems reasonable to me.
History
Date User Action Args
2011-09-16 12:00:59mark.dickinsonsetstatus: open -> closed
2011-09-16 12:00:15mark.dickinsonsetmessages: + msg144127
2011-09-16 02:06:15meador.ingesettype: behavior
resolution: wont fix
messages: + msg144113
stage: resolved
2011-09-14 18:06:24meador.ingesetmessages: + msg144033
2011-09-14 17:29:05meador.ingesetmessages: + msg144032
2011-09-13 11:52:51mark.dickinsonsetmessages: + msg143956
2011-09-12 17:40:10meador.ingesetmessages: + msg143924
2011-09-12 12:10:12mark.dickinsonsetmessages: + msg143894
2011-09-12 04:21:46meador.ingesetassignee: theller ->
messages: + msg143883
nosy: + meador.inge, - theller
2009-10-25 20:22:59mark.dickinsonsetmessages: + msg94460
2009-10-25 12:40:30mancausoftsetmessages: + msg94450
2009-10-25 08:11:32mark.dickinsonsetnosy: + mark.dickinson
messages: + msg94448
2009-10-24 23:52:39mancausoftcreate