Structure fails to correctly cast from a 2 byte bitfield.

From my very limited investigation, is looks like when using a member
type of less than the total size of the structure (or at least the size
of any byte boundaries) the casting is not done correctly?

I created this test.py and appended the results below:

---
import ctypes
import time

class closest_fit(ctypes.BigEndianStructure):
#    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_ubyte, 7),
                   ("Data1",   ctypes.c_ubyte, 8),
                   ]

class all_ulong(ctypes.BigEndianStructure):
#    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_ulonglong, 7),
                   ("Data1",   ctypes.c_ulonglong, 8),
                  ]

def castbytes(type):
    buffer = (ctypes.c_byte * 2)()
    buffer[0] = 0x55
    buffer[1] = 0x55
    return ctypes.cast(ctypes.pointer(buffer),
ctypes.POINTER(type)).contents

def print_members(test):
    print("Data0 is 0x%X, Data1 is 0x%X for %s"%(test.Data0, test.Data1,
test.__class__.__name__))

test_classes = [ closest_fit, all_ulonglong]

Failed = False
tests = [castbytes(type) for type in test_classes]
for test in tests:
    print_members(test)
    if not tests[0].Data0 == tests[1].Data0: 
        Failed = True
    if not tests[0].Data1 == tests[1].Data1: 
        Failed = True

if Failed:
    print("Failed")
else:
    print("Passed")
---

>c:\python25\python.exe test.py
Data0 is 0x2A, Data1 is 0x55 for closest_fit
Data0 is 0x2A, Data1 is 0xAA for all_ulonglong
Failed

>c:\python26\python.exe test.py
Data0 is 0x2A, Data1 is 0x55 for closest_fit
Data0 is 0x2A, Data1 is 0xAA for all_ulonglong
Failed

>c:\python30\python.exe test.py
Data0 is 0x2A, Data1 is 0x55 for closest_fit
Data0 is 0x2A, Data1 is 0xAA for all_ulonglong
Failed

As you can see the second member Data1, should be 0xAA, however when
using c_ubyte types for members the value is not offset by one bit.

As you can see using c_ulonglong for all members avoids this issue,
however this results in a read only structure (see Issue 6068).

I am using structures to cast CAN messages which are 8 bytes, with very
funky bit fields crossing all sorts of byte boundaries, so I essentially
expected that ctypes would provide a method for an arbitrary bit field
definition for use within python.  Hopefully this is just my bad ctypes
driving, or a simple fix.

After reading the documentation for ctypes (specifically "Bit fields are
only possible for integer fields" from section 16.15.1.12) I've updated
the test.

---
import ctypes
import time

class uint(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint, 31),
                   ("Data1",   ctypes.c_uint, 32),
                   ]

class ulonglong(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_ulonglong, 31),
                   ("Data1",   ctypes.c_ulonglong, 32),
                  ]

size_of_structures_in_bytes = 8

def castbytes(type):
    buffer = (ctypes.c_byte * size_of_structures_in_bytes)()
    for index in range(size_of_structures_in_bytes):
        buffer[index] = 0x55
    return ctypes.cast(ctypes.pointer(buffer),
ctypes.POINTER(type)).contents

def print_members(test):
    print("Data0 is 0x%X, Data1 is 0x%X for %s"%(test.Data0, test.Data1,
test.__class__.__name__))

test_classes = [ uint, ulonglong]

Failed = False
tests = [castbytes(type) for type in test_classes]
for test in tests:
    print_members(test)

if not tests[0].Data0 == tests[1].Data0 == 0x2AAAAAAA:
    Failed = True
    print("Data0 failed")
if not tests[0].Data1 == tests[1].Data1 == 0xAAAAAAAA:
    Failed = True
    print("Data1 failed")

if not Failed:
    print("Passed")

Another example of this:
---
import ctypes

correct_data_dict = {
                     'Data0' : 0x55555555,
                     'Data1' : 0x02,
                     'Data2' : 0x0AAA,
                     }

class closest_fit(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint8, 3),
                   ("Data2",   ctypes.c_uint16, 12),
                  ]


class closest_fit_min_16(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint16, 3),
                   ("Data2",   ctypes.c_uint16, 12),
                  ]

class closest_fit_min_32(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint32, 3),
                   ("Data2",   ctypes.c_uint32, 12),
                  ]

class uint32(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint32, 3),
                   ("Data2",   ctypes.c_uint32, 12),
                  ]

class uint64(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint64, 32),
                   ("Data1",   ctypes.c_uint64, 3),
                   ("Data2",   ctypes.c_uint64, 12),
                  ]

size_of_structures_in_bytes = 6

def castbytes(type):
    buffer = (ctypes.c_byte * size_of_structures_in_bytes)()
    for index in range(size_of_structures_in_bytes):
        buffer[index] = 0x55
    return ctypes.cast(ctypes.pointer(buffer),
ctypes.POINTER(type)).contents

def print_members(test):
    print("Data0 is 0x%X, Data1 is 0x%X, Data2 is 0x%X for
%s"%(test.Data0, test.Data1, test.Data2, test.__class__.__name__))



test_classes = [closest_fit, uint32, closest_fit_min_16,
closest_fit_min_32, uint64]

Test_Failed = False
tests = [castbytes(type) for type in test_classes]
for test in tests:
#    print_members(test)
    for data in correct_data_dict:
        if not test.__getattribute__(data) == correct_data_dict[data]:
            Test_Failed = True
            print("%s failed for %s, value was 0x%X but should have been
0x%X"%(data, test.__class__.__name__, test.__getattribute__(data),
correct_data_dict[data]))

if not Test_Failed:
    print("Passed")
---
>c:\python25\python.exe IssueNEW.py
Data2 failed for closest_fit, value was 0x550 but should have been 0xAAA

>c:\python26\python.exe IssueNEW.py
Data2 failed for closest_fit, value was 0x550 but should have been 0xAAA

>c:\python30\python.exe IssueNEW.py
Data2 failed for closest_fit, value was 0x550 but should have been 0xAAA

WinXP,32bit,3.1.2 result
Data0 is 0x2AAAAAAA, Data1 is 0x55555555 for uint
Data0 is 0x2AAAAAAA, Data1 is 0xAAAAAAAA for ulonglong
Data1 failed

but I do not understand ctypes enough to verify that test is correct.

I took a look at this and I believe behavior is correct on Windows, the issue is with the test. For example this test is failing:

class closest_fit(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint8, 3),
                   ("Data2",   ctypes.c_uint16, 12),
                  ]

But you also have this assumption when generating the test data:

size_of_structures_in_bytes = 6

I verified and this does not hold with MSVC compiler. Using VC++ 2005, this code

typedef struct Test {
	unsigned int x: 32; // uint_32 : 32
	unsigned char y: 3; // uint_8 : 3
	unsigned short int z: 12; // uint_16 : 12
} Test;

gives sizeof(Test) == 7. In Python, if you look at sizeof(closest_fit), it will also be 7.

Looking at cfield.c, seems this was taken into account when creating bit fields:

    if (bitsize /* this is a bitfield request */
        && *pfield_size /* we have a bitfield open */
#ifdef MS_WIN32
        /* MSVC, GCC with -mms-bitfields */
        && dict->size * 8 == *pfield_size
#else
        /* GCC */
        && dict->size * 8 <= *pfield_size
#endif
        && (*pbitofs + bitsize) <= *pfield_size) {
        /* continue bit field */
        fieldtype = CONT_BITFIELD;
#ifndef MS_WIN32
    } else if (bitsize /* this is a bitfield request */
        && *pfield_size /* we have a bitfield open */
        && dict->size * 8 >= *pfield_size
        && (*pbitofs + bitsize) <= dict->size * 8) {
        /* expand bit field */
        fieldtype = EXPAND_BITFIELD;
#endif
    } else if (bitsize) {
        /* start new bitfield */
        fieldtype = NEW_BITFIELD;
        *pbitofs = 0;
        *pfield_size = dict->size * 8;
 
Though I don't know this first-hand, above code plus sizeof experiment leads me to believe that gcc packs bitfields differently than MSVC. Seems that gcc will expand existing bitfield trying to pack structure more tightly so indeed on Linux (or I assume Windows gcc build), size of this structure is 6 as gcc will combine these seeing that an unsigned short can hold all 15 bits required but with MSVC this won't work. MSVC will allocate both the c_uint8 and the c_uint16 once is sees that last 12 bits don't fit in remaining c_uint8.

As far as I can tell this is by design and Python matches expected MSVC structure packing for this test case.

Opened http://bugs.python.org/issue12528 to address this.

Hmmm ...  Assuming  a native VC++ compiler on an x86 machine running Windows, then it doesn't make sense to validate these test cases in such an environment.  All the tests are all big-endian.

'ctypes' can't be expected to behave the same as the native compiler that compiled the Python interpreter for structures of non-native endianities produced by 'ctypes'.  That doesn't make sense.  The best we can do is document how 'ctypes' does handle non-native endianites on various platforms.

FWIW, I did try the first set of tests (http://bugs.python.org/msg88145) with GCC for a 32-bit MIPS ELF target using the following test case:

#include <stdio.h>

struct T {
  unsigned int x : 31;
  unsigned int y : 32;
};

struct S {
  unsigned long long x : 31;
  unsigned long long y : 32;
};

int main (int argc, char **argv)
{
  unsigned char buf[8] = {0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55};
  struct T *t = (struct T*)&buf;
  struct S *s = (struct S*)&buf;

  printf ("%X, %X\n", t->x, t->y);
  printf ("%X, %X\n", s->x, s->y);
}


The test output:

Data0 is 0x2AAAAAAA, Data1 is 0x55555555 for uint
Data0 is 0x2AAAAAAA, Data1 is 0xAAAAAAAA for ulonglong

is correct with respect to that environment.

The difference in the first case (uint) and the second case (ulonglong) is that the first is placed into two 4-byte unsigned integer units where as the second is placed into one 8-byte unsigned long long unit.

I am slightly confused how issue12528 is going to address this, when there seems to be no bug;  only what seems to be a test case problem.  I think we should close this issue out.

Another issue should be opened to enhance the documentation, though.  We should document exactly how 'ctypes' does the structure layout for different endianities on different platforms.  Something similar to how VC++ documents this ( http://msdn.microsoft.com/en-us/library/ewwyfdbe(v=vs.71).aspx ).

Meador, I believe this was the first issue on the tracker that got me looking into bitfield allocation. I agree that big-endian on MSVC doesn't make too much sense but you can disregard that - using default endianess will still yield different sizes of bitfields when compiled with GCC and MSVC.

Basically bitfield allocation is compiler specific and patch in issue12528 implements a way to select which allocation strategy to be used at runtime instead of hardcoding the one with which Python is compiled. This should improve cross-compiler interop. I wanted to hyperlink that patch to all other bitfield bugs, that's why I followed up with link to the patch.

Feel free to close this, either as not an issue or as a duplicate of issue12528. 

And yes, this bit about bitfield allocation should be documented and I was planning to look into it at some point after 12528 gets committed.

On Thu, Sep 1, 2011 at 9:45 AM, Vlad Riscutia <report@bugs.python.org> wrote:

> Vlad Riscutia <riscutiavlad@gmail.com> added the comment:
>
> Meador, I believe this was the first issue on the tracker that got me looking into bitfield allocation.
> I agree that big-endian on MSVC doesn't make too much sense but you can disregard that - using default endianess will still yield
> different sizes of bitfields when compiled with GCC and MSVC.

Sure, but this particular issue is purporting that the layout of the
structure is incorrect, not that the size is.

> Basically bitfield allocation is compiler specific and patch in issue12528 implements a way to select which
> allocation strategy to be used at runtime instead of hardcoding the one with which Python is compiled. This
> should improve cross-compiler interop. I wanted to hyperlink that patch to all other bitfield bugs, that's why I
> followed up with link to the patch.

Yes, it is very compiler specific.  I have some thoughts about making
this configurable, but I will comment on issue12528 with those.

> Feel free to close this, either as not an issue or as a duplicate of issue12528.

I will open a documentation bug and close this one out.

Sounds good. Please nosy me in the doc bug.

I opened issue12880 for the doc bug.  Closing this one out ...