classification
Title: ctypes: Strange bitfield structure sizing issue
Type: behavior Stage: resolved
Components: ctypes Versions: Python 3.1, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Implement configurable bitfield allocation strategy
View: 12528
Assigned To: Nosy List: Steve.Thompson, amaury.forgeotdarc, meador.inge, santoso.wijaya, vladris
Priority: normal Keywords:

Created on 2011-04-25 16:52 by Steve.Thompson, last changed 2011-09-02 02:42 by meador.inge. This issue is now closed.

Files
File name Uploaded Description Edit
bitfields.c amaury.forgeotdarc, 2011-04-25 18:57
unnamed Steve.Thompson, 2011-04-25 21:21
Messages (6)
msg134393 - (view) Author: Steve Thompson (Steve.Thompson) Date: 2011-04-25 16:52
Consider the following:
import ctypes

class struct1( ctypes.Structure ):
    _pack_ = 1
    _fields_ =  [
                    ( "first",  ctypes.c_uint8,  1  ),
                    ( "second", ctypes.c_uint8,  1  ),
                    ( "third",  ctypes.c_uint8,  1  ),
                    ( "fourth", ctypes.c_uint8,  1  ),
                    ( "fifth",  ctypes.c_uint8,  1  ),
                    ( "pad",    ctypes.c_uint16, 11 ),
                ]
                
s1 = struct1()
print ctypes.sizeof( s1 )

class struct2( ctypes.Structure ):
    _pack_ = 1
    _fields_ =  [
                    ( "first",  ctypes.c_uint16, 1  ),
                    ( "second", ctypes.c_uint16, 1  ),
                    ( "third",  ctypes.c_uint16, 1  ),
                    ( "fourth", ctypes.c_uint16, 1  ),
                    ( "fifth",  ctypes.c_uint16, 1  ),
                    ( "pad",    ctypes.c_uint16, 11 ),
                ]
                
s2 = struct2()
print ctypes.sizeof( s2 )

The output is:
3
2

I'm generating python code from real c code.  The compiler I'm using for the real c code packs both of these structures into two bytes.  I need a way to make the first example work in python like the compiler without having to modify the source code.

Is this possible?
msg134402 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-04-25 18:57
The output is: "3 2" on windows. It is "2 2" on the linux 64bit I tried.
This is consistent with what the usual compilers do on these platforms: MSVC on windows, and gcc on linux.

The specification of the C language specifies that """If an adjacent bitfield will not fit into the remainder of the unit, the implementation defines whether bitfields are allowed to span units or whether another unit is allocated for the second bitfield"""

Are you using gcc on windows? In this case I suggest to always use the largest type to declare the bitfields.
msg134409 - (view) Author: Santoso Wijaya (santoso.wijaya) * Date: 2011-04-25 19:49
What compilers were used to build your Python distro and the native structure?

I found out in _ctypes/cfield.c (lns. 76-95):

    if (bitsize /* this is a bitfield request */
        && *pfield_size /* we have a bitfield open */
#ifdef MS_WIN32
        /* MSVC, GCC with -mms-bitfields */
        && dict->size * 8 == *pfield_size
#else
        /* GCC */
        && dict->size * 8 <= *pfield_size
#endif
        && (*pbitofs + bitsize) <= *pfield_size) {
        /* continue bit field */
        fieldtype = CONT_BITFIELD;
#ifndef MS_WIN32
    } else if (bitsize /* this is a bitfield request */
        && *pfield_size /* we have a bitfield open */
        && dict->size * 8 >= *pfield_size
        && (*pbitofs + bitsize) <= dict->size * 8) {
        /* expand bit field */
        fieldtype = EXPAND_BITFIELD;
#endif

So the allocation of the extra byte for the structure seems to depend on Python's compiler. To make sure, I compiled a native structure using MSVC:

#pragma pack(1)
typedef struct _struct1
{
    UINT8 first   : 1;
    UINT8 second  : 1;
    UINT8 third   : 1;
    UINT8 fourth  : 1;
    UINT8 fifth   : 1;
    UINT16 pad    : 11;
} struct1;

And I got the same value (sizeof == 3).
msg134414 - (view) Author: Steve Thompson (Steve.Thompson) Date: 2011-04-25 21:21
So, knowing there's a potential cross platform inconsistency here, is there
a proposed way to deal with this that doesn't involve modifying the real c
code I'm interfacing with?  That's not always an option.

On Mon, Apr 25, 2011 at 2:49 PM, Santoso Wijaya <report@bugs.python.org>wrote:

>
> Santoso Wijaya <santoso.wijaya@gmail.com> added the comment:
>
> What compilers were used to build your Python distro and the native
> structure?
>
> I found out in _ctypes/cfield.c (lns. 76-95):
>
>    if (bitsize /* this is a bitfield request */
>        && *pfield_size /* we have a bitfield open */
> #ifdef MS_WIN32
>        /* MSVC, GCC with -mms-bitfields */
>        && dict->size * 8 == *pfield_size
> #else
>        /* GCC */
>        && dict->size * 8 <= *pfield_size
> #endif
>        && (*pbitofs + bitsize) <= *pfield_size) {
>        /* continue bit field */
>        fieldtype = CONT_BITFIELD;
> #ifndef MS_WIN32
>    } else if (bitsize /* this is a bitfield request */
>        && *pfield_size /* we have a bitfield open */
>        && dict->size * 8 >= *pfield_size
>        && (*pbitofs + bitsize) <= dict->size * 8) {
>        /* expand bit field */
>        fieldtype = EXPAND_BITFIELD;
> #endif
>
> So the allocation of the extra byte for the structure seems to depend on
> Python's compiler. To make sure, I compiled a native structure using MSVC:
>
> #pragma pack(1)
> typedef struct _struct1
> {
>    UINT8 first   : 1;
>    UINT8 second  : 1;
>    UINT8 third   : 1;
>    UINT8 fourth  : 1;
>    UINT8 fifth   : 1;
>    UINT16 pad    : 11;
> } struct1;
>
> And I got the same value (sizeof == 3).
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue11920>
> _______________________________________
>
msg140089 - (view) Author: Vlad Riscutia (vladris) Date: 2011-07-10 20:02
Opened http://bugs.python.org/issue12528 to address this.
msg143371 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-02 02:42
Hi Steve,

There is currently no way to deal with this situation.  Vlad has opened issue12528 with a proposal to make the allocation strategy configurable from the 'ctypes' API.  Please follow that issue if you are still interested.  I am closing this issue.
History
Date User Action Args
2011-09-02 02:42:13meador.ingesetstatus: open -> closed

superseder: Implement configurable bitfield allocation strategy

nosy: + meador.inge
messages: + msg143371
resolution: duplicate
stage: resolved
2011-07-10 20:02:27vladrissetnosy: + vladris
messages: + msg140089
2011-04-25 21:21:56Steve.Thompsonsetfiles: + unnamed

messages: + msg134414
2011-04-25 19:49:33santoso.wijayasetmessages: + msg134409
2011-04-25 18:57:43amaury.forgeotdarcsetfiles: + bitfields.c
nosy: + amaury.forgeotdarc
messages: + msg134402

2011-04-25 18:18:07santoso.wijayasetnosy: + santoso.wijaya

type: behavior
versions: + Python 3.1, Python 2.7, Python 3.2, Python 3.3, - Python 2.6
2011-04-25 16:52:55Steve.Thompsoncreate