This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ctypes: Differing results between Python and C.
Type: Stage: resolved
Components: ctypes Versions: Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: ctype with packed bitfields does not match native compiler
View: 15453
Assigned To: Nosy List: Geoff.Clements, eryksun, martin.panter, meador.inge, pitrou
Priority: normal Keywords:

Created on 2014-11-01 17:16 by Geoff.Clements, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
dvdtest.tar.gz Geoff.Clements, 2014-11-01 17:16 Tar file of python and C code to reproduce the problem
Messages (6)
msg230453 - (view) Author: Geoff Clements (Geoff.Clements) Date: 2014-11-01 17:16
I have used ctypes to create a binding to libdvdread but I am getting some bad values back from libdvdread when using python. I have used ctypeslib to generate the python bindings, i.e. h2xml and xml2py and, as far as I can tell, the bind is a faithful representation of the h files.

In order to demonstrate this I have created a small C program and a small Python3 program both doing the same thing. When run they should produce the same output but they do not. In order to run these you need a DVD inserted in a DVD drive, I've hard-coded the device path to the DVD drive as /dev/sr0 in both programs so this may need to be changed.
msg230457 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-11-01 19:28
This doesn't directly address your issue, but ctypeslib doesn't seem to have received any changes since 2009 (*), and has never had an official release. For a non-trivial binding, you might be better served using something providing type safety, such as Cython.

(*) (which makes it possible that ctypeslib produces bindings not compatible with Python 3, by the way)
msg230460 - (view) Author: Geoff Clements (Geoff.Clements) Date: 2014-11-01 21:34
Thanks for the comment Antoine.

I understand that ctypeslib is old but it produces relatively mundane python code which is python version agnostic. Well, that's how it looks to me.

I have not used cython mainly because it would make the distribution of my code more complex but it may be worth trying if only to point the issue at ctypes or otherwise.
msg230468 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2014-11-01 22:38
ctypes doesn't always handle bitfields correctly. In particular it doesn't adapt the storage unit size when packed. A bitfield based on c_uint is always stored in 4 bytes, even if _pack_ is set to 1. This is MSVC rules, so all is well on Windows. 

For example, from ifo_types.h:

    typedef struct {
      unsigned int zero_1                    : 1;
      unsigned int multi_or_random_pgc_title : 1; /* 0: one sequential pgc title */
      unsigned int jlc_exists_in_cell_cmd    : 1;
      unsigned int jlc_exists_in_prepost_cmd : 1;
      unsigned int jlc_exists_in_button_cmd  : 1;
      unsigned int jlc_exists_in_tt_dom      : 1;
      unsigned int chapter_search_or_play    : 1; /* UOP 1 */
      unsigned int title_or_time_play        : 1; /* UOP 0 */
    } ATTRIBUTE_PACKED playback_type_t;

ctypeslib translates this as follows:

    class playback_type_t(Structure):
        pass
    playback_type_t._fields_ = [
        ('zero_1', c_uint, 1),
        ('multi_or_random_pgc_title', c_uint, 1),
        ('jlc_exists_in_cell_cmd', c_uint, 1),
        ('jlc_exists_in_prepost_cmd', c_uint, 1),
        ('jlc_exists_in_button_cmd', c_uint, 1),
        ('jlc_exists_in_tt_dom', c_uint, 1),
        ('chapter_search_or_play', c_uint, 1),
        ('title_or_time_play', c_uint, 1),
    ]

It doesn't set _pack_ = 1, but ctypes wouldn't use that to pack the c_uint bitfield anyway. It's 4 bytes, either way. MSVC agrees, even with #pragma pack(1). OTOH, with __attribute__((packed)), gcc packs the bitfield into a single byte. 

The incorrect packing (for gcc) of playback_type_t results in an incorrect offset for title_set_nr in title_info_t:

    class title_info_t(Structure):
        pass
    title_info_t._pack_ = 1
    title_info_t._fields_ = [
        ('pb_ty', playback_type_t),
        ('nr_of_angles', uint8_t),
        ('nr_of_ptts', uint16_t),
        ('parental_id', uint16_t),
        ('title_set_nr', uint8_t),
        ('vts_ttn', uint8_t),
        ('title_set_sector', uint32_t),
    ]

As a workaround for the immediate problem, on platforms that use gcc you can use c_uint8 in playback_type_t instead of c_uint. You'll want to verify other bitfields as well. I don't know about other compilers on other platforms. 

Working at the ABI level can be painful. Consider using CFFI's API level interface instead:

https://cffi.readthedocs.org
msg230469 - (view) Author: Geoff Clements (Geoff.Clements) Date: 2014-11-01 22:49
Thank you eryksun for the explanation. It's much appreciated. I'll take a
look at CFFI.
On 1 Nov 2014 22:38, "eryksun" <report@bugs.python.org> wrote:

>
> eryksun added the comment:
>
> ctypes doesn't always handle bitfields correctly. In particular it doesn't
> adapt the storage unit size when packed. A bitfield based on c_uint is
> always stored in 4 bytes, even if _pack_ is set to 1. This is MSVC rules,
> so all is well on Windows.
>
> For example, from ifo_types.h:
>
>     typedef struct {
>       unsigned int zero_1                    : 1;
>       unsigned int multi_or_random_pgc_title : 1; /* 0: one sequential pgc
> title */
>       unsigned int jlc_exists_in_cell_cmd    : 1;
>       unsigned int jlc_exists_in_prepost_cmd : 1;
>       unsigned int jlc_exists_in_button_cmd  : 1;
>       unsigned int jlc_exists_in_tt_dom      : 1;
>       unsigned int chapter_search_or_play    : 1; /* UOP 1 */
>       unsigned int title_or_time_play        : 1; /* UOP 0 */
>     } ATTRIBUTE_PACKED playback_type_t;
>
> ctypeslib translates this as follows:
>
>     class playback_type_t(Structure):
>         pass
>     playback_type_t._fields_ = [
>         ('zero_1', c_uint, 1),
>         ('multi_or_random_pgc_title', c_uint, 1),
>         ('jlc_exists_in_cell_cmd', c_uint, 1),
>         ('jlc_exists_in_prepost_cmd', c_uint, 1),
>         ('jlc_exists_in_button_cmd', c_uint, 1),
>         ('jlc_exists_in_tt_dom', c_uint, 1),
>         ('chapter_search_or_play', c_uint, 1),
>         ('title_or_time_play', c_uint, 1),
>     ]
>
> It doesn't set _pack_ = 1, but ctypes wouldn't use that to pack the c_uint
> bitfield anyway. It's 4 bytes, either way. MSVC agrees, even with #pragma
> pack(1). OTOH, with __attribute__((packed)), gcc packs the bitfield into a
> single byte.
>
> The incorrect packing (for gcc) of playback_type_t results in an incorrect
> offset for title_set_nr in title_info_t:
>
>     class title_info_t(Structure):
>         pass
>     title_info_t._pack_ = 1
>     title_info_t._fields_ = [
>         ('pb_ty', playback_type_t),
>         ('nr_of_angles', uint8_t),
>         ('nr_of_ptts', uint16_t),
>         ('parental_id', uint16_t),
>         ('title_set_nr', uint8_t),
>         ('vts_ttn', uint8_t),
>         ('title_set_sector', uint32_t),
>     ]
>
> As a workaround for the immediate problem, on platforms that use gcc you
> can use c_uint8 in playback_type_t instead of c_uint. You'll want to verify
> other bitfields as well. I don't know about other compilers on other
> platforms.
>
> Working at the ABI level can be painful. Consider using CFFI's API level
> interface instead:
>
> https://cffi.readthedocs.org
>
> ----------
> nosy: +eryksun
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue22781>
> _______________________________________
>
msg314789 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2018-04-01 22:36
Eryk Sun’s explanation makes this sound like a duplicate of Issue 15453, which shows GCC on Linux packing structures into a single byte, and ctypes using the size of the expanded integer type.
History
Date User Action Args
2022-04-11 14:58:09adminsetgithub: 66970
2021-03-12 21:11:34eryksunsetstatus: open -> closed
stage: resolved
2018-04-01 22:36:33martin.pantersetnosy: + martin.panter
messages: + msg314789
resolution: duplicate

superseder: ctype with packed bitfields does not match native compiler
2014-11-01 22:49:08Geoff.Clementssetmessages: + msg230469
2014-11-01 22:38:16eryksunsetnosy: + eryksun
messages: + msg230468
2014-11-01 21:34:08Geoff.Clementssetmessages: + msg230460
2014-11-01 19:28:29pitrousetnosy: + meador.inge, pitrou

messages: + msg230457
versions: + Python 3.5, - Python 3.4
2014-11-01 17:16:25Geoff.Clementscreate