classification
Title: Can not set value for structure members larger than 32 bits
Type: behavior Stage: resolved
Components: ctypes, Library (Lib), Windows Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: meador.inge Nosy List: hct, higstar, meador.inge, ned.deily, ocean-city, python-dev, r.david.murray, stutzbach, theller, vladris
Priority: normal Keywords: patch

Created on 2009-07-16 01:11 by higstar, last changed 2012-07-19 05:16 by meador.inge. This issue is now closed.

Files
File name Uploaded Description Edit
py3k_fix_ctypes_cfields.patch ocean-city, 2010-10-03 16:34 review
py3k_fix_ctypes_cfields_v2.patch ocean-city, 2010-10-03 18:51 with test review
py3k_fix_ctypes_cfields_v3.patch ocean-city, 2010-10-03 19:14 without MAX_SIZE_INT review
Messages (14)
msg90549 - (view) Author: higstar (higstar) Date: 2009-07-16 01:11
Added a test to test_bitfields.py:
    def test_uint32(self):
        class X(Structure):
            _fields_ = [("a", c_uint32, 32)]

        x = X()
        x.a = 10
        self.failUnlessEqual(x.a, 10)

Run in Python 2.5.2 and 2.6.2:
======================================================================
FAIL: test_uint32 (__main__.BitFieldTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_bitfields.py", line 73, in test_uint32
    self.failUnlessEqual(x.a, 10)
AssertionError: 0L != 10
msg90550 - (view) Author: higstar (higstar) Date: 2009-07-16 01:34
Also run in 3.0
msg90570 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-07-16 14:14
What's your platform? I could reproduce this on windows. And I found 
attached patch can workaround this.
msg90755 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-07-21 12:17
One of the new tests in r74134 is failing on my box (Gentoo x86), and on
a number of the buildbots:

test test_ctypes failed -- Traceback (most recent call last):
  File "/home/rdmurray/python/trunk/Lib/ctypes/test/test_bitfields.py",
line 255, in test_uint64
    self.failUnlessEqual(x.a, 10)
AssertionError: 0L != 10
msg90756 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-07-21 12:21
> One of the new tests in r74134 is failing on my box (Gentoo x86), and on
> a number of the buildbots:

I've seen that, but do not yet have a patch.
msg90765 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-07-21 17:26
ctypes_workaround_2.patch from ocean-city seems to fix it so I'll apply
that.
msg117915 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-10-03 15:44
I glanced at my patch again, and I noticed it has a problem.
SET() cannot handle type larger than unsigned int on windows.
I'll recreate the patch...
msg117918 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-10-03 16:34
Here is at least better, simple patch. But ideally,
mask should be created from variable type. "short" mask
should be created for "short" variable, "long long" mask
for "long long" variable, vise verse.

I'll create such patch next. I hope it's final solution.

////////////////////////////////////////
// This works on attached patch.

from __future__ import print_function
import ctypes
class Blah(ctypes.Structure):
    _fields_ = [("a", ctypes.c_uint64, 64),
                ("b", ctypes.c_uint16, 16),
                ("c", ctypes.c_uint8, 8),
                ("d", ctypes.c_uint8, 8)]

x = Blah(0xFEDCBA9876543210,0xBEEF,0x44,0x12)
print(Blah.a, hex(x.a))
print(Blah.b, hex(x.b))
print(Blah.c, hex(x.c))
print(Blah.d, hex(x.d))
msg117920 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-10-03 18:02
Sorry´╝îmy patch didn't work again... Because of this
compiler behavior. Is this ANSI standard?

#include <stdio.h>

typedef unsigned __int32 uint32;

static void
print_bits(uint32 n)
{
    int i;

    for (i = 31; i >= 0; --i)
    {
        printf("%c", (n & (1 << i)) ? '1' : '0');
    }
    printf(" : %X\n", n);
}

int main()
{
    uint32 n;

    n = 1;
    print_bits(n << 30);
    print_bits(n << 31);
    print_bits(n << 32);
    print_bits((n << 31) << 1);
}

R:\test\bitshift>a
01000000000000000000000000000000 : 40000000
10000000000000000000000000000000 : 80000000
00000000000000000000000000000001 : 1
00000000000000000000000000000000 : 0

I thought n << 32 should be 0. I hope new patch
is somehow better.
msg117923 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2010-10-03 19:14
This patch removes MAX_SIZE_INT limitation. I hope this
is final patch. I confirmed all ctypes test passed on
my environment. (Windows, VS8.0)
msg140096 - (view) Author: Vlad Riscutia (vladris) Date: 2011-07-11 01:15
I have a similar patch for issue 6068. I wasn't aware of this issue when I looked into it.

I believe both patches fix the same thing (please take a look and correct me if I'm wrong). My fix: we don't need to treat Windows differently, just remove #ifdef and 

#define BIT_MASK(size) ((1LL << NUM_BITS(size))-1)

regardless of platform. Unittests for this patch pass for my patch too. I believe this is some old #ifdef that was put in place due to a compiler bug which got fixed since then.
msg165659 - (view) Author: HCT (hct) Date: 2012-07-16 22:12
Hirokazu's v3 patch is a clean solution for the issue and works on 3.2 

any update on when it will go into 3.2/3.3? I can help if needed
msg165824 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2012-07-19 04:36
The problem is with the 'BIT_MASK' macro, which is currently defined as the following for Windows:

   #define BIT_MASK(size) ((1 << NUM_BITS(size))-1)

The C standard says that any attempt to shift left by exactly the bit width is undefined behavior.  For the given test case 'NUM_BITS(size) == 32' and the '1' constant is of type 'int'.  Thus the behavior is undefined.

For x86 Windows with the MSVC++ compiler this ends up as 'SAL edx, CL' and the Intel manuals say that the shift count is masked by 0x1F.  Thus the bit mask ends up as '(1 << 0) - 1 == 0'.  Plug the zero mask into the 'SET' macro and you will see why the reported behavior is being observed.

I tested Hirokazu's last patch on Windows 7 64-bit and OS X 10.7;  I saw no regression.  I will apply the patch shortly.
msg165826 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-19 05:15
New changeset 77401bd4f567 by Meador Inge in branch '2.7':
Issue #6493: Fix handling of c_uint32 bitfields with width of 32 on Windows.
http://hg.python.org/cpython/rev/77401bd4f567

New changeset 153ae76b963e by Meador Inge in branch '3.2':
Issue #6493: Fix handling of c_uint32 bitfields with width of 32 on Windows.
http://hg.python.org/cpython/rev/153ae76b963e

New changeset 3fbfa61634de by Meador Inge in branch 'default':
Issue #6493: Fix handling of c_uint32 bitfields with width of 32 on Windows.
http://hg.python.org/cpython/rev/3fbfa61634de
History
Date User Action Args
2012-07-19 05:16:21meador.ingesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2012-07-19 05:15:43python-devsetnosy: + python-dev
messages: + msg165826
2012-07-19 04:36:35meador.ingesetassignee: meador.inge
messages: + msg165824
2012-07-19 03:44:05meador.ingelinkissue6068 superseder
2012-07-16 22:15:12pitrousetversions: + Python 3.3, - Python 3.1
nosy: + meador.inge

assignee: theller -> (no value)
components: + Library (Lib), Windows
stage: commit review -> patch review
2012-07-16 22:12:21hctsetnosy: + hct
messages: + msg165659
2011-07-11 01:15:26vladrissetnosy: + vladris
messages: + msg140096
2010-10-03 19:14:47ocean-citysetfiles: + py3k_fix_ctypes_cfields_v3.patch

messages: + msg117923
2010-10-03 18:52:07ocean-citysetfiles: - py3k_fix_ctypes_cfields_v2.patch
2010-10-03 18:51:48ocean-citysetfiles: + py3k_fix_ctypes_cfields_v2.patch
2010-10-03 18:02:59ocean-citysetfiles: + py3k_fix_ctypes_cfields_v2.patch

messages: + msg117920
2010-10-03 16:36:13ocean-citysetfiles: - ctypes_workaround_2.patch
2010-10-03 16:36:00ocean-citysetfiles: - ctypes_workaround.patch
2010-10-03 16:34:29ocean-citysetfiles: + py3k_fix_ctypes_cfields.patch

messages: + msg117918
versions: - Python 2.6
2010-10-03 15:44:40ocean-citysetnosy: + ned.deily, stutzbach
messages: + msg117915
2010-10-03 15:31:37ocean-citylinkissue9989 superseder
2009-07-21 17:26:59thellersetmessages: + msg90765
title: Can not set value for structure members larger than 32 bits -> Can not set value for structure members larger than 32 bits
2009-07-21 12:21:39thellersetmessages: + msg90756
title: Can not set value for structure members larger than 32 bits -> Can not set value for structure members larger than 32 bits
2009-07-21 12:18:54r.david.murraysetpriority: normal
stage: commit review
type: behavior
versions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.5, Python 3.0
2009-07-21 12:17:56r.david.murraysetnosy: + r.david.murray
messages: + msg90755
2009-07-16 14:28:50ocean-citysetfiles: + ctypes_workaround_2.patch
2009-07-16 14:14:04ocean-citysetfiles: + ctypes_workaround.patch

nosy: + ocean-city
messages: + msg90570

keywords: + patch
2009-07-16 01:34:31higstarsetmessages: + msg90550
versions: + Python 3.0
2009-07-16 01:11:57higstarcreate