I am preparing a system with gcc to see if it is compiler related, i.e., goes away with gcc. On the one hand, fingers crossed - but on the other, having bitfields working regardless of the compiler should be preferred.


This issue is similar to a Solaris (C compiler) http://bugs.python.org/issue16275

During the build using clc _ get these messages - which make me hope it is an unexpected compiler issue.

"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for M is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for N is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for O is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for P is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for Q is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for R is not valid. Type unsigned assumed.
"/data/prj/aixtools/python/python-2.7.12.1/Modules/_ctypes/_ctypes_test.c", line 382.5: 1506-159 (E) Bit field type specified for S is not valid. Type unsigned assumed.

Note: I modified test_bitfields to do an additional print. Maybe this gives an idea about what the issue is (when considering the compiler messages)

root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python ./runtests.py
A 0x1001f 0
B 0x2001d 0
C 0x3001a 0
D 0x40016 0
E 0x50011 0
F 0x6000b 0
G 0x70004 0
H 0x80018 4
I 0x9000f 4
M 0x1000e 6
N 0x2000c 6
O 0x30009 6
P 0x40005 6
Q 0x50000 6
R 0x6000a 8
S 0x70003 8
.............s....................ssss............................ssssssssssssssssssssssssssss...s.......s.....................sssssssssssssssssssss......s...........s....s.....................FF...s...............................................s.............sssss.ss....s.......ss..sss...................s......s.....................s..................................................s...........ss.sssss................s.s.............s.
======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 41, in test_ints
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

- ('A', 1, -1)
?          -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

- ('M', 1, -1)
?          -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 440 tests in 0.883s (0 modules skipped)
Unavailable resources: printing, refcount

FAILED (failures=2)
root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]

FYI: similar (exact) results when 64-bit mode:
root@x064:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python ./runtests.py
A 0x1001f 0
B 0x2001d 0
C 0x3001a 0
D 0x40016 0
E 0x50011 0
F 0x6000b 0
G 0x70004 0
H 0x80018 4
I 0x9000f 4
M 0x1000e 6
N 0x2000c 6
O 0x30009 6
P 0x40005 6
Q 0x50000 6
R 0x6000a 8
S 0x70003 8
.............s....................ssss............................ssssssssssssssssssssssssssss...s.......s.....................sssssssssssssssssssss......s...........s....s.....................FF...s...............................................s.............sssss.ss....s.......ss..sss...................s......s.....................s..................................................s...........ss.sssss................s.s.............s.
======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 41, in test_ints
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

- ('A', 1, -1)
?          -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts
    self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

- ('M', 1, -1)
?          -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 440 tests in 0.870s (0 modules skipped)
Unavailable resources: printing, refcount

FAILED (failures=2)

So, it seems to be a compiler issue - when built using gcc (v4.7.4) the tests take a bit longer, but no failures.

root@x065:[/data/prj/aixtools/python/python-2.7.12.1/Lib/ctypes/test]../../../python runtests.py
A 0x1001f 0
B 0x2001d 0
C 0x3001a 0
D 0x40016 0
E 0x50011 0
F 0x6000b 0
G 0x70004 0
H 0x80018 4
I 0x9000f 4
M 0x1000e 6
N 0x2000c 6
O 0x30009 6
P 0x40005 6
Q 0x50000 6
R 0x6000a 8
S 0x70003 8
.............s....................ssss............................ssssssssssssssssssssssssssss...s.......s.....................sssssssssssssssssssss......s...........s....s..........................s...............................................s.............sssss.ss....s.......ss..sss...................s......s.....................s..................................................s...........ss.sssss................s.s.............s.
----------------------------------------------------------------------
Ran 440 tests in 1.028s (0 modules skipped)
Unavailable resources: printing, refcount

OK

The test seems to be comparing ctypes and native C bit-field structures, c_int and c_short versus native int and short. In general in C, if a bit-field has type “int” without a signed or unsigned qualifier, it is up to the implementation which mode is chosen. This is different from normal, where int always means signed int.

Since ctypes documents c_int as both meaning “int” and “signed int”, I would say that a ctypes bit-field using c_int should always behave as “signed int”. So perhaps we should just change the test structure to use “signed int” and “signed short”. That should also help avoid the warnings, and may help the Solaris situation (if that is still relevant).

I presume this also affects Python 3.

> I presume this also affects Python 3.

I presume this as well - however, harder to verify...

michael@x071:[/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test]../../../python test_bitfield.py
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Failed to import the site module
Traceback (most recent call last):
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 571, in <module>
    main()
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 557, in main
    known_paths = addusersitepackages(known_paths)
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 281, in addusersitepackages
    user_site = getusersitepackages()
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 257, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/ctypes/test/../../../Lib/site.py", line 247, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 587, in get_config_var
    return get_config_vars().get(name)
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 536, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/data/prj/aixtools/python/python-3.6.0.162/Lib/sysconfig.py", line 408, in _init_posix
    from _sysconfigdata import build_time_vars
ImportError: No module named '_sysconfigdata'

I guess I will have to install it, rather than only work from a test directory.

> The test seems to be comparing ctypes and native C bit-field structures, c_int and c_short versus native int and short. ... int always means signed int.

I believe int means signed on AIX as well - as the comment from the compiler is that type "unsigned" is assumed for a bitfield.

Note: the single line of code is:
  +382      short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;

And from your comments I am wondering if the fix is 'simple' - to:

  +382      unsigned short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;


I can at least try that.

Looking at Python-3.6.0.0a2:

the line concerned is identical - so that is the verification (for now) that it is also in Python3

FYI - after a couple of test compiles, it seems that IBM XLC does not report a message when the size specified is int, unsigned or long (i.e., does not like short).

Are you interested in the results using a size other than short - or would that break something else I cannot forsee?

Looks like your Python 3 build is messed up. Maybe it doesn’t like running from a different directory. I would try from the main build directory, and note the test_bitfields has an S:

./python -m unittest -v ctypes.test.test_bitfields

What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”. I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short.

Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.

On 30-Jul-16 02:51, Martin Panter wrote:
> Martin Panter added the comment:
>
> Looks like your Python 3 build is messed up. Maybe it doesn’t like running from a different directory. I would try from the main build directory, and note the test_bitfields has an S:
I would not know why - it is just a copy of the sources and then 
configure, make, make DESTDIR=/some/where install (rather than make 
install).
Same procedure I use for python2.7.

>
> ./python -m unittest -v ctypes.test.test_bitfields
No changes: 3.5.1

root@x064:[/data/prj/aixtools/python/python-3.5.1.2]./python -m unittest 
-v ctypes.test.test_bitfields
test_anon_bitfields (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_c_wchar (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_longlong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_1 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_2 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_3 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_4 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_multi_bitfields_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_nonint_types (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_signed (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_single_bitfield_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_big_endian (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_little_endian (ctypes.test.test_bitfields.BitFieldTest) 
... ok
test_uint64 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ulonglong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_unsigned (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ints (ctypes.test.test_bitfields.C_Test) ... FAIL
test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL

======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py", 
line 41, in test_ints
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/data/prj/aixtools/python/python-3.5.1.2/Lib/ctypes/test/test_bitfields.py", 
line 48, in test_shorts
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

----------------------------------------------------------------------
Ran 19 tests in 0.013s

FAILED (failures=2)

> What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”. I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short.
Working from Python-2.7.12 - added your changes, test still fails:

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python -m 
unittest -v ctypes.test.test_bitfields
test_anon_bitfields (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_c_wchar (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_longlong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_1 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_2 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_3 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_mixed_4 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_multi_bitfields_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_nonint_types (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_signed (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_single_bitfield_size (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_big_endian (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_uint32_swap_little_endian (ctypes.test.test_bitfields.BitFieldTest) 
... ok
test_uint64 (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ulonglong (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_unsigned (ctypes.test.test_bitfields.BitFieldTest) ... ok
test_ints (ctypes.test.test_bitfields.C_Test) ... ok
test_shorts (ctypes.test.test_bitfields.C_Test) ... FAIL

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/data/prj/aixtools/python/python-2.7.12.0/Lib/ctypes/test/test_bitfields.py", 
line 48, in test_shorts
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

- ('M', 1, -1)
?          -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 19 tests in 0.068s

FAILED (failures=1)

> Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.
Looks like this may be the approach: as signed int, signed short give 
the same message - it seems that xlc (and maybe Sun C) does not accept 
"short" as a bitfield type.

skipping test - only if compiler is not gcc, as when using gcc, test 
works fine.

> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default:

struct UNSIGNED_BITS {
    unsigned int AU: 1;
    int A: 1; /* Equivalent to unsigned int */
    signed int AS: 1;
    unsigned short MU: 1;
    short M: 1; /* Equivalent to unsigned short; triggers warning */
};

and that it cannot compile the following:

struct SIGNED_BITS {
    signed short MS: 1; /* Not supported */
};

Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler:

#ifndef /* Something to identify XLC */

Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped.

Had some flooding (leaking pipes, rather values) issues to fix. Will 
look at this asap.

On 04-Aug-16 10:58, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default:
>
> struct UNSIGNED_BITS {
>      unsigned int AU: 1;
>      int A: 1; /* Equivalent to unsigned int */
>      signed int AS: 1;
>      unsigned short MU: 1;
>      short M: 1; /* Equivalent to unsigned short; triggers warning */
> };
>
> and that it cannot compile the following:
>
> struct SIGNED_BITS {
>      signed short MS: 1; /* Not supported */
> };
>
> Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler:
>
> #ifndef /* Something to identify XLC */
>
> Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped.
>
> ----------
> keywords: +patch
> Added file: http://bugs.python.org/file44005/disable-signed-short.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

On 8/4/2016 10:58 AM, Martin Panter wrote:
> Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses), and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass, and test_shorts() should be skipped.

Was not as simple as it sounds. I could not find a specific parameter 
provided by xlc and as you wanted something that would only skip on AIX 
- testing for __GNUC__ (in the positive sense) will not work either.

What I am thinking of now is

#if defined(__GNUC__) || !defined(_AIX)

...

#endif

As this will pass on AIX when GCC is used, but only be skipped on AIX 
when GCC is not used.

+++++++++++++++++++++++++++++++++++++++++++++++++++++ More TESTS ++++

Summary - maybe the problem is with the "byref()" routine...

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

My initial test of the patch confused me, so I went back to the original 
code - and started again.

I have three partitions I am testing on - x064 (AIX 5.3 TL7, vac v.11); 
x065 (AIX 5.3 TL7, gcc 4.7.4),
and x067 ( Debian 3.16.7-ckt9-3 (2015-04-23) ppc64, gcc (Debian 
5.3.1-21) 5.3.1 20160528)

My working directory is /data/prj/aixtools/python/python-2.7.12.0 which 
is a copy of the extracted Python-2.7.12.
This directory is on NFS, so each system is using the same files as source:

The first builds - no changes - expect to test_bitfields.py:

##for n in "ABCDEFGHIMNOPQRS":
##    print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset
for n in "ABCDEFGHIMNOPQRS":
     b = BITS()
     print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset, 
func(byref(b), n)

x065:
root@x065:[/data/prj/aixtools/python/python-2.7.12.0]./python 
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0
B 0x2001d 0 0
C 0x3001a 0 0
D 0x40016 0 0
E 0x50011 0 0
F 0x6000b 0 0
G 0x70004 0 0
H 0x80018 4 0
I 0x9000f 4 0
M 0x1000e 6 0
N 0x2000c 6 0
O 0x30009 6 0
P 0x40005 6 0
Q 0x50000 6 0
R 0x6000a 8 0
S 0x70003 8 0

root@x067:/data/prj/aixtools/python/python-2.7.12.0# ./python 
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0
B 0x2001d 0 0
C 0x3001a 0 0
D 0x40016 0 0
E 0x50011 0 0
F 0x6000b 0 0
G 0x70004 0 0
H 0x80018 4 0
I 0x9000f 4 0
M 0x1000e 6 0
N 0x2000c 6 0
O 0x30009 6 0
P 0x40005 6 0
Q 0x50000 6 0
R 0x6000a 8 0
S 0x70003 8 0
...................
----------------------------------------------------------------------
Ran 19 tests in 0.087s

OK

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python 
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0
B 0x2001d 0 0
C 0x3001a 0 0
D 0x40016 0 0
E 0x50011 0 0
F 0x6000b 0 0
G 0x70004 0 0
H 0x80018 4 0
I 0x9000f 4 0
M 0x1000e 6 0
N 0x2000c 6 0
O 0x30009 6 0
P 0x40005 6 0
Q 0x50000 6 0
R 0x6000a 8 0
S 0x70003 8 0
.................FF
======================================================================
FAIL: test_ints (__main__.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 45, in test_ints
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

- ('A', 1, -1)
?          -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (__main__.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 52, in test_shorts
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

- ('M', 1, -1)
?          -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 19 tests in 0.010s

FAILED (failures=2)
root@x064:[/data/prj/aixtools/python/python-2.7.12.0]

Changed the test again to include:
##for n in "ABCDEFGHIMNOPQRS":
##    print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset
for n in "ABCDEFGHIMNOPQRS":
     b = BITS()
     print n, hex(getattr(BITS, n).size), getattr(BITS, n).offset, 
getattr(b,n), func(byref(b), n)

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python 
Lib/ctypes/test/test_bitfields.py
A 0x1001f 0 0 0
B 0x2001d 0 0 0
C 0x3001a 0 0 0
D 0x40016 0 0 0
E 0x50011 0 0 0
F 0x6000b 0 0 0
G 0x70004 0 0 0
H 0x80018 4 0 0
I 0x9000f 4 0 0
M 0x1000e 6 0 0
N 0x2000c 6 0 0
O 0x30009 6 0 0
P 0x40005 6 0 0
Q 0x50000 6 0 0
R 0x6000a 8 0 0
S 0x70003 8 0 0
.................FF
======================================================================
FAIL: test_ints (__main__.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 45, in test_ints
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('A', 1, -1) != ('A', 1, 1)

First differing element 2:
-1
1

- ('A', 1, -1)
?          -

+ ('A', 1, 1)

======================================================================
FAIL: test_shorts (__main__.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "Lib/ctypes/test/test_bitfields.py", line 52, in test_shorts
     self.assertEqual((name, i, getattr(b, name)), (name, i, 
func(byref(b), name)))
AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1)

First differing element 2:
-1
1

- ('M', 1, -1)
?          -

+ ('M', 1, 1)

----------------------------------------------------------------------
Ran 19 tests in 0.010s

FAILED (failures=2)

**********************************************************************
So, I am wondering if the problem could be better repaired in
     setattr(b, name, i)

And changed the test again:
1) comment out the initial changes
2) the following changes (note range limited to 5 for both, rather 512 
for first and 256 for second)

class C_Test(unittest.TestCase):

     def test_ints(self):
         for i in range(5):
             for name in "ABCDEFGHI":
                 b = BITS()
                 setattr(b, name, i)
                 print "name:", name, "range:", i, "getattr():", 
getattr(b,name), "byref():", func(byref(b), name)
                 # self.assertEqual((name, i, getattr(b, name)), (name, 
i, func(byref(b), name)))

     def test_shorts(self):
         for i in range(5):
             for name in "MNOPQRS":
                 b = BITS()
                 setattr(b, name, i)
                 print "name:", name, "range:", i, "getattr():", 
getattr(b,name), "byref():", func(byref(b), name)
                 # self.assertEqual((name, i, getattr(b, name)), (name, 
i, func(byref(b), name)))

***************************

And for most tests (remember the range is 5 now) - all is fine

root@x064:[/data/prj/aixtools/python/python-2.7.12.0]./python 
Lib/ctypes/test/test_bitfields.py
.................name: A range: 0 getattr(): 0 byref(): 0
name: B range: 0 getattr(): 0 byref(): 0
name: C range: 0 getattr(): 0 byref(): 0
name: D range: 0 getattr(): 0 byref(): 0
name: E range: 0 getattr(): 0 byref(): 0
name: F range: 0 getattr(): 0 byref(): 0
name: G range: 0 getattr(): 0 byref(): 0
name: H range: 0 getattr(): 0 byref(): 0
name: I range: 0 getattr(): 0 byref(): 0
name: A range: 1 getattr(): -1 byref(): 1
name: B range: 1 getattr(): 1 byref(): 1
name: C range: 1 getattr(): 1 byref(): 1
name: D range: 1 getattr(): 1 byref(): 1
name: E range: 1 getattr(): 1 byref(): 1
name: F range: 1 getattr(): 1 byref(): 1
name: G range: 1 getattr(): 1 byref(): 1
name: H range: 1 getattr(): 1 byref(): 1
name: I range: 1 getattr(): 1 byref(): 1
name: A range: 2 getattr(): 0 byref(): 0
name: B range: 2 getattr(): -2 byref(): 2
name: C range: 2 getattr(): 2 byref(): 2
name: D range: 2 getattr(): 2 byref(): 2
name: E range: 2 getattr(): 2 byref(): 2
name: F range: 2 getattr(): 2 byref(): 2
name: G range: 2 getattr(): 2 byref(): 2
name: H range: 2 getattr(): 2 byref(): 2
name: I range: 2 getattr(): 2 byref(): 2
name: A range: 3 getattr(): -1 byref(): 1
name: B range: 3 getattr(): -1 byref(): 3
name: C range: 3 getattr(): 3 byref(): 3
name: D range: 3 getattr(): 3 byref(): 3
name: E range: 3 getattr(): 3 byref(): 3
name: F range: 3 getattr(): 3 byref(): 3
name: G range: 3 getattr(): 3 byref(): 3
name: H range: 3 getattr(): 3 byref(): 3
name: I range: 3 getattr(): 3 byref(): 3
name: A range: 4 getattr(): 0 byref(): 0
name: B range: 4 getattr(): 0 byref(): 0
name: C range: 4 getattr(): -4 byref(): 4
name: D range: 4 getattr(): 4 byref(): 4
name: E range: 4 getattr(): 4 byref(): 4
name: F range: 4 getattr(): 4 byref(): 4
name: G range: 4 getattr(): 4 byref(): 4

Back to x065...

root@x065:[/data/prj/aixtools/python/python-2.7.12.0]./python Lib
.................name: A range: 0 getattr(): 0 byref(): 0
name: B range: 0 getattr(): 0 byref(): 0
name: C range: 0 getattr(): 0 byref(): 0
name: D range: 0 getattr(): 0 byref(): 0
name: E range: 0 getattr(): 0 byref(): 0
name: F range: 0 getattr(): 0 byref(): 0
name: G range: 0 getattr(): 0 byref(): 0
name: H range: 0 getattr(): 0 byref(): 0
name: I range: 0 getattr(): 0 byref(): 0
name: A range: 1 getattr(): -1 byref(): -1
name: B range: 1 getattr(): 1 byref(): 1
name: C range: 1 getattr(): 1 byref(): 1
name: D range: 1 getattr(): 1 byref(): 1
name: E range: 1 getattr(): 1 byref(): 1
name: F range: 1 getattr(): 1 byref(): 1
name: G range: 1 getattr(): 1 byref(): 1
name: H range: 1 getattr(): 1 byref(): 1
name: I range: 1 getattr(): 1 byref(): 1
name: A range: 2 getattr(): 0 byref(): 0
name: B range: 2 getattr(): -2 byref(): -2
name: C range: 2 getattr(): 2 byref(): 2
name: D range: 2 getattr(): 2 byref(): 2
name: E range: 2 getattr(): 2 byref(): 2
name: F range: 2 getattr(): 2 byref(): 2
name: G range: 2 getattr(): 2 byref(): 2
name: H range: 2 getattr(): 2 byref(): 2
name: I range: 2 getattr(): 2 byref(): 2
name: A range: 3 getattr(): -1 byref(): -1
name: B range: 3 getattr(): -1 byref(): -1
name: C range: 3 getattr(): 3 byref(): 3
name: D range: 3 getattr(): 3 byref(): 3
name: E range: 3 getattr(): 3 byref(): 3
name: F range: 3 getattr(): 3 byref(): 3
name: G range: 3 getattr(): 3 byref(): 3
name: H range: 3 getattr(): 3 byref(): 3
name: I range: 3 getattr(): 3 byref(): 3
name: A range: 4 getattr(): 0 byref(): 0
name: B range: 4 getattr(): 0 byref(): 0
name: C range: 4 getattr(): -4 byref(): -4
name: D range: 4 getattr(): 4 byref(): 4
name: E range: 4 getattr(): 4 byref(): 4
name: F range: 4 getattr(): 4 byref(): 4
name: G range: 4 getattr(): 4 byref(): 4
name: H range: 4 getattr(): 4 byref(): 4
name: I range: 4 getattr(): 4 byref(): 4
.name: M range: 0 getattr(): 0 byref(): 0
name: N range: 0 getattr(): 0 byref(): 0
name: O range: 0 getattr(): 0 byref(): 0
name: P range: 0 getattr(): 0 byref(): 0
name: Q range: 0 getattr(): 0 byref(): 0
name: R range: 0 getattr(): 0 byref(): 0
name: S range: 0 getattr(): 0 byref(): 0
name: M range: 1 getattr(): -1 byref(): -1
name: N range: 1 getattr(): 1 byref(): 1
name: O range: 1 getattr(): 1 byref(): 1
name: P range: 1 getattr(): 1 byref(): 1
name: Q range: 1 getattr(): 1 byref(): 1
name: R range: 1 getattr(): 1 byref(): 1
name: S range: 1 getattr(): 1 byref(): 1
name: M range: 2 getattr(): 0 byref(): 0
name: N range: 2 getattr(): -2 byref(): -2
name: O range: 2 getattr(): 2 byref(): 2
name: P range: 2 getattr(): 2 byref(): 2
name: Q range: 2 getattr(): 2 byref(): 2
name: R range: 2 getattr(): 2 byref(): 2
name: S range: 2 getattr(): 2 byref(): 2
name: M range: 3 getattr(): -1 byref(): -1
name: N range: 3 getattr(): -1 byref(): -1
name: O range: 3 getattr(): 3 byref(): 3
name: P range: 3 getattr(): 3 byref(): 3
name: Q range: 3 getattr(): 3 byref(): 3
name: R range: 3 getattr(): 3 byref(): 3
name: S range: 3 getattr(): 3 byref(): 3
name: M range: 4 getattr(): 0 byref(): 0
name: N range: 4 getattr(): 0 byref(): 0
name: O range: 4 getattr(): -4 byref(): -4
name: P range: 4 getattr(): 4 byref(): 4
name: Q range: 4 getattr(): 4 byref(): 4
name: R range: 4 getattr(): 4 byref(): 4
name: S range: 4 getattr(): 4 byref(): 4
.
-----------------------------------------------------------------
Ran 19 tests in 0.056s

OK

Now looking at the results - it seems the byref() function is where the
difference lies - as getattr() returns the same value for both x064/xlc
and x065/gcc

Hope this helps!

Michael, byref() is just a helper for passing an object’s address to a C function. Calling func(byref(b), ...) in Python is equivalent to the C code

unpack_bitfields(&b, ...)

I still think the root problem is in unpack_bitfields(). When compiled with XLC, your experiments confirm that it always returns unsigned values, and with GCC, it returns signed values.

It looks like you can detect XLC with the __IBMC__ and __xlC__ macros (not sure if there is a practical difference). So perhaps we can fix this with

#ifndef __xlC__

I'll get to this asap. Have to install a new "clean" environment aka a 
new virtual machine.

On 18-Aug-16 02:43, Martin Panter wrote:
> Martin Panter added the comment:
>
> Michael, byref() is just a helper for passing an object’s address to a C function. Calling func(byref(b), ...) in Python is equivalent to the C code
>
> unpack_bitfields(&b, ...)
>
> I still think the root problem is in unpack_bitfields(). When compiled with XLC, your experiments confirm that it always returns unsigned values, and with GCC, it returns signed values.
>
> It looks like you can detect XLC with the __IBMC__ and __xlC__ macros (not sure if there is a practical difference). So perhaps we can fix this with
>
> #ifndef __xlC__
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

I came across this issue while researching where to post my patch (having come across this while building Python 2.7 & 3.x on AIX via xlc).

> In general in C, if a bit-field has type “int” without a signed or unsigned qualifier, it is up to the implementation which mode is chosen.

Unfortunately, the ISO C standard leaves it up to the compiler to decide whether to default to 'signed' or 'unsigned' for non-qualified bit-field declarations.  gcc defaults to signed (https://gcc.gnu.org/onlinedocs/gcc-3.3.6/gcc/Non_002dbugs.html); however, xlc defaults to unsigned (https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.3/com.ibm.xlc1313.aix.doc/compiler_ref/opt_bitfields.html).

> [...] having bitfields working regardless of the compiler should be preferred.

However, ctypes_test assumes an unqualified bit-field will be signed.  To achieve bit-fields working regardless of the compiler, declarations must explicitly qualifying the sign for bit-fields.  This makes the intent of what is expected explicit and avoids implementation-defined behavior that will differ from one compiler to the next.

With patch ctypes_test_sign_bitfields.diff provided, I have verified ctypes_test passes on Python 2.7, Python 3.4, and Python 3.5 (on AIX & Solaris).  If need be, I'm happy to provide before-&-after output of ctypes_test with & without the patch applied (or verification from others would be greatly appreciated).

On 22-Aug-16 19:43, Eric N. Vander Weele wrote:
> Eric N. Vander Weele added the comment:
>
> I came across this issue while researching where to post my patch (having come across this while building Python 2.7 & 3.x on AIX via xlc).
>
>> In general in C, if a bit-field has type “int” without a signed or unsigned qualifier, it is up to the implementation which mode is chosen.
> Unfortunately, the ISO C standard leaves it up to the compiler to decide whether to default to 'signed' or 'unsigned' for non-qualified bit-field declarations.  gcc defaults to signed (https://gcc.gnu.org/onlinedocs/gcc-3.3.6/gcc/Non_002dbugs.html); however, xlc defaults to unsigned (https://www.ibm.com/support/knowledgecenter/SSGH2K_13.1.3/com.ibm.xlc1313.aix.doc/compiler_ref/opt_bitfields.html).
>
>> [...] having bitfields working regardless of the compiler should be preferred.
> However, ctypes_test assumes an unqualified bit-field will be signed.  To achieve bit-fields working regardless of the compiler, declarations must explicitly qualifying the sign for bit-fields.  This makes the intent of what is expected explicit and avoids implementation-defined behavior that will differ from one compiler to the next.
>
> With patch ctypes_test_sign_bitfields.diff provided, I have verified ctypes_test passes on Python 2.7, Python 3.4, and Python 3.5 (on AIX & Solaris).  If need be, I'm happy to provide before-&-after output of ctypes_test with & without the patch applied (or verification from others would be greatly appreciated).
>
> ----------
> Added file: http://bugs.python.org/file44189/ctypes_test_sign_bitfields.diff
I believe that there is a specific reason that M, N, O, P, Q, R and S 
are "short". For AIX/xlc at least "signed" is equivalent to "signed int" 
and not the same size as "unsigned short".

> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

> I believe that there is a specific reason that M, N, O, P, Q, R and S 
are "short". [...]

Oops - this was an oversight when I created the patch.  I just uploaded ctypes_test_sign_bitfields_2.diff, which is what I originally intended.

Now I am confused. In <https://bugs.python.org/issue27643#msg271773> we have

[Me] If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.

[Michael] Looks like this may be the approach: as signed int, signed short give the same message - it seems that xlc (and maybe Sun C) does not accept "short" as a bitfield type.

Apparently XLC doesn’t accept signed short bitfields for Michael, but does for Eric. What’s going on? Maybe different versions?

Using:

root@x064:[/data/prj/aixtools/python/tests]cat -n bits.c
      1   struct BITS {
      2      signed int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9;
      3      signed short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;
      4      signed char T: 1, U: 2, V: 3;
      5      unsigned char W: 1, X: 2, Y: 3;
      6      signed long J: 1, K: 2, L: 3;
      7   };

Using:

root@x064:[/data/prj/aixtools/python/tests]xlc -qversion
IBM XL C/C++ for AIX, V11.1 (5724-X13)
Version: 11.01.0000.0020

I get:

root@x064:[/data/prj/aixtools/python/tests]xlc -c bits.c
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for M is not 
valid. Type unsigned assumed.
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for N is not 
valid. Type unsigned assumed.
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for O is not 
valid. Type unsigned assumed.
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for P is not 
valid. Type unsigned assumed.
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for Q is not 
valid. Type unsigned assumed.
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for R is not 
valid. Type unsigned assumed.
"bits.c", line 3.5: 1506-159 (E) Bit field type specified for S is not 
valid. Type unsigned assumed.
"bits.c", line 4.5: 1506-159 (E) Bit field type specified for T is not 
valid. Type unsigned assumed.
"bits.c", line 4.5: 1506-159 (E) Bit field type specified for U is not 
valid. Type unsigned assumed.
"bits.c", line 4.5: 1506-159 (E) Bit field type specified for V is not 
valid. Type unsigned assumed.
"bits.c", line 5.5: 1506-159 (E) Bit field type specified for W is not 
valid. Type unsigned assumed.
"bits.c", line 5.5: 1506-159 (E) Bit field type specified for X is not 
valid. Type unsigned assumed.
"bits.c", line 5.5: 1506-159 (E) Bit field type specified for Y is not 
valid. Type unsigned assumed.

On 23-Aug-16 01:06, Martin Panter wrote:
> Martin Panter added the comment:
>
> Now I am confused. In <https://bugs.python.org/issue27643#msg271773> we have
>
> [Me] If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.
As the test also fails for A..I when not specified as signed int, if you 
have not already "changed" that (I forget in the time passed), then that 
should, at least be specified for the A..I BITS definition.

I do not understand what you mean by "supports" - yes, it can be 
assigned, but the result you get when "querying" the bit depends on how 
you query it. Is that "support" when conflicting values are returned? 
Don't mind my not understanding - If skipping "solves" it, then yes, 
skip the test.

>
> [Michael] Looks like this may be the approach: as signed int, signed short give the same message - it seems that xlc (and maybe Sun C) does not accept "short" as a bitfield type.
>
> Apparently XLC doesn’t accept signed short bitfields for Michael, but does for Eric. What’s going on? Maybe different versions?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

On 23-Aug-16 21:23, Michael Felt wrote:
> Michael Felt added the comment:
>
> Using:
>
> root@x064:[/data/prj/aixtools/python/tests]cat -n bits.c
>        1   struct BITS {
>        2      signed int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9;
>        3      signed short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;
>        4      signed char T: 1, U: 2, V: 3;
>        5      unsigned char W: 1, X: 2, Y: 3;
>        6      signed long J: 1, K: 2, L: 3;
>        7   };
>
> Using:
>
> root@x064:[/data/prj/aixtools/python/tests]xlc -qversion
> IBM XL C/C++ for AIX, V11.1 (5724-X13)
> Version: 11.01.0000.0020
>
> I get:
>
> root@x064:[/data/prj/aixtools/python/tests]xlc -c bits.c
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for M is not
> valid. Type unsigned assumed.
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for N is not
> valid. Type unsigned assumed.
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for O is not
> valid. Type unsigned assumed.
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for P is not
> valid. Type unsigned assumed.
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for Q is not
> valid. Type unsigned assumed.
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for R is not
> valid. Type unsigned assumed.
> "bits.c", line 3.5: 1506-159 (E) Bit field type specified for S is not
> valid. Type unsigned assumed.
> "bits.c", line 4.5: 1506-159 (E) Bit field type specified for T is not
> valid. Type unsigned assumed.
> "bits.c", line 4.5: 1506-159 (E) Bit field type specified for U is not
> valid. Type unsigned assumed.
> "bits.c", line 4.5: 1506-159 (E) Bit field type specified for V is not
> valid. Type unsigned assumed.
p.s. I guess we can assume, better infer, this message does not mean 
"unsigned whatever" but unsigned (unspoken int)
> "bits.c", line 5.5: 1506-159 (E) Bit field type specified for W is not
> valid. Type unsigned assumed.
> "bits.c", line 5.5: 1506-159 (E) Bit field type specified for X is not
> valid. Type unsigned assumed.
> "bits.c", line 5.5: 1506-159 (E) Bit field type specified for Y is not
> valid. Type unsigned assumed.
>
> On 23-Aug-16 01:06, Martin Panter wrote:
>> Martin Panter added the comment:
>>
>> Now I am confused. In <https://bugs.python.org/issue27643#msg271773> we have
>>
>> [Me] If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.
> As the test also fails for A..I when not specified as signed int, if you
> have not already "changed" that (I forget in the time passed), then that
> should, at least be specified for the A..I BITS definition.
>
> I do not understand what you mean by "supports" - yes, it can be
> assigned, but the result you get when "querying" the bit depends on how
> you query it. Is that "support" when conflicting values are returned?
> Don't mind my not understanding - If skipping "solves" it, then yes,
> skip the test.
>
>> [Michael] Looks like this may be the approach: as signed int, signed short give the same message - it seems that xlc (and maybe Sun C) does not accept "short" as a bitfield type.
>>
>> Apparently XLC doesn’t accept signed short bitfields for Michael, but does for Eric. What’s going on? Maybe different versions?
>>
>> ----------
>>
>> _______________________________________
>> Python tracker <report@bugs.python.org>
>> <http://bugs.python.org/issue27643>
>> _______________________________________
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

I am able to replicate what Michael has provided (i.e., xlc does not support signed short).  Sorry for the confusion: I overlooked that the compiler is emitting the error with may patch and assuming 'unsigned'.

So it seems like there are two things to address: (1) The bit fields should be explicitly marked as 'signed', since that appears to be the desired intent.  (2) What to do about the test case.

It seems like we all agree on (1).  For (2), is this something that should be stubbed out on AIX/xlc, resolved by xlc for supporting implementation-defined (short) bit-fields, or remove the short members in the struct since C99 (6.7.2.1) allows "a qualified or unqualified version of _Bool, signed int, unsigned int, or some other implementation-defined type"; thus removing the ambiguity for implementation-defined behavior?

Michael: When posting to the bug tracker, please trim irrelevant parts of old messages. It makes it hard to see if you actually added anything new.

> [Me] If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.

Proper support for “signed short” according to standard C I guess would mean if you define

struct BITS {
    signed short M: 1;
} b;
b.M = -1;

then reading back b.M gives -1. However I realized test_bitfields tests overflowing values rather than negative values. In any case, I think we have established that neither of these cases work with XLC.

I haven’t changed the signed int A–I fields yet. That was part of my patch. I was waiting for confirmation about the __xlC__ check, before committing the whole thing.

Eric: I proposed to conditionally skip the test; see disable-signed-short.patch. Since many other compilers apparently pass the test and support signed short, we should probably keep the test.

As I was not responding properly (too verbose) - I'll reread the thread for the initial patch suggestion - and hope it still fits the current 'master'.

My apologies for the long silence on this.

On 30/07/2016 02:51, Martin Panter wrote:
> Martin Panter added the comment:
>
>
> ./python -m unittest -v ctypes.test.test_bitfields
>
> What I am suggesting as a fix is to change line 381 from plain “int” to “signed int”, and 382 to “signed short”.

As some time has passed, I assume you mean look at this change:

struct BITS {
     signed int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9;
     signed short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7;
};

> I can make a patch later if that will help. Hopefully with these changes the C compiler will use signed integer logic, matching what I believe the ctypes library uses for c_int and c_short.

Not helping the compilation in any case - same messages as before.

* Default make:

michael@x071:[/data/prj/python/git/xlc-python3-3.7]xlc_r -DNDEBUG -O 
-I/opt/include -O2 -qmaxmem=-1 -qarch=pwr5 -I/opt/>
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for M is not valid. 
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for N is not valid. 
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for O is not valid. 
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for P is not valid. 
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for Q is not valid. 
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for R is not valid. 
Type unsigned assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for S is not valid. 
Type unsigned assumed.

* Adding (manually) -qbitfields=signed - almost the same.

michael@x071:[/data/prj/python/git/xlc-python3-3.7]xlc_r 
-qbitfields=signed -DNDEBUG -O -I/opt/include -O2 -qmaxmem=-1 >
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for M is not valid. 
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for N is not valid. 
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for O is not valid. 
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for P is not valid. 
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for Q is not valid. 
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for R is not valid. 
Type signed assumed.
"/data/prj/python/git/xlc-python3-3.7/Modules/_ctypes/_ctypes_test.c", 
line 418.5: 1506-159 (E) Bit field type specified for S is not valid. 
Type signed assumed.

> Using a type other than short is not right, because the Python test is explicitly trying to test c_short behaviour. If your compiler does not support “signed short” bitfields, maybe we just have to accept that ctypes supports it even though the compiler doesn’t, and skip the test.
Shall reply, asap, to your suggestion for a test that knows the 
difference in compilers.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

On 04/08/2016 10:58, Martin Panter wrote:
> Martin Panter added the comment:
>
> Okay, so to be clear, I am assuming XLC supports all of the following fields, and uses unsigned bit fields by default:
>
> struct UNSIGNED_BITS {
>      unsigned int AU: 1;
>      int A: 1; /* Equivalent to unsigned int */
>      signed int AS: 1;
>      unsigned short MU: 1;
>      short M: 1; /* Equivalent to unsigned short; triggers warning */
> };
>
> and that it cannot compile the following:
>
> struct SIGNED_BITS {
>      signed short MS: 1; /* Not supported */
> };
>
> Attached is what I think a patch to resolve this would look like. However it needs a line in Modules/_ctypes/_ctypes_test.c completed to detect the compiler:
>
> #ifndef /* Something to identify XLC */
>
> Can you figure out a way to test for XLC (but not GCC, which the AIX buildbot uses),

I'll figure this out. HOWEVER, Just want to mention that both parts of 
C_Test are failing when using xlc (i.e., on "A" and "M"). Note line 40 
and 51.

(Have not done the compiler detect part yet, so it still returns 
something != 999)

    +33  class C_Test(unittest.TestCase):
    +34
    +35      def test_ints(self):
    +36          for i in range(512):
    +37              for name in "ABCDEFGHI":
    +38                  b = BITS()
    +39                  setattr(b, name, i)
    +40                  self.assertEqual(getattr(b, name), 
func(byref(b), name.encode('ascii')))
    +41
    +42      def test_shorts(self):
    +43          b = BITS()
    +44          name = "M"
    +45          if func(byref(b), name.encode('ascii')) == 999:
    +46              self.skipTest("Compiler does not support signed 
short bitfields")
    +47          for i in range(256):
    +48              for name in "MNOPQRS":
    +49                  b = BITS()
    +50                  setattr(b, name, i)
    +51                  self.assertEqual(getattr(b, name), 
func(byref(b), name.encode('ascii')))
    +52

======================================================================
FAIL: test_ints (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/data/prj/python/git/xlc-python3-3.7/Lib/ctypes/test/test_bitfields.py", 
line 40, in test_ints
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

======================================================================
FAIL: test_shorts (ctypes.test.test_bitfields.C_Test)
----------------------------------------------------------------------
Traceback (most recent call last):
   File 
"/data/prj/python/git/xlc-python3-3.7/Lib/ctypes/test/test_bitfields.py", 
line 51, in test_shorts
     self.assertEqual(getattr(b, name), func(byref(b), 
name.encode('ascii')))
AssertionError: -1 != 1

----------------------------------------------------------------------

> and then try my patch out? Hopefully you see no more compiler warnings, test_ints() should now pass,
again - not done the changes to _ctypes_test.c yet - so maybe the 
test_ints() will pass after that.
>   and test_shorts() should be skipped.
>
> ----------
> keywords: +patch
> Added file: http://bugs.python.org/file44005/disable-signed-short.patch
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue27643>
> _______________________________________

@panter - your patch seems to be working well. Thanks.

PR 5164 submitted for review

I know it is not earth shattering - but it will permit one more test to pass. Please review the PR. Thx.

Again, the PR worked months ago. I expect it to still work.

So, I guess the question is: is there any interest. After several weeks of waiting after the last ttt - still waiting :)

well, update: the issue34603 merged 16 days ago has broken this PR - that has been waiting for nearly 10 months.

Unhappy camper.

And, just as a short reminder - there were earlier ¨patches (that I just copied) going back more than 2 years.

Please, some attention.

New changeset 22462da70c1ae015a60a7b821547bc6d2b5bc265 by Nick Coghlan (Michael Felt) in branch 'master':
bpo-27643 - skip test_ctypes test case with XLC compiler. (GH-5164)
https://github.com/python/cpython/commit/22462da70c1ae015a60a7b821547bc6d2b5bc265