Nested 'BigEndianStructure' fails in 2.5 and 2.6.:

TypeError: This type does not support other endian

Example and traceback in attached file.

Currently, the _fields_ of ctypes Structures with non-native byte order
can only contain simple types (like int, char, but not pointers), and
arrays of those.

Since this is all Python code (in Lib/ctypes/endian.py) it should be
possible to extend the code to handle other types as well.

If we do this, it must be decided if a Structure (call it 'part' for
this discussion) of some byte order is contained in a _field_ of a
non-native Structure type:
- Should 'part' be inserted as is, leading to a total structure of
fields with mixed byte order?

- Should a new type be created from the 'part' _fields_, with also
non-native byte-order?

Other approaches would be possible as well...

Here is a simple patch that implements the first approach; I have not
tested if it works correctly:

Index: _endian.py
===================================================================
--- _endian.py	(revision 67045)
+++ _endian.py	(working copy)
@@ -17,6 +17,8 @@
     except AttributeError:
         if type(typ) == _array_type:
             return _other_endian(typ._type_) * typ._length_
+        if issubclass(typ, Structure):
+            return typ
         raise TypeError("This type does not support other endian: %s" %
typ)
 
 class _swapped_meta(type(Structure)):

Correction:
>> Since this is all Python code (in Lib/ctypes/_endian.py) it should be

ping! This issue is still open.

theller> - Should 'part' be inserted as is, [possibly] leading 
theller>   to a total structure of fields with mixed byte order?

+1 for this option.

The test could be build from the attached example.

Added unit test to reproduce the issue (covers both big endian and little endian structures so _other_endian function is hit on any machine).

Test currently fails without fix and passes with proposed fix in place.

Also added diff for fix:

- Implemented proposed issubclass(typ, Structure) solution
- Refactored _other_endian function because we used to getattr, catch exception, then check if type is array/structure. I believe exception throwing should not be on normal code path so I replaced try-except with a check for hasattr
- Removed a unittest which becomes deprecated with this fix (unittest asserts getting _other_endian for nested struct raises exception which is no longer the case)

Is there a unit test about the actual feature: that the bytes are actually swapped in the structure?

For example, with a
class T(BigEndianStructure): _fields_ = [("a", c_int), ("b", c_int)]
cast a pointer to T into a pointer to c_int, and read the values.

New patch containing unittest that actually tests the feature.

I would appreciate it if someone can try this on a bigendian machine.

Changed c_int in tests to c_int32 just to be on the safe side.

I don't like your test because it depends on system endian:

+            if sys.byteorder == "little":
+                struct.menu.spam = 0x000000FF
+            else:
+                struct.menu.spam = 0xFF000000

I would prefer a test creating a structure from a byte string. Something like:
---------------------------
import ctypes

class Nested(ctypes.BigEndianStructure):
    _fields_ = (
        ('x', ctypes.c_uint32),
        ('y', ctypes.c_uint32),
    )

class TestStruct(ctypes.BigEndianStructure):
    _fields_ = (
        ('point', Nested),
    )

data = b'\0\0\0\1\0\0\0\2'
assert len(data) == ctypes.sizeof(TestStruct)
obj = ctypes.cast(data, ctypes.POINTER(TestStruct))[0]
assert obj.point.x == 1
assert obj.point.y == 2
---------------------------

Use b'\1\0\0\0\2\0\0\0' for little endian.

But if you set raw memory to let's say b'\0\0\0\1', when you look at the c_int value afterwards, won't it be different on little endian and big endian machines?

> But if you set raw memory to let's say b'\0\0\0\1',
> when you look at the c_int value afterwards, won't it
> be different on little endian and big endian machines?

A big endian structure is supposed to read and write memory in the... big endian order, not in the host endian.

You're right. I was busy swapping bytes in my head and missed that :)

New changeset 72e73fa03124 by Victor Stinner in branch '3.2':
Close #4376: ctypes now supports nested structures in a endian different than
http://hg.python.org/cpython/rev/72e73fa03124

New changeset 637210b9d054 by Victor Stinner in branch 'default':
(merge 3.2) Close #4376: ctypes now supports nested structures in a endian
http://hg.python.org/cpython/rev/637210b9d054

New changeset a9d0fab19d5e by Victor Stinner in branch '2.7':
Close #4376: ctypes now supports nested structures in a endian different than
http://hg.python.org/cpython/rev/a9d0fab19d5e

I commited your fix, thanks Vlad!