I have had an issue arise with ctypes callbacks with 64bit Python on Windows.
Note: everything works fine with 32bit Python on Windows and on 32bit and 64bit Linux.

I have created a simple example to illustrate the issue I have (see attachment), but the real-life issue occurs with using Python to interact with the EPICS control software (http://www.aps.anl.gov/epics/) used at many major scientific institutes.

Basically, if I have a C callback that takes a struct (by value) greater than 8 bytes then the callback returns nonsense. 8 bytes or less works fine.

Stepping through with the Windows debugger, if appears that something goes amiss between the callback being called in C and the closure_fcn(ffi_cif *cif, void *resp, void **args, void *userdata) function in ctypes's callback.c file. Unfortunately, the debugger won't let me step in between those two points.

Looking at the memory I can see the original data in memory at some memory address, X, and a copy of the data at X+40 bytes, but the args in the closure_fcn points at X-40 bytes (which is junk).

Using 32bit Python the data copy is at X-40 bytes and the args pointer in the closure_fcn also points at this.

EPICS has some 64bit C/C++ clients that work fine using callbacks on Windows. Likewise, doing the same sort of thing as ctypes does with EPICS from C# using PInvoke works fine.
 
Any help would be much appreciated.

Is "_pack_ = 1" correct? Did you compile your C library with /Zp1 or similar?
Also check that ctypes.sizeof(callback_t) matches the one given by the C compiler.

Hi Amaury.

I have tried removing pack and using /Zp1, but it makes no difference.
The size of callback_t and the one in C are 8 bytes.

Thanks,

Matt

On 13 March 2013 13:19, Amaury Forgeot d'Arc <report@bugs.python.org> wrote:

>
> Amaury Forgeot d'Arc added the comment:
>
> Is "_pack_ = 1" correct? Did you compile your C library with /Zp1 or
> similar?
> Also check that ctypes.sizeof(callback_t) matches the one given by the C
> compiler.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17310>
> _______________________________________
>

Sorry, I asked the wrong question; callback_t is a function pointer, so 8 bytes is expected.
What is sizeof(myst_args) both in C and Python?

Hi Amaury.

They are both 12 bytes.

Matt

It seems that any argument greater than 8 bytes is automatically converted
to a references. Thus, changing to using ctypes.POINTER works. For example:

callback_t = ctypes.CFUNCTYPE(None, ctypes.POINTER(myst_args))

Quite a simple solution in the end. Is it worth documenting this on the
ctypes page?

Thanks for your help,

Matt

On 18 March 2013 15:19, Matt Clarke <report@bugs.python.org> wrote:

>
> Matt Clarke added the comment:
>
> Hi Amaury.
>
> They are both 12 bytes.
>
> Matt
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue17310>
> _______________________________________
>

Although Matt was able to work around his problem, this problem seems to still be present in 2.7 as well as all 3.x versions. I think I've found the cause of the problem: in 64-bit code, the calling conventions for passing structures by value are different. From this page:

https://msdn.microsoft.com/en-us/library/zthk2dkh(v=vs.90).aspx

I would point to "Structs/unions of size 8, 16, 32, or 64 bits and __m64 will be passed as if they were integers of the same size. Structs/unions other than these sizes will be passed as a pointer to memory allocated by the caller. For these aggregate types passed as a pointer (including __m128), the caller-allocated temporary memory will be 16-byte aligned."

The code in ffi_prep_incoming_args_SYSV (see https://hg.python.org/cpython/file/a1b76c1c3be8/Modules/_ctypes/libffi_msvc/ffi.c#l368) assumes (see lines 399, 402) that all value parameters are always passed inline on the stack - which is plainly not as per the documentation I linked to, for 64-bit code.

I note that the code for CFFI has a fix in ffi_prep_incoming_args_SYSV, as follows:

#ifdef _WIN64
      if (z > 8)
        {
          /* On Win64, if a single argument takes more than 8 bytes,
             then it is always passed by reference. */
          *p_argv = *((void**) argp);
          z = 8;
        }
      else
#endif
      *p_argv = (void*) argp; /* The original code, works for 32-bit */

(Source: https://bitbucket.org/cffi/cffi/src/abc8ff5b2885b3d9f22bbb314a011b8dd63c9e14/c/libffi_msvc/ffi.c?at=default)

I can confirm that the CFFI patch works as expected on Python 2.7.10.

This also fixes python 3.5

I've attached a patch with an extra fix to the duplicated issue 20160 http://bugs.python.org/issue20160

I'm closing this as a duplicate of issue 20160 - it appears to be the same bug.