classification
Title: WINFUNCTYPE wrapped ctypes callbacks not functioning correctly in Python 2.7
Type: crash Stage: resolved
Components: ctypes Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: theller Nosy List: benjamin.peterson, brian.curtin, georg.brandl, jteh, mdcurran, theller
Priority: release blocker Keywords:

Created on 2010-06-10 07:05 by mdcurran, last changed 2010-09-27 19:00 by brian.curtin. This issue is now closed.

Files
File name Uploaded Description Edit
test_callbackRetval.py mdcurran, 2010-06-10 07:05 Example 1
test_keyHook.py mdcurran, 2010-06-10 07:07 Example 2
Messages (15)
msg107447 - (view) Author: Michael Curran (mdcurran) Date: 2010-06-10 07:05
There seem to be problems with WINFUNCTYPE callbacks causing exceptions, and or getting their return value ignored by the caller, when using Python 2.7 rc1.
Two examples provided.

Example 1:
Providing a WINFUNCTYPE wrapped python function when calling EnumWindows from user32.dll with ctypes, EnumWindows enumerates (calls the python function) for the first 5 windows and then causes a WindowsError, sometimes an access violation (reading, or writing), or even other strange unknown exceptions.
This specific testcase is attached to the bug.
Run this script in  Python 2.7 you should see the WindowsErrors. On Python 2.6 it successfully enumerates through all windows.

Example 2:
Hooking low-level keyboard input in Windows with SetWindowsHookEx from user32.dll, providing a WINFUNCTYPE wrapped python function as the callback, and then typing some keys, causes the callback to be called but the return value is always ignored, which means the hook never blocks keys completely, even if  the value 1 is returned from the callback, to do so.
Again, a testcase is attached to the bug.
Run this script in Python. It will ask you to type some characters and press enter. On Python 2.6 no characters should be echoed to the screen, but you will hear beeps to let you know the keyboard hook is seeing the keys. On Python 2.7, the keys will be echoed to the screen, and you will also hear the beeps (meaning that the hook was seeing the keys, but was unable to actually block them -- its return value was being ignored).
Also in Python 2.7, after the quit message is sent to the hook thread, GetMessageW (in user32.dll) causes a WindowsError exception. This does not happen on Python 2.6.

Perhaps it may be related to this entry in the Python 2.7 What's new:
The underlying libffi library has been updated to version 3.0.9, containing various fixes for different platforms. (Updated by Matthias Klose; issue 8142.)
msg107448 - (view) Author: Michael Curran (mdcurran) Date: 2010-06-10 07:10
I should also note that this has been tested on Windows 7 and XP (32 bit).
I have also seen cases where Python has crashed completely, rather than just causing a WindowsError, in these circomstances.
msg107524 - (view) Author: Michael Curran (mdcurran) Date: 2010-06-11 04:00
Issue does not occure using Python 3.1.2.
msg107528 - (view) Author: Michael Curran (mdcurran) Date: 2010-06-11 05:48
The issue does not occure with Python 2.7 beta 1, but does occure in Python 2.7 beta 2.

So, with this in mind, its probably more likely that it may have been caused by the following change:
"- On Windows, ctypes does no longer check the stack before and after calling a
  foreign function.  This allows to use the unmodified libffi library."

I had previously thought it could have been the libffi upgrade, but this seems to have  happened in Beta 1.
msg107577 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-06-11 19:54
The commit that breaks the examples is revision 80761.

Tested on WinXP 32 bit.

Someone should test on Win 64bit too ;-).
msg107578 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-06-11 20:07
Problem in py3k branch also.
msg107613 - (view) Author: Michael Curran (mdcurran) Date: 2010-06-12 00:18
Tested on Windows 7 x64, with Python 2.7 rc1 [amd64]: both examples work fine.
So it seems the issue is specific to x86.
msg108236 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-06-20 14:07
Is r80761 something that can be safely reverted?
msg108243 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-06-20 19:59
> Is r80761 something that can be safely reverted?

Yes, I think so.
msg108246 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-06-20 20:44
2010/6/20 Thomas Heller <report@bugs.python.org>:
>
> Thomas Heller <theller@ctypes.org> added the comment:
>
>> Is r80761 something that can be safely reverted?
>
> Yes, I think so.

Can you do so? Thanks.
msg108279 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-06-21 16:01
Fixed in rev. 82127 (trunk) and rev 82138 (py3k).
Added test for the  issues reported here so that it doen't happen again.
msg108305 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-06-21 19:34
Thanks, Michael, for reporting this issue.
msg112464 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-08-02 13:27
The test added here is crashing my Windows 7 x64 machine on py3k in debug mode. It hangs indefinitely in release mode. This isn't occurring with the buildbots...


(Sorry for not reporting this sooner...I came to see the status and apparently the message I thought I left was not here, so, no status)
msg115636 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-09-05 08:15
Deferring for 3.2a2.
msg117463 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2010-09-27 19:00
This doesn't seem to be an issue anymore.
History
Date User Action Args
2010-09-27 19:00:52brian.curtinsetstatus: open -> closed

messages: + msg117463
stage: resolved
2010-09-06 08:26:17georg.brandlsetpriority: deferred blocker -> release blocker
2010-09-05 08:15:02georg.brandlsetpriority: release blocker -> deferred blocker

messages: + msg115636
2010-08-02 13:27:53brian.curtinsetstatus: closed -> open
nosy: + brian.curtin, georg.brandl
messages: + msg112464

2010-06-21 19:34:36thellersetmessages: + msg108305
2010-06-21 16:01:46thellersetstatus: open -> closed
resolution: fixed
messages: + msg108279
2010-06-20 20:44:45benjamin.petersonsetmessages: + msg108246
2010-06-20 19:59:56thellersetmessages: + msg108243
2010-06-20 14:07:06benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg108236
2010-06-15 10:57:07jtehsetnosy: + jteh
2010-06-12 00:18:39mdcurransetmessages: + msg107613
2010-06-11 20:07:07thellersetmessages: + msg107578
versions: + Python 3.2
2010-06-11 19:54:35thellersetpriority: normal -> release blocker

messages: + msg107577
2010-06-11 05:48:42mdcurransetmessages: + msg107528
2010-06-11 04:00:59mdcurransetmessages: + msg107524
2010-06-10 07:10:24mdcurransetmessages: + msg107448
2010-06-10 07:07:05mdcurransetfiles: + test_keyHook.py
2010-06-10 07:05:57mdcurrancreate