classification
Title: ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC
Type: behavior Stage: patch review
Components: ctypes Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: theller Nosy List: Arfrever, BreamoreBoy, agoode, dmalcolm, flox, lukasz.langa, mark.dickinson, pitrou, ronaldoussoren, srid, theller
Priority: normal Keywords: needs review, patch

Created on 2009-03-18 03:55 by agoode, last changed 2012-07-21 12:38 by flox. This issue is now closed.

Files
File name Uploaded Description Edit
ctypes-newffi.patch agoode, 2009-03-18 03:55 Patch to make ctypes work on Fedora (against Python 2.6)
issue5504-linux.patch theller, 2009-05-05 19:20 Patch for Python trunk, linux only.
issue5504-py27.patch theller, 2010-08-03 20:18 Patch against release27-maint branch, rev 83687.
issue5504-py27-2.patch theller, 2010-08-08 17:54
issue9662.patch lukasz.langa, 2010-09-05 13:46
Messages (21)
msg83734 - (view) Author: Adam Goode (agoode) Date: 2009-03-18 03:55
On Fedora systems, it is invalid to mmap something with PROT_WRITE and
PROT_EXEC. libffi has been updated to support this, but ctypes has not
been updated to use this new functionality.

Attached is a patch which currently only works if system libffi is used.
Though the embedded version of libffi is new enough, it is missing the
allocation and free functions. I know how I would update the ctypes
libffi/ directory (add the alloc/free files), but not sure about the
other libffi directories (darwin, arm_wince, msvc, osx). I suppose those
would all need to be upgraded, or perhaps even made to use the standard
libffi instead of special forks of it.

Note that this also eliminates the malloc_closure code.
msg87109 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-05-04 09:20
Hm, I don't see any problems with current Python trunk or the
release26-maint branch, on a Fedora 10 system.
msg87187 - (view) Author: Adam Goode (agoode) Date: 2009-05-04 22:22
Currently there is an issue where allow_execstack implies allow_execmem.
Even though allow_execmem is default to off, allow_execstack is default
to on. If this issue is fixed, or if the administrator sets
allow_execstack to off, ctypes will fail.

Try this as root, then repeat your test:
semanage boolean -m --off allow_execstack
msg87227 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-05-05 08:48
> Try this as root, then repeat your test:
> semanage boolean -m --off allow_execstack

Ok, I can reproduce the problem now.  Thanks!
msg87274 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2009-05-05 19:20
Here is a patch for Python trunk (linux only).
msg111840 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-28 20:34
Could someone with linux experience please review the patch.
msg111843 - (view) Author: Dave Malcolm (dmalcolm) (Python committer) Date: 2010-07-28 20:48
FWIW, the patch for this that I'm currently applying to Fedora's python 2.7 rpms can be seen at:
http://cvs.fedoraproject.org/viewvc/devel/python/python-2.7rc1-ctypes-noexecmem.patch?revision=1.1&content-type=text%2Fplain&view=co

It doesn't contain the rebase of libffi (since we use the system copy of libffi in our builds), but otherwise I believe that it's essentially equivalent to:
http://bugs.python.org/file13897/issue5504-linux.patch
albeit rebased to 2.7, taking into account the whitespace cleanup changes.
msg112155 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2010-07-31 15:12
Issue #9385 marked as duplicate.
msg112664 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-08-03 20:18
The libffi library included with Python has been updated in the meantime, so most of the issue5504-linux.patch is unneeded now.

Here is a new patch, issue5504-py27.patch, against the current release27-maint branch.  Seems it is exactly the same as the fedora patch that dmalcolm mentioned.

If someone can test the patch on an selinux system it would be great.
msg113291 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-08-08 17:54
issue5504-py27-2.patch is the updated patch that now also works on Windows.
msg113292 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-08-08 17:59
Fixed in rev 83836 (release27-maint branch) and rev. 83837 (py3k branch).
msg113306 - (view) Author: Łukasz Langa (lukasz.langa) (Python committer) Date: 2010-08-08 20:07
Thomas, it seems this change doesn't work for py3k. Buildbots complain and my working copy does as well.

Example:

http://www.python.org/dev/buildbot/all/builders/x86%20gentoo%203.1/builds/990
msg113460 - (view) Author: Thomas Heller (theller) * (Python committer) Date: 2010-08-09 19:47
> Thomas, it seems this change doesn't work for py3k. Buildbots complain and my working copy does as well.
> 
> Example:
> 
> http://www.python.org/dev/buildbot/all/builders/x86%20gentoo%203.1/builds/990

I do not know why this happens.  It works for me, inside my sandbox
as well as on a newly checkout copy.
Anyway, I'm not able to fix it since I'm going to holidays.
msg113501 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-10 00:18
> I do not know why this happens.

The failure is in 3.1, and the reason is quite obvious: the patch relies on libffi/src/closures.c, which doesn't exist in the 3.1 copy of libffi.
I've reverted your commit in r83924, since you say you won't be able to look at it immediately. Unfortunately, this also reverts the refcounting fix.
msg113502 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-10 00:22
Correction: the revert was done in r83926.
msg113814 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-13 18:59
Since r83837, the py3k _ctypes module fails to build on my OS X 10.6 machine:

*** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-10.6-x86_64-3.2-pydebug/_ctypes.so, 2): Symbol not found: _ffi_closure_alloc
  Referenced from: /Users/dickinsm/python/svn/py3k/build/lib.macosx-10.6-x86_64-3.2-pydebug/_ctypes.so
  Expected in: flat namespace
 in /Users/dickinsm/python/svn/py3k/build/lib.macosx-10.6-x86_64-3.2-pydebug/_ctypes.so

This looks like a different issue from the one above, though.
msg113815 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-13 19:00
Same error on the buildbots, here:

http://www.python.org/dev/buildbot/3.x/builders/PPC%20Leopard%203.x/builds/288/steps/compile/logs/stdio
msg115242 - (view) Author: Sridhar Ratnakumar (srid) Date: 2010-08-30 20:58
Like Mark, I too see an error with ctypes due to this change:

*** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-10.5-intel-3.2/_ctypes.so, 2): Symbol not found: _ffi_closure_alloc
  Referenced from: /Users/sridharr/as/apy/branches/32a1ssl1/build/pyhg_branches_py3k-macosx-hgtip32/python/build/lib.macosx-10.5-intel-3.2/_ctypes.so
  Expected in: flat namespace
 in /Users/sridharr/as/apy/branches/32a1ssl1/build/pyhg_branches_py3k-macosx-hgtip32/python/build/lib.macosx-10.5-intel-3.2/_ctypes.so

MacOSX 10.6 | built with 10.5 SDK | i386 and x86_64 arch | ActivePython 3.2 internal build
msg115647 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-09-05 13:44
Fixing the alloc_closure error is easy enough:


Index: ../setup.py
===================================================================
--- ../setup.py	(revision 84528)
+++ ../setup.py	(working copy)
@@ -1653,7 +1653,9 @@
                    '_ctypes/callbacks.c',
                    '_ctypes/callproc.c',
                    '_ctypes/stgdict.c',
-                   '_ctypes/cfield.c']
+                   '_ctypes/cfield.c',
+                   '_ctypes/malloc_closure.c',
+                   ]
         depends = ['_ctypes/ctypes.h']
 
         if sys.platform == 'darwin':


That's not enough to make it possible to build ctypes on OSX though, I now get the same error for a different symbol:


*** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-10.6-fat-3.2/_ctypes.so, 2): Symbol not found: _ffi_prep_closure_loc
  Referenced from: /Users/ronald/Projects/python/python-3.x/build/build/lib.macosx-10.6-fat-3.2/_ctypes.so
  Expected in: flat namespace
 in /Users/ronald/Projects/python/python-3.x/build/build/lib.macosx-10.6-fat-3.2/_ctypes.so
msg115648 - (view) Author: Łukasz Langa (lukasz.langa) (Python committer) Date: 2010-09-05 13:46
Please find attached a "fix" for #9662. Some explanation is in order:

- the Windows, Linux and OS X implementations of FFI included in the SVN are different in terms of maturity
- Thomas originally when fixing #5504 used a bit of functionality regarding closures that wasn't there in the Windows and OS X versions
- he did some dirty hacking on the Windows version to make it work: changed the ffi_prep_closure function header to ffi_prep_closure_loc (leaving the old implementation) and introduced names for ffi_closure_alloc and ffi_closure_free
- he didn't touch the OS X version so it failed

I did more-less the same thing but in case of OS X we have many ffi_prep_closure implementations for different archs so I simply did an #ifdef in the place ffi_prep_closure is used.

I think a better solution ultimately would be to update all implementation to FFI 3.0.9 and be done with it. Still, the current patch works for me.
msg115652 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2010-09-05 14:19
Łukasz' patch fixes the issue for me.

As the patch only affects code-paths used on OSX (patches to the libffi version for OSX and an #ifdef that makes OSX use ffi_prep_closure instead of ffi_prep_closure_loc) I intend to commit the patch later today to be able to build a working copy of the OSX installer for the next alpha release.

I'm at best -0 on moving all platforms to libffi 3.0.9. 'libffi_osx' works properly on OSX and I have no idea whether all changes in that fork of libffi have been merged back into the canonical version.
History
Date User Action Args
2012-07-21 12:38:58floxsetstatus: open -> closed
2010-09-05 14:19:26ronaldoussorensetmessages: + msg115652
2010-09-05 13:46:49lukasz.langasetfiles: + issue9662.patch

messages: + msg115648
2010-09-05 13:44:45ronaldoussorensetnosy: + ronaldoussoren
messages: + msg115647
2010-08-30 20:58:02sridsetnosy: + srid
messages: + msg115242
2010-08-13 19:00:41mark.dickinsonsetmessages: + msg113815
2010-08-13 18:59:26mark.dickinsonsetnosy: + mark.dickinson
messages: + msg113814
2010-08-10 00:22:18pitrousetmessages: + msg113502
2010-08-10 00:18:54pitrousetnosy: + pitrou
messages: + msg113501
2010-08-09 19:47:35thellersetmessages: + msg113460
title: ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC -> ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC
2010-08-08 20:09:16pitrousetstatus: closed -> open
2010-08-08 20:07:33lukasz.langasetnosy: + lukasz.langa
messages: + msg113306
2010-08-08 17:59:27thellersetstatus: open -> closed
resolution: fixed
messages: + msg113292
2010-08-08 17:54:48thellersetfiles: + issue5504-py27-2.patch

messages: + msg113291
2010-08-03 20:18:56thellersetfiles: + issue5504-py27.patch

messages: + msg112664
2010-08-03 19:29:45terry.reedysetversions: - Python 2.6
2010-07-31 15:12:31floxsetnosy: + flox
messages: + msg112155
2010-07-31 15:10:19floxlinkissue9385 superseder
2010-07-28 20:48:02dmalcolmsetmessages: + msg111843
2010-07-28 20:40:16Arfreversetnosy: + Arfrever
title: ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC -> ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC

versions: + Python 3.1, Python 2.7, Python 3.2
2010-07-28 20:34:08BreamoreBoysetnosy: + BreamoreBoy
messages: + msg111840
2010-02-09 17:01:05brian.curtinsetpriority: normal
keywords: + needs review
type: crash -> behavior
stage: patch review
2010-01-29 23:43:36dmalcolmsetnosy: + dmalcolm
2009-05-05 19:21:05thellersetfiles: + issue5504-linux.patch

messages: + msg87274
2009-05-05 08:48:52thellersetmessages: + msg87227
title: ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC -> ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC
2009-05-04 22:22:41agoodesetmessages: + msg87187
2009-05-04 09:20:23thellersetmessages: + msg87109
2009-03-18 03:55:29agoodecreate