Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC #49754

Closed
agoode mannequin opened this issue Mar 18, 2009 · 21 comments
Closed

ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC #49754

agoode mannequin opened this issue Mar 18, 2009 · 21 comments
Assignees
Labels
topic-ctypes type-bug An unexpected behavior, bug, or error

Comments

@agoode
Copy link
Mannequin

agoode mannequin commented Mar 18, 2009

BPO 5504
Nosy @theller, @ronaldoussoren, @mdickinson, @pitrou, @florentx, @davidmalcolm, @ambv
Files
  • ctypes-newffi.patch: Patch to make ctypes work on Fedora (against Python 2.6)
  • issue5504-linux.patch: Patch for Python trunk, linux only.
  • issue5504-py27.patch: Patch against release27-maint branch, rev 83687.
  • issue5504-py27-2.patch
  • issue9662.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/theller'
    closed_at = <Date 2012-07-21.12:38:58.342>
    created_at = <Date 2009-03-18.03:55:29.824>
    labels = ['ctypes', 'type-bug']
    title = "ctypes should work with systems where mmap can't be\tPROT_WRITE and PROT_EXEC"
    updated_at = <Date 2012-07-21.12:38:58.342>
    user = 'https://bugs.python.org/agoode'

    bugs.python.org fields:

    activity = <Date 2012-07-21.12:38:58.342>
    actor = 'flox'
    assignee = 'theller'
    closed = True
    closed_date = <Date 2012-07-21.12:38:58.342>
    closer = 'flox'
    components = ['ctypes']
    creation = <Date 2009-03-18.03:55:29.824>
    creator = 'agoode'
    dependencies = []
    files = ['13363', '13897', '18352', '18439', '18759']
    hgrepos = []
    issue_num = 5504
    keywords = ['patch', 'needs review']
    message_count = 21.0
    messages = ['83734', '87109', '87187', '87227', '87274', '111840', '111843', '112155', '112664', '113291', '113292', '113306', '113460', '113501', '113502', '113814', '113815', '115242', '115647', '115648', '115652']
    nosy_count = 11.0
    nosy_names = ['theller', 'ronaldoussoren', 'mark.dickinson', 'pitrou', 'Arfrever', 'agoode', 'srid', 'flox', 'dmalcolm', 'BreamoreBoy', 'lukasz.langa']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'patch review'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue5504'
    versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

    @agoode
    Copy link
    Mannequin Author

    agoode mannequin commented Mar 18, 2009

    On Fedora systems, it is invalid to mmap something with PROT_WRITE and
    PROT_EXEC. libffi has been updated to support this, but ctypes has not
    been updated to use this new functionality.

    Attached is a patch which currently only works if system libffi is used.
    Though the embedded version of libffi is new enough, it is missing the
    allocation and free functions. I know how I would update the ctypes
    libffi/ directory (add the alloc/free files), but not sure about the
    other libffi directories (darwin, arm_wince, msvc, osx). I suppose those
    would all need to be upgraded, or perhaps even made to use the standard
    libffi instead of special forks of it.

    Note that this also eliminates the malloc_closure code.

    @agoode agoode mannequin added the type-crash A hard crash of the interpreter, possibly with a core dump label Mar 18, 2009
    @agoode agoode mannequin assigned theller Mar 18, 2009
    @agoode agoode mannequin added the topic-ctypes label Mar 18, 2009
    @theller
    Copy link

    theller commented May 4, 2009

    Hm, I don't see any problems with current Python trunk or the
    release26-maint branch, on a Fedora 10 system.

    @agoode
    Copy link
    Mannequin Author

    agoode mannequin commented May 4, 2009

    Currently there is an issue where allow_execstack implies allow_execmem.
    Even though allow_execmem is default to off, allow_execstack is default
    to on. If this issue is fixed, or if the administrator sets
    allow_execstack to off, ctypes will fail.

    Try this as root, then repeat your test:
    semanage boolean -m --off allow_execstack

    @theller
    Copy link

    theller commented May 5, 2009

    Try this as root, then repeat your test:
    semanage boolean -m --off allow_execstack

    Ok, I can reproduce the problem now. Thanks!

    @theller theller changed the title ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC May 5, 2009
    @theller
    Copy link

    theller commented May 5, 2009

    Here is a patch for Python trunk (linux only).

    @briancurtin briancurtin added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Feb 9, 2010
    @BreamoreBoy
    Copy link
    Mannequin

    BreamoreBoy mannequin commented Jul 28, 2010

    Could someone with linux experience please review the patch.

    @Arfrever Arfrever mannequin changed the title ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC Jul 28, 2010
    @davidmalcolm
    Copy link
    Member

    FWIW, the patch for this that I'm currently applying to Fedora's python 2.7 rpms can be seen at:
    http://cvs.fedoraproject.org/viewvc/devel/python/python-2.7rc1-ctypes-noexecmem.patch?revision=1.1&content-type=text%2Fplain&view=co

    It doesn't contain the rebase of libffi (since we use the system copy of libffi in our builds), but otherwise I believe that it's essentially equivalent to:
    http://bugs.python.org/file13897/issue5504-linux.patch
    albeit rebased to 2.7, taking into account the whitespace cleanup changes.

    @florentx
    Copy link
    Mannequin

    florentx mannequin commented Jul 31, 2010

    Issue bpo-9385 marked as duplicate.

    @theller
    Copy link

    theller commented Aug 3, 2010

    The libffi library included with Python has been updated in the meantime, so most of the bpo-5504-linux.patch is unneeded now.

    Here is a new patch, bpo-5504-py27.patch, against the current release27-maint branch. Seems it is exactly the same as the fedora patch that dmalcolm mentioned.

    If someone can test the patch on an selinux system it would be great.

    @theller
    Copy link

    theller commented Aug 8, 2010

    bpo-5504-py27-2.patch is the updated patch that now also works on Windows.

    @theller
    Copy link

    theller commented Aug 8, 2010

    Fixed in rev 83836 (release27-maint branch) and rev. 83837 (py3k branch).

    @theller theller closed this as completed Aug 8, 2010
    @ambv
    Copy link
    Contributor

    ambv commented Aug 8, 2010

    Thomas, it seems this change doesn't work for py3k. Buildbots complain and my working copy does as well.

    Example:

    http://www.python.org/dev/buildbot/all/builders/x86%20gentoo%203.1/builds/990

    @pitrou pitrou reopened this Aug 8, 2010
    @theller
    Copy link

    theller commented Aug 9, 2010

    Thomas, it seems this change doesn't work for py3k. Buildbots complain and my working copy does as well.

    Example:

    http://www.python.org/dev/buildbot/all/builders/x86%20gentoo%203.1/builds/990

    I do not know why this happens. It works for me, inside my sandbox
    as well as on a newly checkout copy.
    Anyway, I'm not able to fix it since I'm going to holidays.

    @theller theller changed the title ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC ctypes should work with systems where mmap can't be PROT_WRITE and PROT_EXEC Aug 9, 2010
    @pitrou
    Copy link
    Member

    pitrou commented Aug 10, 2010

    I do not know why this happens.

    The failure is in 3.1, and the reason is quite obvious: the patch relies on libffi/src/closures.c, which doesn't exist in the 3.1 copy of libffi.
    I've reverted your commit in r83924, since you say you won't be able to look at it immediately. Unfortunately, this also reverts the refcounting fix.

    @pitrou
    Copy link
    Member

    pitrou commented Aug 10, 2010

    Correction: the revert was done in r83926.

    @mdickinson
    Copy link
    Member

    Since r83837, the py3k _ctypes module fails to build on my OS X 10.6 machine:

    *** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-10.6-x86_64-3.2-pydebug/_ctypes.so, 2): Symbol not found: _ffi_closure_alloc
    Referenced from: /Users/dickinsm/python/svn/py3k/build/lib.macosx-10.6-x86_64-3.2-pydebug/_ctypes.so
    Expected in: flat namespace
    in /Users/dickinsm/python/svn/py3k/build/lib.macosx-10.6-x86_64-3.2-pydebug/_ctypes.so

    This looks like a different issue from the one above, though.

    @mdickinson
    Copy link
    Member

    @srid
    Copy link
    Mannequin

    srid mannequin commented Aug 30, 2010

    Like Mark, I too see an error with ctypes due to this change:

    *** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-10.5-intel-3.2/_ctypes.so, 2): Symbol not found: _ffi_closure_alloc
    Referenced from: /Users/sridharr/as/apy/branches/32a1ssl1/build/pyhg_branches_py3k-macosx-hgtip32/python/build/lib.macosx-10.5-intel-3.2/_ctypes.so
    Expected in: flat namespace
    in /Users/sridharr/as/apy/branches/32a1ssl1/build/pyhg_branches_py3k-macosx-hgtip32/python/build/lib.macosx-10.5-intel-3.2/_ctypes.so

    MacOSX 10.6 | built with 10.5 SDK | i386 and x86_64 arch | ActivePython 3.2 internal build

    @ronaldoussoren
    Copy link
    Contributor

    Fixing the alloc_closure error is easy enough:

    Index: ../setup.py
    ===================================================================

    --- ../setup.py	(revision 84528)
    +++ ../setup.py	(working copy)
    @@ -1653,7 +1653,9 @@
                        '_ctypes/callbacks.c',
                        '_ctypes/callproc.c',
                        '_ctypes/stgdict.c',
    -                   '_ctypes/cfield.c']
    +                   '_ctypes/cfield.c',
    +                   '_ctypes/malloc_closure.c',
    +                   ]
             depends = ['_ctypes/ctypes.h']
     
             if sys.platform == 'darwin':

    That's not enough to make it possible to build ctypes on OSX though, I now get the same error for a different symbol:

    *** WARNING: renaming "_ctypes" since importing it failed: dlopen(build/lib.macosx-10.6-fat-3.2/_ctypes.so, 2): Symbol not found: _ffi_prep_closure_loc
    Referenced from: /Users/ronald/Projects/python/python-3.x/build/build/lib.macosx-10.6-fat-3.2/_ctypes.so
    Expected in: flat namespace
    in /Users/ronald/Projects/python/python-3.x/build/build/lib.macosx-10.6-fat-3.2/_ctypes.so

    @ambv
    Copy link
    Contributor

    ambv commented Sep 5, 2010

    Please find attached a "fix" for bpo-9662. Some explanation is in order:

    • the Windows, Linux and OS X implementations of FFI included in the SVN are different in terms of maturity
    • Thomas originally when fixing bpo-5504 used a bit of functionality regarding closures that wasn't there in the Windows and OS X versions
    • he did some dirty hacking on the Windows version to make it work: changed the ffi_prep_closure function header to ffi_prep_closure_loc (leaving the old implementation) and introduced names for ffi_closure_alloc and ffi_closure_free
    • he didn't touch the OS X version so it failed

    I did more-less the same thing but in case of OS X we have many ffi_prep_closure implementations for different archs so I simply did an #ifdef in the place ffi_prep_closure is used.

    I think a better solution ultimately would be to update all implementation to FFI 3.0.9 and be done with it. Still, the current patch works for me.

    @ronaldoussoren
    Copy link
    Contributor

    Łukasz' patch fixes the issue for me.

    As the patch only affects code-paths used on OSX (patches to the libffi version for OSX and an #ifdef that makes OSX use ffi_prep_closure instead of ffi_prep_closure_loc) I intend to commit the patch later today to be able to build a working copy of the OSX installer for the next alpha release.

    I'm at best -0 on moving all platforms to libffi 3.0.9. 'libffi_osx' works properly on OSX and I have no idea whether all changes in that fork of libffi have been merged back into the canonical version.

    @florentx florentx mannequin closed this as completed Jul 21, 2012
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    topic-ctypes type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants