update internal libffi copy to 3.2.1 #67274

gustavotemple · 2014-12-18T17:13:24Z

BPO	23085
Nosy	@malemburg, @doko42, @tjguk, @ned-deily, @bitdancer, @meadori, @cjerdonek, @xdegaye, @berkerpeksag, @zware, @koobs, @zooba, @yan12125
Superseder	bpo-27976: Deprecate building with bundled copy of libffi on non-Darwin POSIX platforms
Files	libffi.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-06.17:58:36.979>
created_at = <Date 2014-12-18.17:13:23.566>
labels = ['ctypes']
title = 'update internal libffi copy to 3.2.1'
updated_at = <Date 2016-09-06.17:58:36.977>
user = 'https://bugs.python.org/gustavotemple'

bugs.python.org fields:

activity = <Date 2016-09-06.17:58:36.977>
actor = 'zach.ware'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-06.17:58:36.979>
closer = 'zach.ware'
components = ['ctypes']
creation = <Date 2014-12-18.17:13:23.566>
creator = 'gustavotemple'
dependencies = []
files = ['37496']
hgrepos = []
issue_num = 23085
keywords = ['patch']
message_count = 22.0
messages = ['232890', '232891', '232893', '232897', '232899', '232900', '232903', '232934', '232935', '232936', '243198', '243261', '269893', '269897', '269907', '269908', '271388', '271425', '271453', '271474', '271523', '274577']
nosy_count = 16.0
nosy_names = ['lemburg', 'doko', 'tim.golden', 'ned.deily', 'Arfrever', 'r.david.murray', 'meador.inge', 'chris.jerdonek', 'xdegaye', 'berker.peksag', 'zach.ware', 'koobs', 'steve.dower', 'gustavotemple', 'fabiovmp', 'yan12125']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = '27976'
type = None
url = 'https://bugs.python.org/issue23085'
versions = ['Python 3.5', 'Python 3.6']

gustavotemple · 2014-12-18T17:12:09Z

Link to the file:
https://github.com/gustavotemple/cpython/pull/2.diff

Link to the changes:
https://github.com/gustavotemple/cpython/pull/2/commits

doko42 · 2014-12-18T17:17:56Z

some issues:

the local change for Windows still needs upstream forwarding.
Steve, any progress on this?
the libffi.diff is not updated (including Steve's changes)
this should be applied to 2.7 as well.

gustavotemple · 2014-12-18T17:59:29Z

@Doko, sorry, but what are the Steve's changes? The issue bpo-22733?

zooba · 2014-12-18T18:43:39Z

I think Matthias is referring to bpo-20160, but as far as I could tell libffi is multiple versions ahead of the version in Python and already has the fixes. I was told to wait for it to be submitted/accepted upstream, so I've been waiting :)

doko42 · 2014-12-18T18:52:08Z

you should actively drive upstream integration, and ping patches if they are not addressed.

zooba · 2014-12-18T19:00:52Z

As I mentioned on the other post, libffi's current version bears no relation to what we have in CPython, so the patches don't apply. I'm not planning on rewriting CPython patches so that they will apply to libffi, nor do I intend to replace our current version of it with libffi's latest. There's nothing to send upstream.

malemburg · 2014-12-18T20:30:13Z

Will this also fix http://bugs.python.org/issue23042 ?

gustavotemple · 2014-12-19T15:30:31Z

@steve.dower, so, can I abandon this issue?

zooba · 2014-12-19T15:51:53Z

I'm not entirely familiar with our layout of libffi, but I don't think your patch will affect Windows at all. If your intent was to update the version used in Windows builds, then you may as well abandon it. I'm not the one to ask about other platforms - apparently we need new ctypes maintainers...

gustavotemple · 2014-12-19T15:55:03Z

@steve.dower, so no problems, because my patch won't affect Windows.

fabiovmp · 2015-05-14T15:43:25Z

Ping

malemburg · 2015-05-15T09:25:56Z

bpo-23042 is fixed now. libffi 3.2.1 apparently has the same issue, so the bpo-23042 patch would probably have to be reapplied (slightly modified, though).

Seeing that libffi has had a major compilation problem breaking it on at least FreeBSD and most probably a lot of other x86 based systems as well, I don't really have much confidence in the libffi maintenance, so the usual "latest is the greatest" may not be the best approach.

Are there any issue *we* have with libffi which an update to 3.2.1 would solve ?

cjerdonek · 2016-07-06T18:24:19Z

Are there any issue *we* have with libffi which an update to 3.2.1 would solve ?

There are these Ubuntu-specific compiler warnings:
https://bugs.python.org/issue25077

which Berker says have been fixed in upstream libffi: https://bugs.python.org/issue25077#msg266068

berkerpeksag · 2016-07-06T18:58:18Z

which Berker says have been fixed in upstream libffi: https://bugs.python.org/issue25077#msg266068

Correct, here is the actual commit: libffi/libffi@4a677a4

I have also sent a patch to upstream libffi for https://github.com/python/cpython/blob/master/Modules/_ctypes/libffi.diff#L161: libffi/libffi@1e82e1c

But both of them will be released in libffi 4.0.

Note that the current master branch of libffi doesn't seem to be compiled with a C89 compiler (see libffi/libffi#218 for details). See also https://mail.python.org/pipermail/python-dev/2016-June/144816.html for discussion about C99 support in CPython.

yan12125 · 2016-07-06T20:06:04Z

+1 for this. Cross-compile CPython for ARM with clang fails in _ctypes due to https://llvm.org/bugs/show_bug.cgi?id=20595. This bug is already fixed in libffi.

By the way, I can't apply libffi.patch to the default branch. Some hunks are rejcted. Is there something wrong?

bpo-26942 may also be relevant. I did't test it as I'm not sure how to update CPython's internal libffi.

yan12125 · 2016-07-06T20:21:06Z

Oops the proglem of clang on ARM is not completely fixed in libffi 3.2.1. This commit [1] fixes only one stmeqia but not another.

[1] libffi/libffi@6eff9ff#diff-c6400d42bf23866ded49558ca9a54205R220

xdegaye · 2016-07-26T16:03:02Z

Cross-compile CPython for ARM with clang fails in _ctypes due to https://llvm.org/bugs/show_bug.cgi?id=20595. This bug is already fixed in libffi.

I have entered bpo-27627 for this problem.

koobs · 2016-07-27T01:14:07Z

Forgive me for asking a question that may have already been asked, or beaten to death, but what is preventing Python from requiring libffi as an external/required dependency, rather than keeping it and taking on the burden of fixes/backporting in lieu of updates or pending releases from upstream?

Historically (at least the last ~2-3 years), libffi in Python has been plagued with, at least:

Inconsistent / incorrect merging of libffi fixes (including regressions)
Unsolved issues in vendored copy that have been fixed/released upstream
Complex, manual and error-prone updates to vendored copy
Lack of regular maintenance, from what largely appears to be a lack of knowledge about, or confidence in updating the vendored copy (fear of breakage)

I know at least FreeBSD currently requires --sytem-libffi for i386 systems in certain versions due to bpo-22521 (bpo-23042) and there are currently 50 open issues matching libffi (granted not all of them will be root-caused by libffi internal). I note that number to highlight the maintenance requirement.

bitdancer · 2016-07-27T14:11:47Z

See msg243261, I think. On linux and freebsd we could theoretically leave that to the distros, but on OSX we have to deal with it ourselves.

zware · 2016-07-27T22:28:07Z

OSX and Windows are both exempt from this discussion. Each has its own private copy of an ancient version of libffi: Modules/_ctypes/libffi_osx/ and Modules/_ctypes/libffy_msvc/.

I would be in favor of switching to --with-system-ffi by default for 3.6 and deprecating building with the bundled full copy of libffi (Modules/_ctypes/libffi/), to be removed in 3.7. OSX would also default to --with-system-ffi, but would still fall back to its private copy when a system copy isn't available.

It seems like most Linux and BSD distributions would be better off with system-ffi. I'm not sure how cross-builds would have to deal with things in 3.7, but they already have to deal with dependencies like openssl, bz2, and lzma; adding libffi to that mix shouldn't be that big of a deal.

Adding Ned to confirm that building OSX installers wouldn't be affected as long as we don't touch Modules/_ctypes/libffi_osx/ and there's not a system copy of libffi available.

ned-deily · 2016-07-28T11:16:18Z

We should be able to make things work for OS X installer builds one way or another so don't let that be a factor.

zware · 2016-09-06T17:58:37Z

I'm closing this as superseded by bpo-27976, which deprecates building with the bundled libffi.

gustavotemple mannequin added the topic-ctypes label Dec 18, 2014

gustavotemple mannequin changed the title ~~update internal libffi copy to 3.1~~ update internal libffi copy to 3.2.1 Dec 18, 2014

zware closed this as completed Sep 6, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update internal libffi copy to 3.2.1 #67274

update internal libffi copy to 3.2.1 #67274

gustavotemple mannequin commented Dec 18, 2014

gustavotemple mannequin commented Dec 18, 2014

doko42 commented Dec 18, 2014

gustavotemple mannequin commented Dec 18, 2014

zooba commented Dec 18, 2014

doko42 commented Dec 18, 2014

zooba commented Dec 18, 2014

malemburg commented Dec 18, 2014

gustavotemple mannequin commented Dec 19, 2014

zooba commented Dec 19, 2014

gustavotemple mannequin commented Dec 19, 2014

fabiovmp mannequin commented May 14, 2015

malemburg commented May 15, 2015

cjerdonek commented Jul 6, 2016

berkerpeksag commented Jul 6, 2016

yan12125 mannequin commented Jul 6, 2016

yan12125 mannequin commented Jul 6, 2016

xdegaye mannequin commented Jul 26, 2016

koobs commented Jul 27, 2016

bitdancer commented Jul 27, 2016

zware commented Jul 27, 2016

ned-deily commented Jul 28, 2016

zware commented Sep 6, 2016

update internal libffi copy to 3.2.1 #67274

update internal libffi copy to 3.2.1 #67274

Comments

gustavotemple mannequin commented Dec 18, 2014

gustavotemple mannequin commented Dec 18, 2014

doko42 commented Dec 18, 2014

gustavotemple mannequin commented Dec 18, 2014

zooba commented Dec 18, 2014

doko42 commented Dec 18, 2014

zooba commented Dec 18, 2014

malemburg commented Dec 18, 2014

gustavotemple mannequin commented Dec 19, 2014

zooba commented Dec 19, 2014

gustavotemple mannequin commented Dec 19, 2014

fabiovmp mannequin commented May 14, 2015

malemburg commented May 15, 2015

cjerdonek commented Jul 6, 2016

berkerpeksag commented Jul 6, 2016

yan12125 mannequin commented Jul 6, 2016

yan12125 mannequin commented Jul 6, 2016

xdegaye mannequin commented Jul 26, 2016

koobs commented Jul 27, 2016

bitdancer commented Jul 27, 2016

zware commented Jul 27, 2016

ned-deily commented Jul 28, 2016

zware commented Sep 6, 2016