classification
Title: update internal libffi copy to 3.2.1
Type: Stage: resolved
Components: ctypes Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: rejected
Dependencies: Superseder: Deprecate building with bundled copy of libffi on non-Darwin POSIX platforms
View: 27976
Assigned To: Nosy List: Arfrever, berker.peksag, chris.jerdonek, doko, fabiovmp, gustavotemple, koobs, lemburg, meador.inge, ned.deily, r.david.murray, steve.dower, tim.golden, xdegaye, yan12125, zach.ware
Priority: normal Keywords: patch

Created on 2014-12-18 17:13 by gustavotemple, last changed 2016-09-06 17:58 by zach.ware. This issue is now closed.

Files
File name Uploaded Description Edit
libffi.patch gustavotemple, 2014-12-18 17:12 review
Messages (22)
msg232890 - (view) Author: Gustavo Frederico Temple Pedrosa (gustavotemple) * Date: 2014-12-18 17:12
Link to the file:
https://github.com/gustavotemple/cpython/pull/2.diff

Link to the changes:
https://github.com/gustavotemple/cpython/pull/2/commits
msg232891 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-12-18 17:17
some issues:

 - the local change for Windows still needs upstream forwarding.
   Steve, any progress on this?

 - the libffi.diff is not updated (including Steve's changes)

 - this should be applied to 2.7 as well.
msg232893 - (view) Author: Gustavo Frederico Temple Pedrosa (gustavotemple) * Date: 2014-12-18 17:59
@doko, sorry, but what are the Steve's changes? The issue #22733?
msg232897 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-12-18 18:43
I think Matthias is referring to #20160, but as far as I could tell libffi is multiple versions ahead of the version in Python and already has the fixes. I was told to wait for it to be submitted/accepted upstream, so I've been waiting :)
msg232899 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2014-12-18 18:52
you should actively drive upstream integration, and ping patches if they are not addressed.
msg232900 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-12-18 19:00
As I mentioned on the other post, libffi's current version bears no relation to what we have in CPython, so the patches don't apply. I'm not planning on rewriting CPython patches so that they will apply to libffi, nor do I intend to replace our current version of it with libffi's latest. There's nothing to send upstream.
msg232903 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2014-12-18 20:30
Will this also fix http://bugs.python.org/issue23042 ?
msg232934 - (view) Author: Gustavo Frederico Temple Pedrosa (gustavotemple) * Date: 2014-12-19 15:30
@steve.dower, so, can I abandon this issue?
msg232935 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-12-19 15:51
I'm not entirely familiar with our layout of libffi, but I don't think your patch will affect Windows at all. If your intent was to update the version used in Windows builds, then you may as well abandon it. I'm not the one to ask about other platforms - apparently we need new ctypes maintainers...
msg232936 - (view) Author: Gustavo Frederico Temple Pedrosa (gustavotemple) * Date: 2014-12-19 15:55
@steve.dower, so no problems, because my patch won't affect Windows.
msg243198 - (view) Author: Fabio Perez (fabiovmp) * Date: 2015-05-14 15:43
Ping
msg243261 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-05-15 09:25
issue23042 is fixed now. libffi 3.2.1 apparently has the same issue, so the issue23042 patch would probably have to be reapplied (slightly modified, though).

Seeing that libffi has had a major compilation problem breaking it on at least FreeBSD and most probably a lot of other x86 based systems as well, I don't really have much confidence in the libffi maintenance, so the usual "latest is the greatest" may not be the best approach.

Are there any issue *we* have with libffi which an update to 3.2.1 would solve ?
msg269893 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2016-07-06 18:24
> Are there any issue *we* have with libffi which an update to 3.2.1 would solve ?

There are these Ubuntu-specific compiler warnings:
https://bugs.python.org/issue25077

which Berker says have been fixed in upstream libffi: https://bugs.python.org/issue25077#msg266068
msg269897 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-07-06 18:58
> which Berker says have been fixed in upstream libffi: https://bugs.python.org/issue25077#msg266068

Correct, here is the actual commit: https://github.com/libffi/libffi/commit/4a677a425c18eda4bc5357b2485da57f133f908d

I have also sent a patch to upstream libffi for https://github.com/python/cpython/blob/master/Modules/_ctypes/libffi.diff#L161: https://github.com/libffi/libffi/commit/1e82e1cda43dacd8b6ab2d9ac4db33523d86f5dc

But both of them will be released in libffi 4.0.

Note that the current master branch of libffi doesn't seem to be compiled with a C89 compiler (see https://github.com/libffi/libffi/issues/218 for details). See also https://mail.python.org/pipermail/python-dev/2016-June/144816.html for discussion about C99 support in CPython.
msg269907 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-07-06 20:06
+1 for this. Cross-compile CPython for ARM with clang fails in _ctypes due to https://llvm.org/bugs/show_bug.cgi?id=20595. This bug is already fixed in libffi.

By the way, I can't apply libffi.patch to the default branch. Some hunks are rejcted. Is there something wrong?

issue26942 may also be relevant. I did't test it as I'm not sure how to update CPython's internal libffi.
msg269908 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-07-06 20:21
Oops the proglem of clang on ARM is not completely fixed in libffi 3.2.1. This commit [1] fixes only one `stmeqia` but not another.

[1] https://github.com/libffi/libffi/commit/6eff9ff9e72463b9783be2514f944b6f05692054#diff-c6400d42bf23866ded49558ca9a54205R220
msg271388 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2016-07-26 16:03
> Cross-compile CPython for ARM with clang fails in _ctypes due to https://llvm.org/bugs/show_bug.cgi?id=20595. This bug is already fixed in libffi.

I have entered issue 27627 for this problem.
msg271425 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-07-27 01:14
Forgive me for asking a question that may have already been asked, or beaten to death, but what is preventing Python from requiring libffi as an external/required dependency, rather than keeping it and taking on the burden of fixes/backporting in lieu of updates or pending releases from upstream?

Historically (at least the last ~2-3 years), libffi in Python has been plagued with, at least:

* Inconsistent / incorrect merging of libffi fixes (including regressions)
* Unsolved issues in vendored copy that have been fixed/released upstream
* Complex, manual and error-prone updates to vendored copy
* Lack of regular maintenance, from what largely appears to be a lack of knowledge about, or confidence in updating the vendored copy (fear of breakage)

I know at least FreeBSD currently requires --sytem-libffi for i386 systems in certain versions due to issue 22521 (issue 23042) and there are currently 50 open issues matching libffi (granted not all of them will be root-caused by libffi internal). I note that number to highlight the maintenance requirement.
msg271453 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-07-27 14:11
See msg243261, I think.  On linux and freebsd we could theoretically leave that to the distros, but on OSX we have to deal with it ourselves.
msg271474 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-07-27 22:28
OSX and Windows are both exempt from this discussion.  Each has its own private copy of an ancient version of libffi: Modules/_ctypes/libffi_osx/ and Modules/_ctypes/libffy_msvc/.

I would be in favor of switching to --with-system-ffi by default for 3.6 and deprecating building with the bundled full copy of libffi (Modules/_ctypes/libffi/), to be removed in 3.7.  OSX would also default to --with-system-ffi, but would still fall back to its private copy when a system copy isn't available.

It seems like most Linux and BSD distributions would be better off with system-ffi.  I'm not sure how cross-builds would have to deal with things in 3.7, but they already have to deal with dependencies like openssl, bz2, and lzma; adding libffi to that mix shouldn't be that big of a deal.

Adding Ned to confirm that building OSX installers wouldn't be affected as long as we don't touch Modules/_ctypes/libffi_osx/ and there's not a system copy of libffi available.
msg271523 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-07-28 11:16
We should be able to make things work for OS X installer builds one way or another so don't let that be a factor.
msg274577 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-06 17:58
I'm closing this as superseded by #27976, which deprecates building with the bundled libffi.
History
Date User Action Args
2016-09-06 17:58:36zach.waresetstatus: open -> closed
superseder: Deprecate building with bundled copy of libffi on non-Darwin POSIX platforms
messages: + msg274577

resolution: rejected
stage: resolved
2016-07-28 11:16:18ned.deilysetmessages: + msg271523
2016-07-27 22:28:26zach.waresetnosy: + ned.deily
2016-07-27 22:28:07zach.waresetmessages: + msg271474
2016-07-27 14:11:47r.david.murraysetnosy: + r.david.murray
messages: + msg271453
2016-07-27 01:14:07koobssetmessages: + msg271425
2016-07-26 16:03:02xdegayesetnosy: + xdegaye
messages: + msg271388
2016-07-06 20:21:05yan12125setmessages: + msg269908
2016-07-06 20:18:08zach.waresetversions: + Python 3.6, - Python 3.4
2016-07-06 20:06:04yan12125setnosy: + yan12125
messages: + msg269907
2016-07-06 18:58:18berker.peksagsetnosy: + berker.peksag
messages: + msg269897
2016-07-06 18:24:19chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg269893
2015-05-15 09:25:56lemburgsetmessages: + msg243261
2015-05-14 15:43:24fabiovmpsetnosy: + fabiovmp
messages: + msg243198
2015-01-02 02:50:15koobssetnosy: + koobs
2014-12-19 15:55:02gustavotemplesetmessages: + msg232936
2014-12-19 15:51:53steve.dowersetmessages: + msg232935
2014-12-19 15:30:30gustavotemplesetmessages: + msg232934
2014-12-18 23:57:46Arfreversetnosy: + Arfrever
2014-12-18 20:30:12lemburgsetnosy: + lemburg
messages: + msg232903
2014-12-18 19:00:52steve.dowersetmessages: + msg232900
2014-12-18 18:52:07dokosetmessages: + msg232899
2014-12-18 18:43:38steve.dowersetmessages: + msg232897
2014-12-18 17:59:28gustavotemplesetmessages: + msg232893
2014-12-18 17:22:23gustavotemplesettitle: update internal libffi copy to 3.1 -> update internal libffi copy to 3.2.1
2014-12-18 17:17:56dokosetnosy: + tim.golden, zach.ware, steve.dower
messages: + msg232891
2014-12-18 17:13:23gustavotemplecreate