classification
Title: libffi-3.0.13 import
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.4, Python 3.3, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: Arfrever, benjamin.peterson, doko, eric.araujo, georg.brandl, gregory.p.smith, koobs, larry, python-dev, ronaldoussoren
Priority: release blocker Keywords: patch

Created on 2013-02-12 13:52 by doko, last changed 2013-04-30 08:32 by gregory.p.smith. This issue is now closed.

Files
File name Uploaded Description Edit
issue17192-refix-3.2-gps01.diff gregory.p.smith, 2013-04-30 07:20 the patch to bring the 3.2 branch libffi back where it should be. review
Messages (22)
msg181956 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-02-12 13:52
issue for tracking the libffi-3.0.12 import. checked that builds on x86_64-linux-gnu and arm-linux-gnueabihf do work.

still needs updating/checking the extra copies of:

 - libffi_arm_wince
 - libffi_msvc
 - libffi_osx
msg181958 - (view) Author: Roundup Robot (python-dev) Date: 2013-02-12 14:33
New changeset 7727be7613f9 by doko in branch 'default':
- Issue #17192: Import libffi-3.0.12.
http://hg.python.org/cpython/rev/7727be7613f9
msg181960 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-02-12 14:43
libffi_osx is not a copy of the regular libffi, but a (fairly old) fork. I don't know how far the two "branches" have diverged.  

An important feature  of liffi_osx is that is compiles cleanly when all intel and ppc related sources are compiled with '-arch ppc -arch i386 -arch x86_64' (that is, the relevant sources contain preprocessor guards to ensure the code in those files is only compiled for the right archictures). This greatly simplifies the build process because it is not necessary to build libffi a couple of times and then merge the resulting binaries.

I'm -0 at this point w.r.t. trying to remove libffi_osx, that tree works and I'll work on it anyway (it is also used in PyObjC).
msg182047 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2013-02-13 16:41
Maybe Modules/_ctypes/libffi in 2.7, 3.2 and 3.3 branches should be updated?
msg182066 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2013-02-13 23:20
Sound like a large change for stable releases.  Usually we would not do that.
msg182205 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2013-02-16 01:00
However internal copy of expat was updated in 2.7 and 3.2 branches about 2 weeks ago (issue #14340).
msg182440 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-02-19 22:05
An update to libffi is needed for all maintained versions of Python.

In 2.7, we're running into the stack being misaligned in 32-bit x86 code which is something a libffi update fixes.  It is a simple patch:

 http://patchwork.ozlabs.org/patch/58128/

which comes to the official libffi releases via https://github.com/atgreen/libffi/commit/3f5b1375ab1e2b8e3d593e21b27097a4a50f9b83#src/x86/sysv.S.

The problem: without the stack being 16-byte aligned, code generated by modern compilers like recent gcc/g++ or clang assumed that the stack is 16 byte aligned and uses SSE instructions in some circumstances that require this.  Without this fix, any ctypes call into such code will crash.

Sure, that tiny patch could be cherry picked into our libffi, but we IMNSHO may as well just update the entire thing given that we embed a very old copy rather than use it as an external dependency.
msg182443 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-02-19 22:11
http://bugs.python.org/issue17245 filed to track the stack alignment issue.  The only reason i set this as release blocker is to let a release manager decide which of these two issues to proceed with for 2.7.4 and 3.2.4 (and 3.3.1).
msg182444 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-02-19 22:12
By all means, upgrade it.
msg182448 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2013-02-19 22:29
before the updates, ... there seem to be two test failures on sparc solaris. the local libffi/src/sparc/v8.S change was integrated upstream, so I don't yet what could cause these failures. or did they fail before too?
msg182870 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-02-24 11:50
Updating 3.x branches is fine with me.
msg184560 - (view) Author: Roundup Robot (python-dev) Date: 2013-03-18 23:59
New changeset b10ec5083a53 by Gregory P. Smith in branch '2.7':
Fixes issue #17192: Update the ctypes module's libffi to v3.0.13.  This
http://hg.python.org/cpython/rev/b10ec5083a53
msg184693 - (view) Author: Roundup Robot (python-dev) Date: 2013-03-19 22:33
New changeset 43e8da639462 by Gregory P. Smith in branch '3.2':
Fixes issue #17192: Update the ctypes module's libffi to v3.0.13.  This
http://hg.python.org/cpython/rev/43e8da639462

New changeset f23274357fc8 by Gregory P. Smith in branch '3.3':
Fixes Issue #17192: Update the ctypes module's libffi to v3.0.13.  This
http://hg.python.org/cpython/rev/f23274357fc8

New changeset a94b3b4599f1 by Gregory P. Smith in branch 'default':
Fixes Issue #17192: Update the ctypes module's libffi to v3.0.13.  This
http://hg.python.org/cpython/rev/a94b3b4599f1
msg184703 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-03-19 23:13
2.7, 3.2, 3.3 and 3.4 have all be updated to ctypes v3.0.13.

fwiw, I tried a build of 3.3 on Sparc Solaris 10 using snakebite (nitrogen) and ctypes builds and links but two tests fail: ctypes.test.test_bitfields.C_Test test_ints and test_shorts failed.

That should be dealt with via a separate issue and by someone who cares about Solaris.
msg187427 - (view) Author: koobs (koobs) Date: 2013-04-20 14:06
These break what was addressed in #11729 for default, 3,x and 3.3.

2.7 seems to have made it through unscathed.

I'm not sure where or how the old code was introduced, but the clang fix has been upstreamed and is correct in the pure libffi 3.0.13 sources

Failure to build ctypes can be seen here:

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.x/builds/1246/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.x/builds/1245/steps/test/logs/stdio
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/538/steps/test/logs/stdio
msg188017 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-04-28 19:20
Greg?
msg188129 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-04-30 06:37
it _looks_ like the libffi updates i put in 3.2, 3.3 and default (3.4) are messed up with some changes missing in 3.2 and more missing in 3.3 and 3.4.  (i'm comparing vs upstream 3.0.13 <-> 2.7 <-> 3.2 <-> 3.3 <-> 3.4).

i blame the mess of merging that was required by hg due to the plethora of versions of libffi we had in different states in 2.7, 3.2 and 3.3+ before i did the update.  i'm fixing it now.
msg188131 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-04-30 06:59
Thanks. For 3.2, please attach a patch here.
msg188135 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2013-04-30 07:20
I've got this and all of the fixes necessary for 3.3, default and 2.7 in my local repositories.

odd thing in the 3.2 patch: it is entirely unclear to me if the issue 10309 fix needs to be kept.  i included it as it was missing from 3.2 after previous commits (3.3 and 3.4 still kept it).  2.7 needs it as well.  the other part of the patch is the issue 11729 fix that already exists upstream being reapplied; it was inadvertently undone during the upgrade.

I _could_ push fixes for all branches at once right now unless pushes to 3.2 are blocked and prevent me from pushing anything.

(rant: at which point i declare "hg hell" and feel like giving up because it simply has gotten in the way throughout this entire process rather than helping... i'll be forced to manually create diffs and reapply them to _other_ hg checkouts?  not so useful.  surely someone will stupidly claim i'm holding hg wrong instead of the tool being unhelpful).

We need to GET RID of libffi being bundled as part of the source release in 3.4.  this is disgusting.
msg188136 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-04-30 07:31
OK, 3.2 block disabled, push away.
msg188138 - (view) Author: koobs (koobs) Date: 2013-04-30 08:03
Ready to confirm on buildbots when this lands

Also, libffi-3.0.13 is installed on my buildbots if you want to use -custom to have a play, I can be pinged anytime on IRC if I can be of any assistance. koobs @ #python-dev
msg188139 - (view) Author: Roundup Robot (python-dev) Date: 2013-04-30 08:05
New changeset 7a2e0c1bb1a9 by Gregory P. Smith in branch '3.2':
* Fix issue 17192 for 3.2 - reapply the issue11729 patch that was undone
http://hg.python.org/cpython/rev/7a2e0c1bb1a9

New changeset aa3371fa9773 by Gregory P. Smith in branch '3.3':
* Fix PART of issue 17192 for 3.3 - reapply the issue11729 patch
http://hg.python.org/cpython/rev/aa3371fa9773

New changeset e0fdc21c7d13 by Gregory P. Smith in branch 'default':
* Fix PART of issue 17192 for 3.4 - reapply the issue11729 patch
http://hg.python.org/cpython/rev/e0fdc21c7d13

New changeset c665abe15c4a by Gregory P. Smith in branch '3.3':
* Fix remaining bits of issue 17192 for 3.3 - these changes
http://hg.python.org/cpython/rev/c665abe15c4a

New changeset 688bc0b44d96 by Gregory P. Smith in branch 'default':
* Fix remaining bits of issue 17192 for 3.4 - these changes
http://hg.python.org/cpython/rev/688bc0b44d96

New changeset 663800e18fef by Gregory P. Smith in branch '3.2':
This local change was lost during the fixing of issue17192 to update
http://hg.python.org/cpython/rev/663800e18fef

New changeset 7438d202e380 by Gregory P. Smith in branch '2.7':
This local change was lost during the fixing of issue17192 to update
http://hg.python.org/cpython/rev/7438d202e380
History
Date User Action Args
2013-04-30 08:32:24gregory.p.smithsetstatus: open -> closed
resolution: fixed
2013-04-30 08:05:37python-devsetmessages: + msg188139
2013-04-30 08:03:32koobssetmessages: + msg188138
2013-04-30 07:31:55georg.brandlsetmessages: + msg188136
2013-04-30 07:20:14gregory.p.smithsetfiles: + issue17192-refix-3.2-gps01.diff
keywords: + patch
messages: + msg188135
2013-04-30 06:59:06georg.brandlsetmessages: + msg188131
2013-04-30 06:37:41gregory.p.smithsetresolution: fixed -> (no value)
messages: + msg188129
2013-04-28 19:20:44georg.brandlsetmessages: + msg188017
2013-04-20 14:33:04r.david.murraysetstatus: closed -> open
2013-04-20 14:06:35koobssetnosy: + koobs
messages: + msg187427
2013-03-19 23:13:59gregory.p.smithsetstatus: open -> closed
type: behavior
messages: + msg184703

resolution: fixed
stage: resolved
2013-03-19 22:33:47python-devsetmessages: + msg184693
2013-03-19 00:12:57Arfreversettitle: libffi-3.0.12 import -> libffi-3.0.13 import
2013-03-18 23:59:15python-devsetmessages: + msg184560
2013-03-18 20:15:56gregory.p.smithsetassignee: gregory.p.smith
2013-02-24 11:50:04georg.brandlsetmessages: + msg182870
2013-02-19 22:29:51dokosetmessages: + msg182448
2013-02-19 22:12:40benjamin.petersonsetmessages: + msg182444
2013-02-19 22:11:59gregory.p.smithsetmessages: + msg182443
2013-02-19 22:05:17gregory.p.smithsetpriority: normal -> release blocker
versions: + Python 2.7, Python 3.2, Python 3.3
nosy: + larry, benjamin.peterson, gregory.p.smith, georg.brandl

messages: + msg182440
2013-02-16 01:00:12Arfreversetmessages: + msg182205
2013-02-13 23:20:12eric.araujosetnosy: + eric.araujo
messages: + msg182066
2013-02-13 16:41:40Arfreversetnosy: + Arfrever
messages: + msg182047
2013-02-12 14:43:30ronaldoussorensetmessages: + msg181960
2013-02-12 14:35:20dokosetnosy: + ronaldoussoren
2013-02-12 14:33:33python-devsetnosy: + python-dev
messages: + msg181958
2013-02-12 13:52:52dokocreate