This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Disallow untagged C extension import on major platforms
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Decorater, Marcus.Smith, barry, brett.cannon, christian.heimes, dstufft, eric.snow, gregory.p.smith, josh.r, ncoghlan, ned.deily, njs, paul.moore, pitrou, ronaldoussoren, scoder, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2017-12-20 15:47 by pitrou, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
changes.diff Decorater, 2017-12-20 21:43
Pull Requests
URL Status Linked Edit
PR 4943 open pitrou, 2017-12-20 15:50
Messages (20)
msg308741 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-20 15:47
Windows and the major POSIX platforms (including Linux and macOS) now support loading C extension with a version-tagged name (and build them likewise), such as "foobar.cpython-36.so".

Following the discussion in https://mail.python.org/pipermail/python-dev/2017-December/151328.html, I propose to remove the ability to load untagged C extension names on those platforms.  IOW, "foobar.dll" or "foobar.so" will not be recognized when "import foobar" is executed anymore.
msg308742 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-20 15:50
Interestingly, the test suite doesn't seem affected at all.
msg308748 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-20 16:43
Actually, let me correct this: the build fails on Windows (see AppVeyor).  That's because the Windows build process, for some reason, produces untagged pyd files.
msg308749 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-20 16:45
If I'm understanding correctly, the ".pyd" extension is hardcoded in the various ".vcxproj" files.  Which means this PR would need the help of a Windows expert to progress forward :-S
msg308789 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-12-20 19:35
+1 on doing this.

It might also be worth examining all of the latest package version wheels on pypi to see which packages claiming to support Python 3 have binary wheels containing untagged .pyd or .so files.
msg308791 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-12-20 19:37
Also, even if they are common, it seems reasonable that pip could be trained to rename untagged binaries on install after doing something to determine if they should be compatible with >=3.7 or not.
msg308792 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-20 19:38
cc'ing people for the pip/PyPI aspects of this.
msg308822 - (view) Author: Decorater (Decorater) * Date: 2017-12-20 21:21
If needed I could help upload the updated project files here for this issue.
msg308829 - (view) Author: Decorater (Decorater) * Date: 2017-12-20 21:43
Attached is my changes to the vcxproj files based on an field already in python.props.
msg308878 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2017-12-21 16:41
Please don't rush this in, at least for Windows. I deliberately decided to keep the included pyd files untagged for a reason, and need to recall what that was. (It might be in the issue when we added the tags.)

Considering a .pyd file linking against pythonXY.dll is only going to load in the context of that version anyway, this is hardly urgent. There is also no supported tag on Windows for building against the stable ABI, so that would have to be added and enabled in the various build tools.
msg308879 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-12-21 16:42
Thanks for the heads-up Steve.

> Considering a .pyd file linking against pythonXY.dll is only going to load in the context of that version anyway, this is hardly urgent.

I see.  That's a good argument for keeping the current extensions list, then.  We can narrow the change to POSIX platforms.
msg308899 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-12-21 20:01
With my macOS hat on, I don't see a problem with doing this.  C extensions built via Distutils have been version-tagged on macOS since 3.5.0 (for some reason, macOS was skipped when PEP 3149 was originally implemented).

With my release manager hat on, it sounds like a good idea.  But it could introduce a compatibility problem for anyone who doesn't use Distutils to produce extension modules.  So I'd like to see this proposal get a little more visibility, at the minimum, bringing it up on the Distutils SIG mailing list.  We should also ensure that the change gets mentioned in the 3.7 What's New document.  And somewhere in the docset there should be some documentation for the .so file name requirements.  AFAICT, today it's really not mentioned anywhere in the docs other than the reference to PEP 3149 in the 3.2 What's New.  Until the packaging documents for extension modules get overhauled perhaps something could be added to the Building chapter of the Extending and Embedding document:
https://docs.python.org/3/extending/building.html#building-c-and-c-extensions
msg311255 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-01-30 08:55
did this get discussed on distutils-sig?  It's a trivial change (for the POSIX only PR) and would be neat to go through one of the 3.7 beta cycles to see if anything serious an unexpected is likely to come up for anyone.
msg311256 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-30 08:56
+1
msg311309 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2018-01-31 01:38
So, just for clarity, is there another approved mechanism for distributing an extension that just happens to act as a (Python version specific) C support library? I've wanted to be able to build a library (static or dynamic) that is installed to lib/pythonX.Y/site-packages so it can be found by other extensions that use it without them having to ship a copy themselves.

The best solution I found was building and installing untagged extensions with pip; the extension features weren't actually used for anything but unittesting the library, but making it an extension meant it got installed to the Python version specific site-packages.

If untagged extensions aren't just old-fashioned, but banned, my use case is broken, and I've been completely unable to find a decent, non-Cython based, setup.py based pattern for building a C library that provides wrappers for Python API calls (and therefore must target specific Python versions).
msg311314 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2018-01-31 06:49
Interesting use case Josh.  I don't know of a good way to deal with non-extension-module Python API using .so files.  I like your extension hack, but what is the problem with having the version embedded in the .so name?
msg314328 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-03-23 21:08
Not sure if this has missed the 3.7 train, but bumping this to check if the discussion on distutils-sig ever occurred and what the  result of it was. (Otherwise the PR just needs a news entry and it should be ready.)
msg314333 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-03-23 21:40
As far as I know, it has not been discussed elsewhere and I think the potential for breaking third-party distributions is too high to risk making this change this late in the release cycle, independent of Steve's concerns in msg308878.  Unless someone can dispel those concerns prior to the imminent 3.7.0b3 ABI freeze, I think this should be deferred to 3.8.
msg336956 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-03-01 20:45
It's 3.8 time now. :) Should we try and resolve this?
msg336977 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-03-02 00:45
I think it'd be worth doing on POSIX systems for some 3.8 alpha/beta release cycles before making a final call there.
History
Date User Action Args
2022-04-11 14:58:55adminsetgithub: 76568
2019-03-02 00:45:01gregory.p.smithsetmessages: + msg336977
2019-03-01 20:45:18brett.cannonsetmessages: + msg336956
2018-05-02 21:14:47barrysetnosy: + barry
2018-03-23 21:40:50ned.deilysetmessages: + msg314333
versions: - Python 3.7
2018-03-23 21:08:39brett.cannonsetmessages: + msg314328
versions: + Python 3.8
2018-02-19 11:16:30scodersetnosy: + scoder
2018-01-31 06:49:00gregory.p.smithsetmessages: + msg311314
2018-01-31 01:38:19josh.rsetnosy: + josh.r
messages: + msg311309
2018-01-30 08:56:40christian.heimessetnosy: + christian.heimes
messages: + msg311256
2018-01-30 08:55:22gregory.p.smithsetmessages: + msg311255
2017-12-21 20:01:36ned.deilysetmessages: + msg308899
2017-12-21 16:42:40pitrousetmessages: + msg308879
2017-12-21 16:41:00steve.dowersetmessages: + msg308878
2017-12-20 21:43:25Decoratersetfiles: + changes.diff

messages: + msg308829
2017-12-20 21:21:32Decoratersetnosy: + Decorater
messages: + msg308822
2017-12-20 19:38:03pitrousetnosy: + dstufft, Marcus.Smith
messages: + msg308792
2017-12-20 19:37:15gregory.p.smithsetmessages: + msg308791
2017-12-20 19:35:23gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg308789
2017-12-20 16:45:07pitrousetmessages: + msg308749
2017-12-20 16:43:15pitrousetmessages: + msg308748
2017-12-20 15:50:25pitrousetmessages: + msg308742
2017-12-20 15:50:04pitrousetkeywords: + patch
stage: patch review
pull_requests: + pull_request4835
2017-12-20 15:49:45pitrousetnosy: + njs
2017-12-20 15:47:54pitroucreate