classification
Title: Don't support the PEP384 stable ABI in pydebug builds
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6, Python 3.3, Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, doko, eric.snow, miss-islington, mitya57, ncoghlan, stefanor
Priority: normal Keywords: patch

Created on 2016-10-09 17:31 by stefanor, last changed 2018-11-21 22:16 by brett.cannon. This issue is now closed.

Files
File name Uploaded Description Edit
pep384-pydbg.patch stefanor, 2016-10-09 17:31 Patch version 1 review
Pull Requests
URL Status Linked Edit
PR 1766 merged python-dev, 2017-05-23 18:13
Messages (8)
msg278381 - (view) Author: Stefano Rivera (stefanor) * Date: 2016-10-09 17:31
setup.py build for a library using py_limited_api will always generate a stable ABI tagged shared library, even under the pydebug interpreter.

This means that extensions that are built for a pydebug interpreter may be accidentally (and brokenly) imported in a non-dbg interpreter and vice-versa.

e.g. in python-librtmp, with cffi 1.8.3:
$ python3-dbg setup.py build
...
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -Og -fdebug-prefix-map=/build/python3.5-H9Fri6/python3.5-3.5.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5-pydebug/build/temp.linux-x86_64-3.5-pydebug/librtmp._librtmp.o -lrtmp -o build/lib.linux-x86_64-3.5-pydebug/librtmp/_librtmp.abi3.so

Then:
$ cd build/lib.linux-x86_64-3.5-pydebug
$ python3 -c 'import librtmp'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/python-librtmp-0.3.0/build/lib.linux-x86_64-3.5-pydebug/librtmp/__init__.py", line 8, in <module>
    from ._librtmp import ffi, lib as librtmp
ImportError: /tmp/python-librtmp-0.3.0/build/lib.linux-x86_64-3.5-pydebug/librtmp/_librtmp.abi3.so: undefined symbol: _Py_RefTotal

setuptools decides whether to use the stable ABI, by looking at imp.get_suffixes(). And obviously, the importer is looking at that too. So, the stable ABI tag should simply not be in there.

PEP3149 agrees with this. It has this quote from Martin v. Löwis:
 --with-pydebug would not be supported by the stable ABI because this changes the layout of PyObject , which is an exposed structure.

So, here's a patch, to disable support for the stable ABI under pydebug builds.
msg279540 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-10-27 15:45
I'm not sure that you really want this, because it would make it impossible to build an extension for the stable ABI for a debug build.  The problem is Debian specific, because we install the extension modules for normal and debug builds in the same location.  A Debian solution would be to use a different soname for stable API debug mode extensions.
msg279542 - (view) Author: Stefano Rivera (stefanor) * Date: 2016-10-27 15:48
I wouldn't say it's *entirely* Debian-specific. It just bites anyone who actually needs these tags to differentiate between built extensions. (Mostly Debian)

Yes, changing the tag is a more complete solution. It just seemed that that option was decided against, in the relevant PEPs.
msg284853 - (view) Author: Dmitry Shachnev (mitya57) * Date: 2017-01-06 21:27
[Matthias Klose (doko) 2016-10-27 15:45]
> I'm not sure that you really want this, because it would make it impossible to build an extension for the stable ABI for a debug build.

It looks like that is already impossible:

/usr/include/python3.5dm/object.h:65:2: error: #error Py_LIMITED_API is incompatible with Py_DEBUG, Py_TRACE_REFS, and Py_REF_DEBUG
 #error Py_LIMITED_API is incompatible with Py_DEBUG, Py_TRACE_REFS, and Py_REF_DEBUG
  ^~~~~

So in my opinion Stefano's patch makes sense.
msg294273 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-05-23 18:27
So limited ABI modules can't be imported by a Py_DEBUG build. Stefano's patch just skips over them. That seems reasonable, but one question I do have is whether this would confuse users since they will simply get an ImportError instead of some other error and thus will they be able to realize why there's a problem?
msg294276 - (view) Author: Stefano Rivera (stefanor) * Date: 2017-05-23 18:59
> whether this would confuse users since they will simply get an ImportError instead of some other error and thus will they be able to realize why there's a problem?

It's the same behaviour we have for any other module on the import path, that doesn't have the right tag. I suppose abi3 is a bit of a special case there, because it's expected to be widely supported...

Also, there is a related problem: Because abi3 is in the supported extension list, setup.py build will build an abi3 extension (which isn't actually abi3), under a pydebug interpreter.
msg330014 - (view) Author: miss-islington (miss-islington) Date: 2018-11-16 23:52
New changeset 338d54f0a59dc5e5b6c9e7397340169f3a3f8ea4 by Miss Islington (bot) (Stefano Rivera) in branch 'master':
bpo-28401: prevent Py_DEBUG builds from trying to import limited ABI modules (GH-1766)
https://github.com/python/cpython/commit/338d54f0a59dc5e5b6c9e7397340169f3a3f8ea4
msg330221 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-11-21 22:16
Thanks for the PR!
History
Date User Action Args
2018-11-21 22:16:48brett.cannonsetmessages: + msg330221
2018-11-19 20:41:18brett.cannonsetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2018-11-16 23:52:58miss-islingtonsetnosy: + miss-islington
messages: + msg330014
2017-05-23 18:59:36stefanorsetmessages: + msg294276
2017-05-23 18:27:23brett.cannonsetnosy: + brett.cannon, ncoghlan, eric.snow
messages: + msg294273

components: + Interpreter Core, - Library (Lib)
stage: commit review
2017-05-23 18:13:32python-devsetpull_requests: + pull_request1847
2017-01-06 21:27:54mitya57setmessages: + msg284853
2017-01-01 08:27:56mitya57setnosy: + mitya57
2016-10-27 15:48:59stefanorsetmessages: + msg279542
2016-10-27 15:45:19dokosetnosy: + doko
messages: + msg279540
2016-10-09 17:31:32stefanorcreate