setup.py build for a library using py_limited_api will always generate a stable ABI tagged shared library, even under the pydebug interpreter.

This means that extensions that are built for a pydebug interpreter may be accidentally (and brokenly) imported in a non-dbg interpreter and vice-versa.

e.g. in python-librtmp, with cffi 1.8.3:
$ python3-dbg setup.py build
...
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -Og -fdebug-prefix-map=/build/python3.5-H9Fri6/python3.5-3.5.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5-pydebug/build/temp.linux-x86_64-3.5-pydebug/librtmp._librtmp.o -lrtmp -o build/lib.linux-x86_64-3.5-pydebug/librtmp/_librtmp.abi3.so

Then:
$ cd build/lib.linux-x86_64-3.5-pydebug
$ python3 -c 'import librtmp'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/python-librtmp-0.3.0/build/lib.linux-x86_64-3.5-pydebug/librtmp/__init__.py", line 8, in <module>
    from ._librtmp import ffi, lib as librtmp
ImportError: /tmp/python-librtmp-0.3.0/build/lib.linux-x86_64-3.5-pydebug/librtmp/_librtmp.abi3.so: undefined symbol: _Py_RefTotal

setuptools decides whether to use the stable ABI, by looking at imp.get_suffixes(). And obviously, the importer is looking at that too. So, the stable ABI tag should simply not be in there.

PEP-3149 agrees with this. It has this quote from Martin v. Löwis:
--with-pydebug would not be supported by the stable ABI because this changes the layout of PyObject , which is an exposed structure.

So, here's a patch, to disable support for the stable ABI under pydebug builds.

I'm not sure that you really want this, because it would make it impossible to build an extension for the stable ABI for a debug build. The problem is Debian specific, because we install the extension modules for normal and debug builds in the same location. A Debian solution would be to use a different soname for stable API debug mode extensions.

I wouldn't say it's *entirely* Debian-specific. It just bites anyone who actually needs these tags to differentiate between built extensions. (Mostly Debian)

Yes, changing the tag is a more complete solution. It just seemed that that option was decided against, in the relevant PEPs.

[Matthias Klose (doko) 2016-10-27 15:45]

I'm not sure that you really want this, because it would make it impossible to build an extension for the stable ABI for a debug build.

It looks like that is already impossible:

/usr/include/python3.5dm/object.h:65:2: error: #error Py_LIMITED_API is incompatible with Py_DEBUG, Py_TRACE_REFS, and Py_REF_DEBUG
#error Py_LIMITED_API is incompatible with Py_DEBUG, Py_TRACE_REFS, and Py_REF_DEBUG
^~~~~

So in my opinion Stefano's patch makes sense.

So limited ABI modules can't be imported by a Py_DEBUG build. Stefano's patch just skips over them. That seems reasonable, but one question I do have is whether this would confuse users since they will simply get an ImportError instead of some other error and thus will they be able to realize why there's a problem?

whether this would confuse users since they will simply get an ImportError instead of some other error and thus will they be able to realize why there's a problem?

It's the same behaviour we have for any other module on the import path, that doesn't have the right tag. I suppose abi3 is a bit of a special case there, because it's expected to be widely supported...

Also, there is a related problem: Because abi3 is in the supported extension list, setup.py build will build an abi3 extension (which isn't actually abi3), under a pydebug interpreter.

New changeset 338d54f by Miss Islington (bot) (Stefano Rivera) in branch 'master':
bpo-28401: prevent Py_DEBUG builds from trying to import limited ABI modules (GH-1766)
338d54f

Thanks for the PR!