classification
Title: New matmul operator crashes modules compiled with CPython3.4
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arfrever, amaury.forgeotdarc, benjamin.peterson, docs@python, lemburg, loewis, pitrou
Priority: normal Keywords:

Created on 2014-12-09 12:31 by amaury.forgeotdarc, last changed 2015-07-05 10:41 by pitrou.

Files
File name Uploaded Description Edit
mytest.c amaury.forgeotdarc, 2014-12-09 12:42
Messages (10)
msg232372 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2014-12-09 12:31
When an extension module is compiled with CPython3.4, the nb_matrix_multiply slot is not filled, and no memory is allocated for it.
If the extension module is imported by CPython3.5, nb_matrix_multiply contains garbage and segfaults the interpreter.

In Python 2.7 there are flags like Py_TPFLAGS_HAVE_INDEX to gate the access  to recently added slots. I think Python3.5 should have a similar one.
msg232373 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2014-12-09 12:42
Here is a test module that segfaults on Python3.5.
It's derived from _testcapi, but it can be anything with a PyNumberMethods.

I compiled with
    gcc -g -I path/to/cpython3.4/Include/ -I /path/to/cpython3.4 mytest.c -fPIC --shared -o mytest.so

Then I ran python3.5 and:
    import mytest
    print(mytest.matmulType() @ 1)

When the module is compiled with 3.5, TypeError is correctly raised.
msg232381 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-12-09 15:00
This technically falls under the fact that we don't have ABI compatibility between 3.N and 3.(N + 1).
msg232395 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2014-12-09 17:37
Oh, has this ABI compatibility requirement changed from the 2.x series?
msg232396 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-12-09 17:39
We never had compatibility between 2.N and 2.(N + 1) either.
msg232398 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2014-12-09 18:15
I would not mind the change, but I've always thought the opposite (except on Windows, where the string Python27.dll is stored in the .pyd)

There are traces of this binary compatibility still today in the 3.4 headers:
https://hg.python.org/cpython/file/v3.4.0/Include/pymem.h#l23
"""...if you do use the macros you must recompile your extensions with each Python release..."""
Which seems to imply that extensions usually don't need to be recompiled!
msg232413 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2014-12-10 05:49
Extensions need not to be rebuilt only when using stable ABI:
https://www.python.org/dev/peps/pep-0384/

But mytest.c fails to build with -DPy_LIMITED_API so it uses features from outside of stable ABI and must be rebuilt.
msg232418 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2014-12-10 09:54
PEP384 is presented as a new way to write modules that can be loaded by multiple Python versions, specially on Windows.
I could not find any place that says that modules not using the stable ABI need to be recompiled.

I'm not against changing this rule (after all, I started on Windows platform where all this discussion does not apply), but it should be clearly stated somewhere in the API docs, and also in the "whatsnew" pages (all of them?)
msg232430 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2014-12-10 15:15
This is why C-extensions' PEP 3147 file tags depend on Python version.

I agree the policy could stand to be documented.
msg246305 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-07-05 10:41
There was probably an informal "best effort ABI compatibility" in 2.x that we de facto dropped in 3.x. Otherwise, as Amaury points out, several Py_TPFLAGS_HAVE_XXX flags would have no purpose.
History
Date User Action Args
2015-07-05 10:41:46pitrousetnosy: + pitrou
messages: + msg246305
2015-07-05 01:09:51larrysetpriority: release blocker -> normal
type: crash -> enhancement
stage: needs patch
2015-06-18 10:35:57taleinatsetassignee: docs@python

components: + Documentation
nosy: + docs@python
2014-12-10 15:15:59benjamin.petersonsetmessages: + msg232430
2014-12-10 09:54:29amaury.forgeotdarcsetmessages: + msg232418
2014-12-10 05:49:32Arfreversetnosy: + Arfrever
messages: + msg232413
2014-12-10 00:34:00pitrousetnosy: + lemburg, loewis
2014-12-09 18:15:34amaury.forgeotdarcsetmessages: + msg232398
2014-12-09 17:39:34benjamin.petersonsetmessages: + msg232396
2014-12-09 17:37:51amaury.forgeotdarcsetmessages: + msg232395
2014-12-09 15:00:30benjamin.petersonsetmessages: + msg232381
2014-12-09 12:42:13amaury.forgeotdarcsetfiles: + mytest.c

messages: + msg232373
2014-12-09 12:32:46pitrousetpriority: normal -> release blocker
versions: + Python 3.5
2014-12-09 12:31:43amaury.forgeotdarccreate