classification
Title: Make release and debug ABI compatible
Type: Stage: resolved
Components: Build Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: pitrou, vstinner
Priority: normal Keywords: patch

Created on 2019-03-28 23:01 by vstinner, last changed 2019-04-24 23:46 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12614 closed vstinner, 2019-03-28 23:02
PR 12615 merged vstinner, 2019-03-28 23:15
PR 12945 merged vstinner, 2019-04-24 23:28
Messages (7)
msg339079 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-28 23:01
When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI.

I propose to modify the debug build (Py_DEBUG) to not imply Py_TRACE_REFS anymore. Antoine Pitrou proposed this idea when the C API was discussed to get a stable ABI.

Another more radical idea is to completely remove Py_TRACE_REFS special build.
msg339080 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-28 23:07
> Another more radical idea is to completely remove Py_TRACE_REFS special build.

I wrote PR 12614 to implement this idea. I wrote it to see which code depends on it:

commit 63509498761a0e7f72585a8cd7df325ea2abd1b2 (HEAD -> remove_trace_refs, origin/remove_trace_refs)
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu Mar 28 23:26:58 2019 +0100

    WIP: bpo-36465: Remove Py_TRACE_REFS special build
    
    Remove _ob_prev and _ob_next fields of PyObject when Python is
    compiled in debug mode to make debug ABI closer to the release ABI.
    
    Remove:
    
    * sys.getobjects()
    * PYTHONDUMPREFS environment variable
    * _PyCoreConfig.dump_refs
    * PyObject._ob_prev and PyObject._ob_next fields
    * _PyObject_HEAD_EXTRA and _PyObject_EXTRA_INIT macros
    * _Py_AddToAllObjects()
    * _Py_PrintReferenceAddresses()
    * _Py_PrintReferences()
    * _Py_ForgetReference(op) is replaced with _Py_INC_TPFREES(op)

I never used PYTHONDUMPREFS. I just tried in Python 3.7: Python does crash with this option... so this option doesn't sound popuplar. Otherwise, many users would complain.

$ PYTHONDUMPREFS=1 python3.7-dbg -c pass
...
0x7f7eae14aa90 [1] 'Thread-local dummy'
0x7f7eae19b448 [1] (<class '_thread._localdummy'>, <class 'object'>)
0x7f7eae14aa30 [1] {'__doc__': 'Thread-local dummy'}
0x7f7eae1356d8 [1] (<class 'object'>,)
0x7d0940 [2] <class '_thread._localdummy'>
0x7f7eae120d58 [1] Segmentation fault (core dumped)

I never used sys.getobjects() neither, but I can imagine that someone might want to use for very specific needs. So maybe it's safer to not immediately remove the feature. At least, a deprecation period would be needed.

I suggest to reject PR 12614 and not remove Py_TRACE_REFS.
msg339081 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-03-28 23:17
PR 12615 changes Py_DEBUG to no longer imply Py_TRACE_REFS. IMHO it's a more reasonable approach.

I'm not sure if it's enough to magically get exactly the same ABI than Python built in release mode.
msg340746 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-23 23:09
Since my goal here is to support a C extension compiled in release mode on a Python compiled in debug mode, Py_TRACE_REFS should be a new separated ABI since it modify Py_INCREF/Py_DECREF. Maybe a new configure option should be added to opt-in for Py_TRACE_REFS and add "t" (T stands for Trace references) to SOABI.
msg340750 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 00:51
I started a thread on python-dev to discuss this issue:
"[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode"
https://mail.python.org/pipermail/python-dev/2019-April/157178.html

It's a follow-up of my previous thread:
"[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build"
https://mail.python.org/pipermail/python-dev/2019-April/157015.html
msg340812 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 22:56
New changeset f4e4703e746067d6630410408d414b11003334d6 by Victor Stinner in branch 'master':
bpo-36465: Make release and debug ABI compatible (GH-12615)
https://github.com/python/cpython/commit/f4e4703e746067d6630410408d414b11003334d6
msg340813 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-04-24 23:46
New changeset 235e7b2b0d937ba8b4a9172aa72206187e3e1f54 by Victor Stinner in branch 'master':
bpo-36465: Fix test_regrtest on Windows (GH-12945)
https://github.com/python/cpython/commit/235e7b2b0d937ba8b4a9172aa72206187e3e1f54
History
Date User Action Args
2019-04-24 23:46:34vstinnersetmessages: + msg340813
2019-04-24 23:28:25vstinnersetpull_requests: + pull_request12869
2019-04-24 22:58:10vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-04-24 22:56:34vstinnersetmessages: + msg340812
2019-04-24 00:52:03vstinnersettitle: No longer enable Py_TRACE_REFS by default in debug build -> Make release and debug ABI compatible
2019-04-24 00:51:29vstinnersetmessages: + msg340750
2019-04-23 23:09:32vstinnersetmessages: + msg340746
2019-03-28 23:17:20vstinnersetmessages: + msg339081
2019-03-28 23:15:56vstinnersetpull_requests: + pull_request12551
2019-03-28 23:07:30vstinnersetmessages: + msg339080
2019-03-28 23:02:28vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request12550
2019-03-28 23:01:42vstinnercreate