With an additonal change on SOABI (I will open a separated issue for that), PR 12946 allows to load lxml built in release mode in a Python built in debug mode! That's *very* useful for debugging: see my gdb example below.

---

I just modified the ABI of debug build so release and debug build now have the same ABI: bpo-36465.

I wrote a patch to use the same sys.implementation.cache_tag (SOABI) in release and debug mode:

diff --git a/configure b/configure
index b02d17c053..38eb7f1bd6 100755
--- a/configure
+++ b/configure
@@ -6325,7 +6325,6 @@ $as_echo "#define Py_DEBUG 1" >>confdefs.h
   { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
 $as_echo "yes" >&6; };
   Py_DEBUG='true'
-  ABIFLAGS="${ABIFLAGS}d"
 else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
 $as_echo "no" >&6; }; Py_DEBUG='false'
 fi
diff --git a/configure.ac b/configure.ac
index 65d3f8e691..1b2cd3076c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1223,7 +1223,6 @@ then
   [Define if you want to build an interpreter with many run-time checks.])
   AC_MSG_RESULT(yes);
   Py_DEBUG='true'
-  ABIFLAGS="${ABIFLAGS}d"
 else AC_MSG_RESULT(no); Py_DEBUG='false'
 fi],
 [AC_MSG_RESULT(no)])


(That's a temporary patch, I will design a better solution later.)

Using this patch + PR 12946, it becomes possible to load a C extension compiled in release mode in a debug Python!


---

Example building lxml in release mode and then load it in debug mode.


Install Python in *release* mode into /opt/py38release (shared libpython):

git clean -fdx; ./configure --enable-shared --prefix /opt/py38release && make && make install


Install Python in *debug* mode into /opt/py38debug (shared libpython):

git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix /opt/py38debug --with-pydebug && make && make install


Build lxml in release mode:

LD_LIBRARY_PATH=/opt/py38release/lib/ /opt/py38release/bin/python3.8 -m pip install lxml


By default, the debug Python doesn't have lxml:

$ LD_LIBRARY_PATH=/opt/py38debug/lib/ /opt/py38debug/bin/python3.8  -c 'import lxml'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'lxml'


Give access to C extensions compiled in release mode to the debug Python:

$ LD_LIBRARY_PATH=/opt/py38debug/lib/ PYTHONPATH=/opt/py38release/lib/python3.8/site-packages /opt/py38debug/bin/python3.8 
Python 3.8.0a3+ (heads/omit_libpython-dirty:8a03782387, Apr 25 2019, 02:52:01) 
>>> import lxml
>>> import lxml.etree
>>> lxml.etree
<module 'lxml.etree' from '/opt/py38release/lib/python3.8/site-packages/lxml/etree.cpython-38-x86_64-linux-gnu.so'>
>>> import sys
>>> sys.gettotalrefcount()
108304

It works!

Explanation:

* This is a debug build: sys.gettotalrefcount() is present (and works)
* lxml has been compiled in release mode
* etree.cpython-38-x86_64-linux-gnu.so is not linked to libpython and so doesn't rely on the *release* /opt/py38release/lib/libpython3.8.so
* this lxml can be loaded in a Python compiled in debug mode!

---

Now let's have a look at the gdb experience of release vs debug Python build.

I put a breakpoint on lxml.etree.iterparse('example.xml'): the C function is called __pyx_tp_new_4lxml_5etree_iterparse.


Using the release build, gdb fails to read many C local variables:

$ LD_LIBRARY_PATH=/opt/py38release/lib/ gdb -args /opt/py38release/bin/python3.8 parse.py 

(gdb) source /home/vstinner/prog/python/master/python-gdb.py 
(gdb) b __pyx_tp_new_4lxml_5etree_iterparse
Function "__pyx_tp_new_4lxml_5etree_iterparse" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
(gdb) run

Breakpoint 1, __pyx_tp_new_4lxml_5etree_iterparse (t=0x7fffea724900 <__pyx_type_4lxml_5etree_iterparse>, a=('example.xml',), k=0x0) at src/lxml/etree.c:218930
218930	src/lxml/etree.c: No such file or directory.

(gdb) py-bt
Traceback (most recent call first):
  File "parse.py", line 4, in func
    context = etree.iterparse('example.xml')
  File "parse.py", line 12, in <module>
    func("arg")

(gdb) frame 4
#4  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3268
3268	                res = call_function(&sp, oparg, NULL);
(gdb) p f
$1 = <optimized out>
(gdb) p co
$2 = <optimized out>
(gdb) p sp
$3 = <optimized out>
(gdb) p oparg
$4 = <optimized out>

The basic function "py-bt" works as expected, but inspecting Python internals doesn't work: most local C variables are "optimized out" :-(



New attempt using a debug build:

$ LD_LIBRARY_PATH=/opt/py38debug/lib/ PYTHONPATH=/opt/py38release/lib/python3.8/site-packages gdb -args /opt/py38debug/bin/python3.8 parse.py 

... same commands to load python-gdb.py and put a breakpoint ...

Breakpoint 1, __pyx_tp_new_4lxml_5etree_iterparse (t=0x7fffea606900 <__pyx_type_4lxml_5etree_iterparse>, a=('example.xml',), k=0x0) at src/lxml/etree.c:218930
218930	src/lxml/etree.c: No such file or directory.

(gdb) py-bt
Traceback (most recent call first):
  File "parse.py", line 4, in func
    context = etree.iterparse('example.xml')
  File "parse.py", line 12, in <module>
    func("arg")

(gdb) frame 4
#4  0x00007ffff7d6e4f8 in _PyEval_EvalFrameDefault (f=Frame 0x458790, for file parse.py, line 4, in func (arg='arg'), throwflag=0) at Python/ceval.c:3268
3268	                res = call_function(&sp, oparg, NULL);
(gdb) p f
$1 = Frame 0x458790, for file parse.py, line 4, in func (arg='arg')
(gdb) p co
$2 = (PyCodeObject *) 0x7fffea682388
(gdb) p co->co_consts
$4 = (None, 'example.xml', 'None', ' => ')
(gdb) p sp
$5 = (PyObject **) 0x458938
(gdb) p sp[0]
$6 = <module at remote 0x7fffea63a2f0>
(gdb) p sp[-1]
$7 = 'example.xml'
(gdb) p oparg
$8 = 1

The debugging experience is *much* better: it's possible to inspect Python internals!