classification
Title: Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
Type: crash Stage: patch review
Components: Interpreter Core Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, christian.heimes, inada.naoki, jdemeyer, lukasz.langa, miss-islington, pablogsal, petr.viktorin, pitrou, vstinner
Priority: release blocker Keywords: 3.8regression, patch

Created on 2019-09-02 09:48 by christian.heimes, last changed 2019-09-10 14:32 by vstinner.

Files
File name Uploaded Description Edit
reproducer.tar.gz vstinner, 2019-09-03 08:45
gc_crash.py vstinner, 2019-09-03 14:20
gc_crash.patch vstinner, 2019-09-03 14:20
Pull Requests
URL Status Linked Edit
PR 15641 merged vstinner, 2019-09-02 11:37
PR 15787 merged miss-islington, 2019-09-09 14:56
PR 15789 merged vstinner, 2019-09-09 15:00
Messages (41)
msg350974 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-02 09:48
I have a case of a segfault on Fedora 32 (currently rawhide) with Python 3.8b4 and FreeIPA. The ipactl Python helper crashes with a segfault when the Python process exits. The segfault occurs in _PyFunction_Vectorcall() with a weakref related local scope function that has a NULL func_code pointer. _PyFunction_Vectorcall() does not check the code object for NULL and access the variable without check:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7be354f in _PyFunction_Vectorcall (func=<function at remote 0x7fffe6ebfd30>, stack=0x7fffffffdb50, nargsf=1, kwnames=0x0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/call.c:395
395         Py_ssize_t nkwargs = (kwnames == NULL) ? 0 : PyTuple_GET_SIZE(kwnames);
(gdb) bt
#0  0x00007ffff7be354f in _PyFunction_Vectorcall (func=<function at remote 0x7fffe6ebfd30>, stack=0x7fffffffdb50, nargsf=1, kwnames=0x0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/call.c:395
#1  0x00007ffff7bb16b0 in _PyObject_Vectorcall (kwnames=0x0, nargsf=1, args=0x7fffffffdb50, callable=<function at remote 0x7fffe6ebfd30>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/cpython/abstract.h:127
#2  _PyObject_FastCall (nargs=1, args=0x7fffffffdb50, func=<function at remote 0x7fffe6ebfd30>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/cpython/abstract.h:147
#3  object_vacall (base=<optimized out>, callable=<function at remote 0x7fffe6ebfd30>, vargs=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/call.c:1186
#4  0x00007ffff7bb19e1 in PyObject_CallFunctionObjArgs (callable=callable@entry=<function at remote 0x7fffe6ebfd30>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/call.c:1259
#5  0x00007ffff7c3f7c3 in handle_callback (ref=<optimized out>, callback=<function at remote 0x7fffe6ebfd30>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/weakrefobject.c:877
#6  0x00007ffff7c01bf7 in PyObject_ClearWeakRefs (object=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/weakrefobject.c:922
#7  0x00007fffe905babc in ctypedescr_dealloc () from /usr/lib64/python3.8/site-packages/_cffi_backend.cpython-38-x86_64-linux-gnu.so
#8  0x00007fffe9058825 in cfield_dealloc () from /usr/lib64/python3.8/site-packages/_cffi_backend.cpython-38-x86_64-linux-gnu.so
#9  0x00007ffff7ba4c65 in _Py_DECREF (filename=<synthetic pointer>, lineno=541, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:2000
#10 _Py_XDECREF (op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:541
#11 free_keys_object (keys=0x55555609f590) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:580
#12 dictkeys_decref (dk=0x55555609f590) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:324
#13 dict_dealloc (mp=0x7fffe6dd5340) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:1994
#14 0x00007fffe905bb25 in ctypedescr_dealloc () from /usr/lib64/python3.8/site-packages/_cffi_backend.cpython-38-x86_64-linux-gnu.so
#15 0x00007ffff7ba5058 in _Py_DECREF (filename=<optimized out>, lineno=541, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:478
#16 _Py_XDECREF (op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:541
#17 tupledealloc (op=0x7fffe6e3bbb0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/tupleobject.c:247
#18 0x00007ffff7ba5058 in _Py_DECREF (filename=<optimized out>, lineno=541, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:478
#19 _Py_XDECREF (op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:541
#20 tupledealloc (op=0x7fffe6e55b80) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/tupleobject.c:247
#21 0x00007ffff7c16b76 in _Py_DECREF (filename=0x7ffff7d2e6a8 "/builddir/build/BUILD/Python-3.8.0b4/Objects/typeobject.c", lineno=1110, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:478
#22 clear_slots (self=<optimized out>, type=0x55555559d0d0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/typeobject.c:1110
#23 subtype_dealloc (self=<KeyedRef at remote 0x7fffe6deb220>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/typeobject.c:1262
#24 0x00007ffff7ba4c65 in _Py_DECREF (filename=<synthetic pointer>, lineno=541, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:2000
#25 _Py_XDECREF (op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:541
#26 free_keys_object (keys=0x555556078d30) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:580
#27 dictkeys_decref (dk=0x555556078d30) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:324
#28 dict_dealloc (mp=0x7fffe6eba6c0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/dictobject.c:1994
#29 0x00007ffff7b98a4b in _Py_DECREF (filename=<synthetic pointer>, lineno=541, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:478
#30 _Py_XDECREF (op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:541
#31 cell_dealloc (op=0x7fffe6ebe2e0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/cellobject.c:84
#32 0x00007ffff7ba5058 in _Py_DECREF (filename=<optimized out>, lineno=541, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:478
#33 _Py_XDECREF (op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:541
#34 tupledealloc (op=0x7fffe6ebe310) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/tupleobject.c:247
#35 0x00007ffff7b991c2 in _Py_DECREF (filename=0x7ffff7d2e460 "/builddir/build/BUILD/Python-3.8.0b4/Objects/funcobject.c", lineno=584, op=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Include/object.h:478
#36 func_clear (op=0x7fffe6ebfd30) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/funcobject.c:584
#37 0x00007ffff7ba5f84 in delete_garbage (state=<optimized out>, state=<optimized out>, old=<optimized out>, collectable=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/gcmodule.c:929
#38 collect (generation=2, n_collected=0x0, n_uncollectable=0x0, nofail=1, state=0x7ffff7dd9598 <_PyRuntime+344>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/gcmodule.c:1106
#39 0x00007ffff7cac8c2 in _PyGC_CollectNoFail () at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/gcmodule.c:1848
#40 0x00007ffff7cacb74 in PyImport_Cleanup () at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/import.c:541
#41 0x00007ffff7cacf56 in Py_FinalizeEx () at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pylifecycle.c:1226
#42 0x00007ffff7cad088 in Py_Exit (sts=0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pylifecycle.c:2248
#43 0x00007ffff7cad0cf in handle_system_exit () at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pythonrun.c:658
#44 0x00007ffff7cad219 in _PyErr_PrintEx (set_sys_last_vars=1, tstate=0x55555555bfb0) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pythonrun.c:755
#45 PyErr_PrintEx (set_sys_last_vars=set_sys_last_vars@entry=1) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pythonrun.c:755
#46 0x00007ffff7cad23a in PyErr_Print () at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pythonrun.c:761
#47 0x00007ffff7b91b42 in PyRun_SimpleFileExFlags (fp=<optimized out>, filename=<optimized out>, closeit=<optimized out>, flags=0x7fffffffe178) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Python/pythonrun.c:434
#48 0x00007ffff7cae36f in pymain_run_file (cf=0x7fffffffe178, config=0x55555555b410) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/main.c:383
#49 pymain_run_python (exitcode=0x7fffffffe170) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/main.c:567
#50 Py_RunMain () at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/main.c:646
#51 0x00007ffff7cae559 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Modules/main.c:700
#52 0x00007ffff7e21193 in __libc_start_main () from /lib64/libc.so.6
#53 0x000055555555508e in _start ()

It looks like the func object is the local remove function of weakref.WeakValueDictionary() implementation, https://github.com/python/cpython/blob/353053d9ad08fea0e205e6c008b8a4350c0188e6/Lib/weakref.py#L90-L112

All function object fields except func_qualname and vectorcall are already NULL.

(gdb) print ((PyFunctionObject*)func).func_annotations 
$19 = 0x0
(gdb) print ((PyFunctionObject*)func).func_closure 
$20 = 0x0
(gdb) print ((PyFunctionObject*)func).func_code
$21 = 0x0
(gdb) print ((PyFunctionObject*)func).func_defaults 
$22 = 0x0
(gdb) print ((PyFunctionObject*)func).func_dict
$23 = 0x0
(gdb) print ((PyFunctionObject*)func).func_doc
$24 = 0x0
(gdb) print ((PyFunctionObject*)func).func_globals 
$25 = 0x0
(gdb) print ((PyFunctionObject*)func).func_kwdefaults 
$26 = 0x0
(gdb) print ((PyFunctionObject*)func).func_module 
$27 = 0x0
(gdb) print ((PyFunctionObject*)func).func_name
$28 = 0x0
(gdb) print ((PyFunctionObject*)func).func_qualname 
$29 = 'WeakValueDictionary.__init__.<locals>.remove'
(gdb) print ((PyFunctionObject*)func).func_weakreflist 
$30 = 0x0
(gdb) print ((PyFunctionObject*)func).vectorcall 
$31 = (vectorcallfunc) 0x7ffff7be3530 <_PyFunction_Vectorcall>
(gdb) print func.ob_refcnt 
$32 = 135
(gdb) print func.ob_type 
$33 = (struct _typeobject *) 0x7ffff7dd4e80 <PyFunction_Type>

The Fedora downstream bug is https://bugzilla.redhat.com/show_bug.cgi?id=1747901
msg350975 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-02 10:06
I don't understand how the function ended up with func_code=NULL. That shouldn't be a valid function to call, IMO.
Do you have any info on how the function ended up in that state?
msg350976 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-02 10:11
Not yet.

My current hypothesis is the function code object is already cleaned up *somehow*.
msg350981 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 10:57
> I don't understand how the function ended up with func_code=NULL. That shouldn't be a valid function to call, IMO. Do you have any info on how the function ended up in that state?

It doesn't seem possible to create a function with func_code=NULL, nor to set func_code to NULL. func_code can be be set to NULL by func_clear() which is called by func_dealloc().

I bet that func_clear() has been called since most func fields are set to NULL, which is consistent with:

static int
func_clear(PyFunctionObject *op)
{
    Py_CLEAR(op->func_code);
    Py_CLEAR(op->func_globals);
    Py_CLEAR(op->func_module);
    Py_CLEAR(op->func_name);
    Py_CLEAR(op->func_defaults);
    Py_CLEAR(op->func_kwdefaults);
    Py_CLEAR(op->func_doc);
    Py_CLEAR(op->func_dict);
    Py_CLEAR(op->func_closure);
    Py_CLEAR(op->func_annotations);
    Py_CLEAR(op->func_qualname);
    return 0;
}

The question is how is it possible that a deallocated function is still accessed? It smells like a borrowed reference somewhere in the call chain.
msg350983 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 11:08
Using gdb, I checked if func_clear() can be cleared outside func_dealloc(): yes, delete_garbage() (gcmodule.c) calls type->clear(). But I'm surprised that the function would be seen as "unreachable" if it's reference counter was equal to 135:

(gdb) print func.ob_refcnt 
$32 = 135
msg350985 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 11:27
(gdb) print ((PyFunctionObject*)func).func_qualname 
$29 = 'WeakValueDictionary.__init__.<locals>.remove'

This is a temporary remove() function declared in weakref.WeakValueDictionary constructor:

class WeakKeyDictionary(_collections_abc.MutableMapping):
    def __init__(self, dict=None):
        def remove(k, selfref=ref(self)):
            ...
        self._remove = remove
        ...

    def __setitem__(self, key, value):
        self.data[ref(key, self._remove)] = value

    ...

Simplified gdb traceback:

Py_FinalizeEx()
-> PyImport_Cleanup()
-> _PyGC_CollectNoFail()
-> delete_garbage()
-> func_clear(op=0x7fffe6ebfd30) at /usr/src/debug/python3-3.8.0~b4-1.fc32.x86_64/Objects/funcobject.c:584
    Py_CLEAR(op->func_closure);
-> tupledealloc()
-> cell_dealloc()
-> dict_dealloc()
-> dictkeys_decref()
-> cfield_dealloc () from _cffi_backend
-> PyObject_ClearWeakRefs()
-> _PyFunction_Vectorcall (func=<function at remote 0x7fffe6ebfd30>, ...)

PyImport_Cleanup() calls delete_garbage() which calls func_clear(op=0x7fffe6ebfd30).

But while the function closure is destroyed, a cffi object stored in a dict (namespace?) (stored in the function closure) is cleared, and there was a weak reference to this cffi object with a callback... which is the cleared function (Python object 0x7fffe6ebfd30)...

--

I don't understand why/how remove() gets a closure. I modified weakref.py to ensure that remove.__closure__ is None: test_weakref still pass.
msg350987 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 11:40
> I don't understand why/how remove() gets a closure.

Ok, I found the reason and I wrote PR 15641 to fix it.


Previous fix of WeakValueDictionary remove() function:

commit 9cd7e17640a49635d1c1f8c2989578a8fc2c1de6
Author: Łukasz Langa <lukasz@langa.pl>
Date:   Fri Feb 10 00:14:55 2017 -0800

    Fix #29519: weakref spewing exceptions during interp finalization
msg350990 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-02 12:09
Your workaround solves the segfault issue for me.
msg350991 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 12:34
I think the real problem is in vectorcall. tp_clear will make sure all internal pointers are either NULL or valid (by using Py_CLEAR, which also protects against re-entrancy) and it the responsibility of the object or its users to check that the fields that are needed are valid. From the docs on tp_clear:

> The Py_CLEAR() macro should be used, because clearing references is delicate: the reference to the contained object must not be decremented until after the pointer to the contained object is set to NULL. This is because decrementing the reference count may cause the contained object to become trash, triggering a chain of reclamation activity that may include invoking arbitrary Python code (due to finalizers, or weakref callbacks, associated with the contained object). If it’s possible for such code to reference self again, it’s important that the pointer to the contained object be NULL at that time, so that self knows the contained object can no longer be used. 

Notice the "so that self knows the contained object can no longer be used". I think _PyFunction_Vectorcall is missing the check "to see of the contained object can or cannot be used". Here the contained object being everything that has being cleaned before:

    Py_CLEAR(op->func_code);
    Py_CLEAR(op->func_globals);
    Py_CLEAR(op->func_module);
    Py_CLEAR(op->func_name);
    Py_CLEAR(op->func_defaults);
    Py_CLEAR(op->func_kwdefaults);
    Py_CLEAR(op->func_doc);
    Py_CLEAR(op->func_dict);
    Py_CLEAR(op->func_closure); <----- Everything before this
    Py_CLEAR(op->func_annotations);
    Py_CLEAR(op->func_qualname);

I think it will be enough to make a check for func_code being NULL in _PyFunction_Vectorcall or before.
msg350992 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 12:41
In particular, this was not happening before because the function type did not implement tp_clear:

https://github.com/python/cpython/blob/3.7/Objects/funcobject.c#L615

The new implementation of tp_clear without checks is allowing this to happen.
msg350995 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 12:46
See also https://bugs.python.org/issue33418 as the potential source of the problem.
msg350996 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-02 12:55
I'm not sure adding a check would solve this. What should be done when a function with func_code=NULL is called? "Silently do nothing" is not really an option; raising an exception wouldn't help much in this case.

I wonder if a function's tp_clear we should clear the mutable parts (like func_closure) before the immutable ones (func_code).
Any access to the immutable ones should have checks (and probably raise exceptions if those are NULL).
msg350997 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 13:07
I'm now able to reproduce the FreeIPA crash. In short:

* Get a Fedora Rawhide VM (to get Python 3.8 as "python3", it's more convenient)
* Install FreeIPA (dnf install freeipa-server)
* Run: ipa-server-install --help
* Python does crash at exit


> In particular, this was not happening before because the function type did not implement tp_clear:
> https://github.com/python/cpython/blob/3.7/Objects/funcobject.c#L615

I confirm that the patch below does fix the FreeIPA crash:

diff --git a/Objects/funcobject.c b/Objects/funcobject.c
index df5cc2d3f5..357372c565 100644
--- a/Objects/funcobject.c
+++ b/Objects/funcobject.c
@@ -669,7 +669,7 @@ PyTypeObject PyFunction_Type = {
     Py_TPFLAGS_METHOD_DESCRIPTOR,               /* tp_flags */
     func_new__doc__,                            /* tp_doc */
     (traverseproc)func_traverse,                /* tp_traverse */
-    (inquiry)func_clear,                        /* tp_clear */
+    (inquiry)0,                        /* tp_clear */
     0,                                          /* tp_richcompare */
     offsetof(PyFunctionObject, func_weakreflist), /* tp_weaklistoffset */
     0,                                          /* tp_iter */
msg350998 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 13:09
PyFunction_Type.tp_clear changed in bpo-33418 (previously, it was equal to 0: no clear method):

commit 3c452404ae178b742967589a0bb4a5ec768d76e0
Author: INADA Naoki <methane@users.noreply.github.com>
Date:   Wed Jul 4 11:15:50 2018 +0900

    bpo-33418: Add tp_clear for function object (GH-8058)
    
    Without tp_clear, GC can't break cyclic reference.
    It will cause memory leak when cyclic reference is
    created intentionally.
msg350999 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 13:15
In https://bugs.python.org/issue33418 I proposed reverting tp_clear on function objects.

>What should be done when a function with func_code=NULL is called?

We can set the error indicator at least. Although I agree that it seems suboptimal. At least I think we should add an assertion to check that the function called is valid so we can identify this situation easier.

What downsides do we see raising an exception?

Notice that this can happen without involving weak references. For example:

You have object A and B. A is a function. The destructor of B calls A. A and B are in a cycle. When calling tp_clear on A the refs of B reaches 0, trying to call A and .... Boom!

Either we remove tp_clear or somehow we have to protect all callers of the function. Technically, any field cleared from tp_cleared leaves the function in an inconsistent state
msg351006 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-02 14:46
> What downsides do we see raising an exception?

Yeah, on second thought, that would probably be best. We still want PR #15641 as well, so the exception doesn't actually happen in practice.

Note that _PyEval_EvalCodeWithName (the main use of func_globals) already fails with an exception if globals is NULL.

> Either we remove tp_clear or somehow we have to protect all callers of the function. Technically, any field cleared from tp_cleared leaves the function in an inconsistent state

tp_clear has a purpose; it prevents memory leaks. That leaves protecting access to the cleared fields.

(But we don't need to clear func_name and func_qualname as these must be strings.)
msg351007 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 15:06
The more I think about this the more I think there is something else at play. If the GC is able to follow the dependency chain, all weakrefs should have been already managed correctly and PyObject_ClearWeakRefs should have never be invoked.

I think *something* is not informing the gc about the dependency chain
msg351009 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-02 15:11
It's going to be hard to figure that out. FreeIPA uses a ton of C extensions. Some are hand-written, some uses Cython, ctypes, and cffi.
msg351010 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-09-02 15:13
The traceback does have some ctypedescr_dealloc and cfield_dealloc from _cffi. Could you check what those are?
msg351012 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 15:15
I think the problem is that whatever is weak-referenced by the weak ref (CField_Type or similar) is not on the gc list. Because is not on the gc list, handle_weakrefs (https://github.com/python/cpython/blob/master/Modules/gcmodule.c#L1090) is not acting correctly on the weakreferences, clearing them before the tp_clear calls and therefore producing the crash.
msg351015 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 15:44
Unless we are missing something I think this may be caused by something in cffi that is not implementing gc-related functions correctly, as PyObject_ClearWeakRefs should not be called from a gc run. Given how complicated this is and even if the possible solution is to fix the problem in cffi or elsewhere, we can add an extra check in PyObject_ClearWeakRefs to NOT call callbacks that are in the process of being collected. Technically we should never reach that code, but we won't be segfaulting and the cost is a redundant-check that will be false in the normal path.
msg351017 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-02 16:10
The biggest CFFI based dependency of FreeIPA is cryptography, but cryptography does not use weakref in its sources. CFFI on the other hand has a WeakValueDictionary object attached to its internal typecache backend, https://bitbucket.org/cffi/cffi/src/bf80722dea36237710083315e336c81bc8571fd0/cffi/model.py#lines-572 . Perhaps that is the culprit?

I checked the CTypeDescrObject from the stacktrace. Three out of three times it's for "unsigned long(*)(void *)". I wasn't able to get any useful information from the other fields yet.

(gdb) up 7
#7  0x00007fffe905babc in ctypedescr_dealloc (ct=0x7fffe6d958b0) at c/_cffi_backend.c:401
401             PyObject_ClearWeakRefs((PyObject *) ct);
(gdb) p (char*)ct.ct_name
$1 = 0x7fffe6d95908 "unsigned long(*)(void *)"
(gdb) p ct
$2 = (CTypeDescrObject *) 0x7fffe6d958b0
msg351020 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 16:23
For the weakref to be handled correctly the ctypedescr needs to be identified correctly as part of the isolated cycle. If is not in the isolated cycle something may be missing tp_traverse or the GC flags. 

Can you check if the ctypedescr is part of GC.objects()?
msg351037 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 21:09
I investigated the FreeIPA crash.

* Python 3.8 behaves differently because func_clear() has been implemented (bpo-33418, commit 3c452404ae178b742967589a0bb4a5ec768d76e0)

* The bug is a crash on a function call (_PyFunction_Vectorcall) because the function has been cleared (by func_clear), but there was still a weak reference using this function as its callback.

* Note: the function is called *during* it's being cleared by func_clear().

* The GC has a workaround for weak references part of "unreachable" objects, but its handle_weakrefs() function doesn't work because CFFI CField_Type type doesn't implement tp_traverse.

--

PR 15641 just hides the real bug.

One issue is that CFFI doesn't implement correctly the GC protocol. If an object contains another object, its type must:

* Have Py_TPFLAGS_HAVE_GC flag
* Implement tp_traverse
* Use PyObject_GC_Malloc() to allocate an object
* Call PyObject_GC_Track() on created object

Another issue is that the GC doesn't prevent the crash. Would it be possible to prevent the crash without changing the behavior (ex: still call weakref callbacks)?
msg351038 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 21:23
For the record, here is an annotated traceback of a FreeIPA crash.

PyGC_list_contains() is a hack that I wrote to check if an object was in the unreachable argument of delete_garbage().


(gdb) where
#0  0x0000000000434b78 in _PyFunction_Vectorcall (func=0x7ffff45d5690, stack=0x7fffffffd170, nargsf=1, kwnames=0x0) at Objects/call.c:406

    remove() func=0x7ffff45d5690

    (gdb) p _PyObject_Dump( ((PyFunctionObject*)func)->func_qualname )
    object  : 'WeakValueDictionary.__init__.<locals>.remove'

    (gdb) p PyGC_list_contains(0x7ffff7a2bac0, func)  # was unreachable?
    $8 = 1


#1  0x0000000000433640 in _PyObject_Vectorcall (callable=0x7ffff45d5690, args=0x7fffffffd170, nargsf=1, kwnames=0x0) at ./Include/cpython/abstract.h:127
#2  0x000000000043368b in _PyObject_FastCall (func=0x7ffff45d5690, args=0x7fffffffd170, nargs=1) at ./Include/cpython/abstract.h:147
#3  0x0000000000436ccc in object_vacall (base=0x0, callable=0x7ffff45d5690, vargs=0x7fffffffd1e0) at Objects/call.c:1186
#4  0x000000000043705b in PyObject_CallFunctionObjArgs (callable=0x7ffff45d5690) at Objects/call.c:1259
#5  0x0000000000502280 in handle_callback (ref=0x7ffff447dc90, callback=0x7ffff45d5690) at Objects/weakrefobject.c:877
#6  0x0000000000502403 in PyObject_ClearWeakRefs (object=0x7ffff42c1550) at Objects/weakrefobject.c:922

    (gdb) p object->ob_type->tp_name
    $13 = 0x7ffff6be404b "_cffi_backend.CTypeDescr"
    (gdb) p PyGC_list_contains(0x7ffff7a2bac0, object)  # was unreachable?
    $9 = 0

#7  0x00007ffff6a41abc in ctypedescr_dealloc (ct=0x7ffff42c1550) at c/_cffi_backend.c:401
#8  0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff42c1550) at Objects/object.c:2213
#9  0x00007ffff6a3e825 in _Py_DECREF (filename=<synthetic pointer>, lineno=703, op=<optimized out>) at /usr/include/python3.8/object.h:478
#10 cfield_dealloc (cf=0x7ffff444dc70) at c/_cffi_backend.c:703

    Py_DECREF(cf->cf_type);

    (gdb) p cf->ob_base.ob_type
    $63 = (struct _typeobject *) 0x7ffff6a62460 <CField_Type>
    (gdb) p PyGC_list_contains(0x7ffff7a2bac0, cf)  # was unreachable?
    $14 = 0

#11 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff444dc70) at Objects/object.c:2213
#12 0x0000000000463af8 in _Py_DECREF (filename=0x6987a0 "./Include/object.h", lineno=541, op=0x7ffff444dc70) at ./Include/object.h:478
#13 0x0000000000463b46 in _Py_XDECREF (op=0x7ffff444dc70) at ./Include/object.h:541
#14 0x00000000004652de in free_keys_object (keys=0x11de780) at Objects/dictobject.c:580
#15 0x0000000000464877 in dictkeys_decref (dk=0x11de780) at Objects/dictobject.c:324
#16 0x00000000004694b0 in dict_dealloc (mp=0x7ffff44a6950) at Objects/dictobject.c:1994

    {'C_Initialize': ..., 'C_CreateObject': ..., 'C_Finalize': ...}

    (gdb) p PyGC_list_contains(0x7ffff7a2bac0, mp)  # was unreachable?
    $16 = 1

#17 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff44a6950) at Objects/object.c:2213
#18 0x00007ffff6a41b25 in _Py_DECREF (filename=<synthetic pointer>, lineno=541, op=<optimized out>) at /usr/include/python3.8/object.h:478
#19 _Py_XDECREF (op=<optimized out>) at /usr/include/python3.8/object.h:541
#20 ctypedescr_dealloc (ct=0x7ffff42c12d0) at c/_cffi_backend.c:412

    Py_XDECREF(ct->ct_stuff);

    (gdb) p (char*)(((CTypeDescrObject*)ct)->ct_name)
    $38 = 0x7ffff42c1328 "struct _CK_FUNCTION_LIST"

#21 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff42c12d0) at Objects/object.c:2213
#22 0x000000000048f002 in _Py_DECREF (filename=0x6a2500 "./Include/object.h", lineno=541, op=0x7ffff42c12d0) at ./Include/object.h:478
#23 0x000000000048f02e in _Py_XDECREF (op=0x7ffff42c12d0) at ./Include/object.h:541
#24 0x000000000048fc0a in tupledealloc (op=0x7ffff451e050) at Objects/tupleobject.c:247

    (CTypeDescrObject,)

#25 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff451e050) at Objects/object.c:2213
#26 0x000000000048f002 in _Py_DECREF (filename=0x6a2500 "./Include/object.h", lineno=541, op=0x7ffff451e050) at ./Include/object.h:478
#27 0x000000000048f02e in _Py_XDECREF (op=0x7ffff451e050) at ./Include/object.h:541
#28 0x000000000048fc0a in tupledealloc (op=0x7ffff48e13c0) at Objects/tupleobject.c:247

    (str, <a tuple>)

#29 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff48e13c0) at Objects/object.c:2213
#30 0x0000000000491815 in _Py_DECREF (filename=0x6a34bf "Objects/typeobject.c", lineno=1110, op=0x7ffff48e13c0) at ./Include/object.h:478
#31 0x000000000049431d in clear_slots (type=0x8e0330, self=0x7ffff447d9f0) at Objects/typeobject.c:1110
#32 0x0000000000494833 in subtype_dealloc (self=0x7ffff447d9f0) at Objects/typeobject.c:1262

    weakref.KeyedRef

#33 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff447d9f0) at Objects/object.c:2213
#34 0x0000000000463af8 in _Py_DECREF (filename=0x6987a0 "./Include/object.h", lineno=541, op=0x7ffff447d9f0) at ./Include/object.h:478
#35 0x0000000000463b46 in _Py_XDECREF (op=0x7ffff447d9f0) at ./Include/object.h:541
#36 0x00000000004652de in free_keys_object (keys=0x1209c90) at Objects/dictobject.c:580
#37 0x0000000000464877 in dictkeys_decref (dk=0x1209c90) at Objects/dictobject.c:324
#38 0x00000000004694b0 in dict_dealloc (mp=0x7ffff45d64d0) at Objects/dictobject.c:1994

    WeakValueDictionary.data dict

#39 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff45d64d0) at Objects/object.c:2213
#40 0x000000000062bb37 in _Py_DECREF (filename=0x745d20 "./Include/object.h", lineno=541, op=0x7ffff45d64d0) at ./Include/object.h:478
#41 0x000000000062bb85 in _Py_XDECREF (op=0x7ffff45d64d0) at ./Include/object.h:541
#42 0x000000000062bf4a in cell_dealloc (op=0x7ffff45d7050) at Objects/cellobject.c:84

    remove() closure contains "d": WeakValueDictionary.data dict

#43 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff45d7050) at Objects/object.c:2213
#44 0x000000000048f002 in _Py_DECREF (filename=0x6a2500 "./Include/object.h", lineno=541, op=0x7ffff45d7050) at ./Include/object.h:478
#45 0x000000000048f02e in _Py_XDECREF (op=0x7ffff45d7050) at ./Include/object.h:541
#46 0x000000000048fc0a in tupledealloc (op=0x7ffff45d4370) at Objects/tupleobject.c:247
#47 0x000000000047f1a5 in _Py_Dealloc (op=0x7ffff45d4370) at Objects/object.c:2213
#48 0x0000000000449e1b in _Py_DECREF (filename=0x693783 "Objects/funcobject.c", lineno=584, op=0x7ffff45d4370) at ./Include/object.h:478

#49 0x000000000044b730 in func_clear (op=0x7ffff45d5690) at Objects/funcobject.c:584

    remove() func = 0x7ffff45d5690

#50 0x000000000058ebbd in delete_garbage (state=0x7fb798 <_PyRuntime+344>, collectable=0x7fffffffdc10, old=0x7fb7e0 <_PyRuntime+416>) at Modules/gcmodule.c:929

#51 0x000000000058f082 in collect (state=0x7fb798 <_PyRuntime+344>, generation=2, n_collected=0x0, n_uncollectable=0x0, nofail=1) at Modules/gcmodule.c:1106
#52 0x0000000000590861 in _PyGC_CollectNoFail () at Modules/gcmodule.c:1849
#53 0x0000000000542a1e in PyImport_Cleanup () at Python/import.c:541
#54 0x000000000055f8c3 in Py_FinalizeEx () at Python/pylifecycle.c:1226
#55 0x00000000005618d6 in Py_Exit (sts=0) at Python/pylifecycle.c:2248

#56 0x0000000000566726 in handle_system_exit () at Python/pythonrun.c:658
#57 0x000000000056673d in _PyErr_PrintEx (tstate=0x8044e0, set_sys_last_vars=1) at Python/pythonrun.c:668
#58 0x0000000000566a32 in PyErr_PrintEx (set_sys_last_vars=1) at Python/pythonrun.c:755
#59 0x0000000000566a43 in PyErr_Print () at Python/pythonrun.c:761
#60 0x000000000056602a in PyRun_SimpleFileExFlags (fp=0x800370, filename=0x7ffff7a1ff60 "/usr/sbin/ipa-server-install", closeit=1, flags=0x7fffffffe080) at Python/pythonrun.c:434
#61 0x000000000056555b in PyRun_AnyFileExFlags (fp=0x800370, filename=0x7ffff7a1ff60 "/usr/sbin/ipa-server-install", closeit=1, flags=0x7fffffffe080) at Python/pythonrun.c:86
#62 0x000000000042264a in pymain_run_file (config=0x8037e0, cf=0x7fffffffe080) at Modules/main.c:383
#63 0x0000000000422bfd in pymain_run_python (exitcode=0x7fffffffe0cc) at Modules/main.c:567
#64 0x0000000000422cee in Py_RunMain () at Modules/main.c:646
#65 0x0000000000422d68 in pymain_main (args=0x7fffffffe130) at Modules/main.c:676
#66 0x0000000000422de2 in Py_BytesMain (argc=3, argv=0x7fffffffe258) at Modules/main.c:700
#67 0x00000000004217a6 in main (argc=3, argv=0x7fffffffe258) at ./Programs/python.c:16
msg351039 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 21:25
> Another issue is that the GC doesn't prevent the crash. Would it be possible to prevent the crash without changing the behavior (ex: still call weakref callbacks)?

Pablo opened bpo-38009 with a PR. I'm not sure if his PR is correct or not yet.
msg351040 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 21:38
> Another issue is that the GC doesn't prevent the crash. Would it be possible to prevent the crash without changing the behavior (ex: still call weakref callbacks)?

I discussed with Pablo and it doesn't seem possible to prevent to clear a function if it is indirectly used as a callback for a weak reference to a object which doesn't implement tp_traverse properly (like the CFFI type).

The best that we can do is to detect the bug in C extension and emit a warning explaining the bug and suggesting a way to fix it.
msg351041 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 21:49
This bug is quite complex. Let me try to explain it more simply.

* Create an object A
* Create a weak reference to A with a callback CB
* A and CB are part of a reference cycle
* Removing the last references to A and CB are not enough to destroy them
* Trigger a garbage collection to break the cycle
* A and CB are seen as unreachable by the GC
* The GC "clears" CB which makes CB inconsistent (tp_clear)
* A is deleted which calls CB
* Crash on calling inconsistent CB

--

In the case of FreeIPA:

* A is a _cffi_backend.CTypeDescr instance
* CB is the remove() function defined in weakref.WeakValueDictionary constructor

The FreeIPA reference cycle is quite complex:

remove()
-> remove().__closure__ (tuple)
-> WeakValueDictionary.data (dict) = remove().__closure__[0]
-> weakref.KeyedRef
-> (<str 1>, <tuple 2>)
-> (<CTypeDescrObject 3>,) = <tuple 2>
-> <CTypeDescrObject 3>
-> {'C_Initialize': ..., 'C_CreateObject': ..., 'C_Finalize': ...} (dict)
-> <_cffi_backend.CField 4>
-> <_cffi_backend.CTypeDescr 5>
-> remove() = callback of a weak reference to <_cffi_backend.CTypeDescr 5>

<_cffi_backend.CField 4> is not seen as unreachable by the GC because CField_Type.tp_traverse is not implemented (and CField_Type doesn't use Py_TPFLAGS_HAVE_GC flag).
msg351042 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-02 21:53
> * A and CB are seen as unreachable by the GC

Oh, that's not correct:

* Only CB is seen as unreachable by the GC
* The GC "clears" CB which makes CB inconsistent (tp_clear)
* A is deleted indirectly
* Deleting A calls CB through the weak reference to A

The "A is deleted indirectly" step is also complex...
msg351067 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-03 07:07
Thanks Victor!

I have opened a bug for CFFI, https://bitbucket.org/cffi/cffi/issues/416/python-38-segfault-cfield_type-does-not
msg351072 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-03 07:56
Pablo, Victor, could you please assist Armin on the CFFI issue? As usual the situation is more complex than I anticipated.
msg351077 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-03 08:45
Attached crash.tar.gz: minimum reproducer which doesn't depend on FreeIPA anymore.

Depends on:

* cffi (_cffi_backend dynamic library)
* pycparser
* Python 3.8 (need "python3.8")

To reproduce the crash:

$ tar -xf reproducer.tar.gz 
$ cd reproducer/
$ ./bug.sh 

Output:

(...)
++ env/bin/python ./crash.py
exit
./bug.sh : ligne 7 : 14529 Segmentation fault      (core dumped)env/bin/python ./crash.py
++ echo 'exitcode: 139'
exitcode: 139
msg351082 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-03 12:21
I failed to write a reproducer from scratch. So let me share my notes here. The first point is that remove() function of WeakValueDictionary keeps WeakValueDictionary.data alive like that:

---
class NoisyDel:
    def __del__(self):
        print("dealloc data dict")

def create_closure():
    data = {0: NoisyDel()}
    def remove():
        return data
    return remove

remove = create_closure()
print("clear ")
remove = None
print("exit")
---

data is only deleted once remove is cleared.
msg351083 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-03 12:45
Second step: when func_clear() is called, the closure is cleared. Clearing the closure should call the function which is being cleared (or is already cleared).

---
import gc

class CallFunc:
    def __del__(self):
        func = self.func
        if func is None:
            return
        print("CallFunc", func)
        func("call func from CallFunc.__del__")

def create_remove():
    call_func = CallFunc()
    def remove(msg):
        x = call_func
        print(msg)
    remove.__name__ = "evil_func"
    call_func.func = remove
    return call_func, remove

call_func, remove = create_remove()
print("== clear ==")
call_func = None
remove = None
print()

print("== collect ==")
gc.collect()
print()

print("== exit ==")
---

Output (with my hacked CPython):
------------------
== clear ==

== collect ==
CallFunc <function create_remove.<locals>.remove at 0x7fab8a189eb0>
call func from CallFunc.__del__
func_clear remove() -- in delete_garbage? 1

== exit ==
------------------

call_func and remove are part of a reference cycle. A forced garbage collection breaks the cycle and removes the two objects, but they are not removed in the expected order:

* first: call_func
* then: remove

The crash requires to destroy the objects in the reverse order
msg351086 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-03 14:20
Aha! I reproduced the crash with attached gc_crash.py and gc_crash.patch:

# cd /path/to/python/source
$ git apply gc_crash.patch
$ make
$ ./python gc_crash.py
Segmentation fault (core dumped)
msg351098 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-03 19:22
gc_crash.py:

* PyObject_GC_UnTrack() + PyObject_GC_Track() is used to order objects in the GC collection 0, the objects which are involved in the reference cycle.

* BadGC2Type type is implemented in C, uses Py_TPFLAGS_HAVE_GC, implements tp_traverse, but it doesn't implement tp_clear.

* IHMO gc_crash.py is only legit code. BadGC2Type implements the GC protocol (except to tp_clear).


I'm not sure that it's the same bug than reproducer.tar.gz which uses a weak reference.

BadGC2Type is used to call a function in tp_dealloc. It should be possible to use a weak reference for that: the weak reference can use a callback. But I failed write a reproducer script using a weak reference.
msg351099 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-03 19:30
collect() of gcmodule.c:

* collect() builds an "unreachable" list which is quite important in this bug
* the bug occurs in delete_garbage() which uses the unreachable list
* weak references part of unreachable are handled before delete_garbage() in handle_weakrefs(): "Clear weakrefs and invoke callbacks as necessary".

It seems like reproducer.tar.gz and gc_crash.py crashes involve a reference cycle.

reproducer.tar.gz uses a weak reference which is *not* part of the reference cycle.
msg351492 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-09 14:56
New changeset a2af05a0d3f0da06b8d432f52efa3ecf29038532 by Victor Stinner in branch 'master':
bpo-38006: Avoid closure in weakref.WeakValueDictionary (GH-15641)
https://github.com/python/cpython/commit/a2af05a0d3f0da06b8d432f52efa3ecf29038532
msg351519 - (view) Author: miss-islington (miss-islington) Date: 2019-09-09 16:24
New changeset 78d15faf6c522619098e94be3e7f6d88a9e61123 by Miss Islington (bot) in branch '3.8':
bpo-38006: Avoid closure in weakref.WeakValueDictionary (GH-15641)
https://github.com/python/cpython/commit/78d15faf6c522619098e94be3e7f6d88a9e61123
msg351561 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-09 22:31
New changeset 23669330b7d0d5ad1a9aac40315ba4c2e765f9dd by Victor Stinner in branch '3.7':
bpo-38006: Avoid closure in weakref.WeakValueDictionary (GH-15641) (GH-15789)
https://github.com/python/cpython/commit/23669330b7d0d5ad1a9aac40315ba4c2e765f9dd
msg351676 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-10 14:32
I reverted the change which added func_clear() to get more time to investigate this bug:

New changeset ccaea525885e41c5f1e566bb68698847faaa82ca by T. Wouters (Victor Stinner) in branch '3.8':
Revert "bpo-33418: Add tp_clear for function object (GH-8058)" (GH-15826)
https://github.com/python/cpython/commit/ccaea525885e41c5f1e566bb68698847faaa82ca

This issue looks quite complex, and I prefer to wait Python 3.9 to fix it. We are stabilizing Python 3.8 to prepare 3.8.0 final release.
History
Date User Action Args
2019-09-10 14:32:57vstinnersetmessages: + msg351676
2019-09-09 22:31:25vstinnersetmessages: + msg351561
2019-09-09 16:24:19miss-islingtonsetnosy: + miss-islington
messages: + msg351519
2019-09-09 15:00:00vstinnersetpull_requests: + pull_request15442
2019-09-09 14:56:12miss-islingtonsetpull_requests: + pull_request15440
2019-09-09 14:56:03vstinnersetmessages: + msg351492
2019-09-04 10:49:03inada.naokisetnosy: + inada.naoki
2019-09-03 19:30:29vstinnersetmessages: + msg351099
2019-09-03 19:22:20vstinnersetmessages: + msg351098
2019-09-03 14:20:39vstinnersetfiles: + gc_crash.patch
2019-09-03 14:20:30vstinnersetfiles: + gc_crash.py

messages: + msg351086
2019-09-03 12:45:01vstinnersetmessages: + msg351083
2019-09-03 12:21:13vstinnersetmessages: + msg351082
2019-09-03 08:45:31vstinnersetfiles: + reproducer.tar.gz

messages: + msg351077
2019-09-03 07:56:10christian.heimessetmessages: + msg351072
2019-09-03 07:07:54christian.heimessetmessages: + msg351067
2019-09-02 21:53:39vstinnersetmessages: + msg351042
2019-09-02 21:49:58vstinnersetmessages: + msg351041
2019-09-02 21:38:36vstinnersetmessages: + msg351040
2019-09-02 21:25:34vstinnersetmessages: + msg351039
2019-09-02 21:23:30vstinnersetmessages: + msg351038
2019-09-02 21:09:22vstinnersetmessages: + msg351037
2019-09-02 19:38:25pablogsalsetmessages: - msg351022
2019-09-02 16:26:02pablogsalsetmessages: + msg351022
2019-09-02 16:23:15pablogsalsetmessages: + msg351020
2019-09-02 16:10:40christian.heimessetmessages: + msg351017
2019-09-02 15:44:02pablogsalsetmessages: + msg351015
2019-09-02 15:15:57pablogsalsetmessages: + msg351012
2019-09-02 15:13:37petr.viktorinsetmessages: + msg351010
2019-09-02 15:11:22christian.heimessetmessages: + msg351009
2019-09-02 15:06:40pablogsalsetmessages: + msg351007
2019-09-02 14:46:17petr.viktorinsetmessages: + msg351006
2019-09-02 13:15:12pablogsalsetmessages: + msg350999
2019-09-02 13:09:25vstinnersetmessages: + msg350998
2019-09-02 13:07:54vstinnersetmessages: + msg350997
2019-09-02 12:55:34petr.viktorinsetmessages: + msg350996
2019-09-02 12:46:24pablogsalsetmessages: + msg350995
2019-09-02 12:41:34pablogsalsetmessages: + msg350992
2019-09-02 12:34:51pablogsalsetmessages: + msg350991
2019-09-02 12:29:24pablogsalsetnosy: + pitrou, pablogsal
2019-09-02 12:09:12christian.heimessetmessages: + msg350990
2019-09-02 11:40:33vstinnersetmessages: + msg350987
2019-09-02 11:37:57vstinnersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request15307
2019-09-02 11:27:53vstinnersetmessages: + msg350985
2019-09-02 11:10:03vstinnersettitle: _PyFunction_Vectorcall() can segfault on process exit -> Crash in remove() weak reference callback of weakref.WeakValueDictionary at Python exit
2019-09-02 11:08:05vstinnersetmessages: + msg350983
2019-09-02 11:00:31xtreaksetnosy: + jdemeyer
2019-09-02 10:57:55vstinnersetmessages: + msg350981
2019-09-02 10:11:48christian.heimessetmessages: + msg350976
2019-09-02 10:06:56petr.viktorinsetmessages: + msg350975
2019-09-02 09:48:27christian.heimescreate