I'm working on porting an application from Python 3.6.8 to 3.7.3 and am running into this segfault.
I'm not able to pin point exactly what Python code is causing this as it's a very large enterprise application
and I can't seem to pinpoint exactly where this is coming from.

The application consistently crashes on start every time with this stack trace.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7922ae5 in type_dealloc (type=0x7fffed185480) at /usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283
3283	    _PyObject_GC_UNTRACK(type);
(gdb) bt
#0  0x00007ffff7922ae5 in type_dealloc (type=0x7fffed185480) at /usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283
#1  0x00007ffff78ee721 in free_keys_object (keys=0x7fffee258bc0) at /usr/src/debug/Python-3.7.3/Objects/dictobject.c:559
#2  0x00007ffff78f20af in dict_dealloc (mp=0x7fffee209c60) at /usr/src/debug/Python-3.7.3/Objects/dictobject.c:1913
#3  0x00007ffff7904d15 in module_clear (m=0x7fffee211048) at /usr/src/debug/Python-3.7.3/Objects/moduleobject.c:749
#4  0x00007ffff7a3aeda in delete_garbage (collectable=0x7fffffff7420, old=0x7ffff7dd75f0 <_PyRuntime+400>) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:769
#5  0x00007ffff7a3b41d in collect (generation=2, n_collected=0x7fffffff7488, n_uncollectable=0x7fffffff7490, nofail=0)
    at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:924
#6  0x00007ffff7a3b845 in collect_with_callback (generation=2) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1036
#7  0x00007ffff7a3bcef in gc_collect_impl (module=0x7ffff615d9f8, generation=2) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1138
#8  0x00007ffff7a3ba13 in gc_collect (module=0x7ffff615d9f8, args=0xcc6cd0, nargs=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Modules/clinic/gcmodule.c.h:100
#9  0x00007ffff78b48e5 in _PyMethodDef_RawFastCallKeywords (method=0x7ffff7d8b800 <GcMethods+256>, self=0x7ffff615d9f8, args=0xcc6cd0, nargs=0, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:656
#10 0x00007ffff78b4bab in _PyCFunction_FastCallKeywords (func=0x7ffff615ecf0, args=0xcc6cd0, nargs=0, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:730
#11 0x00007ffff79cbf2c in call_function (pp_stack=0x7fffffff78b8, oparg=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4568
#12 0x00007ffff79c6344 in _PyEval_EvalFrameDefault (f=0xcc6b08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3093
#13 0x00007ffff79b8067 in PyEval_EvalFrameEx (f=0xcc6b08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547
#14 0x00007ffff78b3b1a in function_code_fastcall (co=0x7ffff620bc00, args=0x7ffff64fef90, nargs=1, globals=0x7ffff64e6120)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:283
#15 0x00007ffff78b4096 in _PyFunction_FastCallKeywords (func=0x7fffedb63488, stack=0x7ffff64fef88, nargs=1, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:408
#16 0x00007ffff79cc357 in call_function (pp_stack=0x7fffffff9140, oparg=1, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4616
#17 0x00007ffff79c64c9 in _PyEval_EvalFrameDefault (f=0x7ffff64fee08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3124
#18 0x00007ffff79b8067 in PyEval_EvalFrameEx (f=0x7ffff64fee08, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547
#19 0x00007ffff79ca14f in _PyEval_EvalCodeWithName (_co=0x7ffff620bc90, globals=0x7ffff64e6120, locals=0x7ffff64e6120, args=0x0, argcount=0, kwnames=0x0, 
    kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3930
#20 0x00007ffff79ca347 in PyEval_EvalCodeEx (_co=0x7ffff620bc90, globals=0x7ffff64e6120, locals=0x7ffff64e6120, args=0x0, argcount=0, kws=0x0, kwcount=0, 
    defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3959
#21 0x00007ffff79b7ff1 in PyEval_EvalCode (co=0x7ffff620bc90, globals=0x7ffff64e6120, locals=0x7ffff64e6120) at /usr/src/debug/Python-3.7.3/Python/ceval.c:524
#22 0x00007ffff79b2963 in builtin_exec_impl (module=0x7ffff7e61c28, source=0x7ffff620bc90, globals=0x7ffff64e6120, locals=0x7ffff64e6120)
    at /usr/src/debug/Python-3.7.3/Python/bltinmodule.c:1079
#23 0x00007ffff79aff3b in builtin_exec (module=0x7ffff7e61c28, args=0x6c5b58, nargs=2) at /usr/src/debug/Python-3.7.3/Python/clinic/bltinmodule.c.h:283
#24 0x00007ffff78b48bd in _PyMethodDef_RawFastCallKeywords (method=0x7ffff7d7bae0 <builtin_methods+480>, self=0x7ffff7e61c28, args=0x6c5b58, nargs=2, 
    kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:651
#25 0x00007ffff78b4bab in _PyCFunction_FastCallKeywords (func=0x7ffff6566d80, args=0x6c5b58, nargs=2, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:730
#26 0x00007ffff79cbf2c in call_function (pp_stack=0x7fffffffada0, oparg=2, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4568
#27 0x00007ffff79c64c9 in _PyEval_EvalFrameDefault (f=0x6c5998, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3124
#28 0x00007ffff79b8067 in PyEval_EvalFrameEx (f=0x6c5998, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547
#29 0x00007ffff79ca14f in _PyEval_EvalCodeWithName (_co=0x7ffff648d930, globals=0x7ffff651b360, locals=0x0, args=0x67c820, argcount=5, kwnames=0x0, 
    kwargs=0x67c848, kwcount=0, kwstep=1, defs=0x7ffff6487a58, defcount=5, kwdefs=0x0, closure=0x0, name=0x7ffff61fa230, qualname=0x7ffff61fa230)
    at /usr/src/debug/Python-3.7.3/Python/ceval.c:3930
#30 0x00007ffff78b41c1 in _PyFunction_FastCallKeywords (func=0x7ffff61e89d8, stack=0x67c820, nargs=5, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:433
#31 0x00007ffff79cc357 in call_function (pp_stack=0x7fffffffc830, oparg=5, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:4616
#32 0x00007ffff79c64c9 in _PyEval_EvalFrameDefault (f=0x67c678, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3124
#33 0x00007ffff79b8067 in PyEval_EvalFrameEx (f=0x67c678, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547
--Type <RET> for more, q to quit, c to continue without paging--
#34 0x00007ffff79ca14f in _PyEval_EvalCodeWithName (_co=0x7ffff6485270, globals=0x7ffff651b360, locals=0x0, args=0x7ffff6462620, argcount=2, kwnames=0x0, 
    kwargs=0x0, kwcount=0, kwstep=2, defs=0x7ffff64db4c0, defcount=1, kwdefs=0x0, closure=0x0, name=0x7ffff64a0150, qualname=0x7ffff64a0150)
    at /usr/src/debug/Python-3.7.3/Python/ceval.c:3930
#35 0x00007ffff78b3f8b in _PyFunction_FastCallDict (func=0x7ffff6218730, args=0x7ffff6462620, nargs=2, kwargs=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:376
#36 0x00007ffff78b38af in PyObject_Call (callable=0x7ffff6218730, args=0x7ffff6462608, kwargs=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:226
#37 0x00007ffff7a3195e in pymain_run_module (modname=0x603490 L"cb.enterprise.main", set_argv0=1) at /usr/src/debug/Python-3.7.3/Modules/main.c:320
#38 0x00007ffff7a399ab in pymain_run_python (pymain=0x7fffffffe1d0) at /usr/src/debug/Python-3.7.3/Modules/main.c:2874
#39 0x00007ffff7a39ee2 in pymain_main (pymain=0x7fffffffe1d0) at /usr/src/debug/Python-3.7.3/Modules/main.c:3038
#40 0x00007ffff7a39f9b in _Py_UnixMain (argc=5, argv=0x7fffffffe318) at /usr/src/debug/Python-3.7.3/Modules/main.c:3073
#41 0x00007ffff67c6d20 in __libc_start_main () from /lib64/libc.so.6
#42 0x0000000000400691 in _start ()

Alright I was finally able to reproduce the issue with a small code snip and capture the stack in gdb.  It looks like the issue has something to do with an exception thrown while importing a class (and possibly only when the class is a c_ext, but I haven't confirmed that)

[933fdf38d838">root@933fdf38d838 ~]# python3.7 -m venv crash-test
[933fdf38d838">root@933fdf38d838 ~]# crash-test/bin/pip install protobuf
Looking in indexes: https://files.pythonhosted.org/simple
Collecting protobuf
  Downloading https://files.pythonhosted.org/packages/ce/86/9f6123c4c6f481862f286dbe13aa2e97bdedd7662f5fc3033c1a41f32f88/protobuf-3.8.0-cp37-cp37m-manylinux1_x86_64.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 2.3MB/s 
Collecting six>=1.9 (from protobuf)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./crash-test/lib/python3.7/site-packages (from protobuf) (40.8.0)
Installing collected packages: six, protobuf
Successfully installed protobuf-3.8.0 six-1.12.0
[933fdf38d838">root@933fdf38d838 ~]# scl enable devtoolset-8 "gdb ./crash-test/bin/python"
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-2.el6
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./crash-test/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python3.7m.debug...
warning: Skipping deprecated .gdb_index section in /usr/lib/debug/usr/bin/python3.7m.debug.
Do "set use-deprecated-index-sections on" before the file is read
to use the section anyway.
done.
done.
(gdb) run
Starting program: /root/crash-test/bin/python 
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Python 3.7.3 (default, Jun 27 2019, 15:27:42) 
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import warnings
>>> import gc
>>> 
>>> gc.disable()
>>> 
>>> for i in range(100):
...     warnings.filterwarnings('error')
...     try:
...         from google.protobuf.pyext import _message
...     except Exception as ex:
...         print(i)
...     
...     gc.collect()
... 
0
1
2
3
4
5

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7922ae5 in type_dealloc (type=0x7ffff2277480) at /usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283
3283	    _PyObject_GC_UNTRACK(type);
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.5-7.el6_0.x86_64 expat-2.0.1-13.el6_8.x86_64 libgcc-4.4.7-23.el6.x86_64 libstdc++-4.4.7-23.el6.x86_64 ncurses-libs-5.7-4.20090207.el6.x86_64 nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64 openresty-openssl-1.0.2s-1.el6.x86_64 readline-6.0-4.el6.x86_64 xz-libs-4.999.9-0.5.beta.20091007git.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff7922ae5 in type_dealloc (type=0x7ffff2277480) at /usr/src/debug/Python-3.7.3/Objects/typeobject.c:3283
#1  0x00007ffff78ee721 in free_keys_object (keys=0x7ffff64e3bc0) at /usr/src/debug/Python-3.7.3/Objects/dictobject.c:559
#2  0x00007ffff78f20af in dict_dealloc (mp=0x7ffff47a1708) at /usr/src/debug/Python-3.7.3/Objects/dictobject.c:1913
#3  0x00007ffff7904d15 in module_clear (m=0x7ffff22f3ea8) at /usr/src/debug/Python-3.7.3/Objects/moduleobject.c:749
#4  0x00007ffff7a3aeda in delete_garbage (collectable=0x7fffffffc250, old=0x7ffff7dd75f0 <_PyRuntime+400>)
    at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:769
#5  0x00007ffff7a3b41d in collect (generation=2, n_collected=0x7fffffffc2b8, n_uncollectable=0x7fffffffc2c0, nofail=0)
    at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:924
#6  0x00007ffff7a3b845 in collect_with_callback (generation=2) at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1036
#7  0x00007ffff7a3bcef in gc_collect_impl (module=0x7ffff61f6138, generation=2)
    at /usr/src/debug/Python-3.7.3/Modules/gcmodule.c:1138
#8  0x00007ffff7a3ba13 in gc_collect (module=0x7ffff61f6138, args=0x7ffff64e7768, nargs=0, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Modules/clinic/gcmodule.c.h:100
#9  0x00007ffff78b48e5 in _PyMethodDef_RawFastCallKeywords (method=0x7ffff7d8b800 <GcMethods+256>, self=0x7ffff61f6138, 
    args=0x7ffff64e7768, nargs=0, kwnames=0x0) at /usr/src/debug/Python-3.7.3/Objects/call.c:656
#10 0x00007ffff78b4bab in _PyCFunction_FastCallKeywords (func=0x7ffff61f8438, args=0x7ffff64e7768, nargs=0, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Objects/call.c:730
#11 0x00007ffff79cbf2c in call_function (pp_stack=0x7fffffffc6e8, oparg=0, kwnames=0x0)
    at /usr/src/debug/Python-3.7.3/Python/ceval.c:4568
#12 0x00007ffff79c6344 in _PyEval_EvalFrameDefault (f=0x7ffff64e75e8, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3093
#13 0x00007ffff79b8067 in PyEval_EvalFrameEx (f=0x7ffff64e75e8, throwflag=0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:547
#14 0x00007ffff79ca14f in _PyEval_EvalCodeWithName (_co=0x7ffff6484810, globals=0x7ffff64e6120, locals=0x7ffff64e6120, args=0x0, 
    argcount=0, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0)
    at /usr/src/debug/Python-3.7.3/Python/ceval.c:3930
#15 0x00007ffff79ca347 in PyEval_EvalCodeEx (_co=0x7ffff6484810, globals=0x7ffff64e6120, locals=0x7ffff64e6120, args=0x0, 
    argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at /usr/src/debug/Python-3.7.3/Python/ceval.c:3959
#16 0x00007ffff79b7ff1 in PyEval_EvalCode (co=0x7ffff6484810, globals=0x7ffff64e6120, locals=0x7ffff64e6120)
    at /usr/src/debug/Python-3.7.3/Python/ceval.c:524
#17 0x00007ffff7a0efcf in run_mod (mod=0x6b7790, filename=0x7ffff6514ae8, globals=0x7ffff64e6120, locals=0x7ffff64e6120, 
    flags=0x7fffffffe1dc, arena=0x7ffff7e692d0) at /usr/src/debug/Python-3.7.3/Python/pythonrun.c:1035
#18 0x00007ffff7a0ca53 in PyRun_InteractiveOneObjectEx (fp=0x7ffff6b36f60 <_IO_2_1_stdin_>, filename=0x7ffff6514ae8, 
    flags=0x7fffffffe1dc) at /usr/src/debug/Python-3.7.3/Python/pythonrun.c:256
#19 0x00007ffff7a0c42e in PyRun_InteractiveLoopFlags (fp=0x7ffff6b36f60 <_IO_2_1_stdin_>, filename_str=0x7ffff7ada976 "<stdin>", 
    flags=0x7fffffffe1dc) at /usr/src/debug/Python-3.7.3/Python/pythonrun.c:120
#20 0x00007ffff7a0c27e in PyRun_AnyFileExFlags (fp=0x7ffff6b36f60 <_IO_2_1_stdin_>, filename=0x7ffff7ada976 "<stdin>", closeit=0, 
    flags=0x7fffffffe1dc) at /usr/src/debug/Python-3.7.3/Python/pythonrun.c:78
#21 0x00007ffff7a31e5e in pymain_run_file (fp=0x7ffff6b36f60 <_IO_2_1_stdin_>, filename=0x0, p_cf=0x7fffffffe1dc)
    at /usr/src/debug/Python-3.7.3/Modules/main.c:427
#22 0x00007ffff7a34e51 in pymain_run_filename (pymain=0x7fffffffe230, cf=0x7fffffffe1dc)
    at /usr/src/debug/Python-3.7.3/Modules/main.c:1627
#23 0x00007ffff7a399cf in pymain_run_python (pymain=0x7fffffffe230) at /usr/src/debug/Python-3.7.3/Modules/main.c:2877
#24 0x00007ffff7a39ee2 in pymain_main (pymain=0x7fffffffe230) at /usr/src/debug/Python-3.7.3/Modules/main.c:3038
#25 0x00007ffff7a39f9b in _Py_UnixMain (argc=1, argv=0x7fffffffe398) at /usr/src/debug/Python-3.7.3/Modules/main.c:3073
#26 0x0000000000400772 in main (argc=1, argv=0x7fffffffe398) at /usr/src/debug/Python-3.7.3/Programs/python.c:15
(gdb) p type->tp_name
$1 = 0x7ffff20034c0 "google.protobuf.pyext._message.MessageMeta"
(gdb) p type->tp_doc
$2 = 0x7ffff20034f0 "The metaclass of ProtocolMessages"
(gdb)

I've realized this might not be Python 3.7 specific, the reason protobuf is throwing an exception during import in my above example (as well as in our application) is because it's throwing a deprecation warning that was only introduced in 3.7.  

Python 3.6 doesn't cause any warnings importing protobuf so it doesn't abort the import.  Might need a different test case to confirm what versions this does apply to.

I've made a change to our application's error handling in this area so we aren't having a segfault anymore, but this is still a bug in Python itself.

Could also be a bug in however protobuf handles exceptions on module initialization.

Could also be a bug in however protobuf handles exceptions on module initialization.

I'll admit I don't know enough about Python c extensions to make that call, but the fact that it wasn't crashing in the protobuf extension itself during the import but instead during the next garbage collection made me believe it was a Python bug. Maybe that's not a correct assumption though.

I haven't used protobuf, but it's _generally_ true that crashes that occur for the first time in the presence of C or C++ extension modules are due to subtle (or not so subtle) mistakes in using the sometimes-delicate Python C API.  So it's too soon to play "pin the blame on the donkey here" ;-)  But resolving it really needs someone who knows what protobuf is doing.

I'm unable to reproduce the crash on Fedora 31.

A crash during a garbage collection usually means that there is a corrupted object somewhere, and suddenly, we discover the inconsistent. The crash can be far from where the inconsistency was created.

You may try python3.7 -X dev which might provide more information. The best would be to test a Python 3.9 compiled in debug mode.

The setup.py file of Python binding of protobuf has an optional --cpp_implementation option to build C++ extension modules. It seems like they are not built by default. I'm not sure if this bug requires these C++ extension modules, or if they must miss.

It seems like "pip install protobuf" installs protobuf without these C++ extension modules.

See also https://pythondev.readthedocs.io/debug_tools.html to debug a crash.

It's likely a bug in protobuf, rather than a bug in Python itself. I close the issue.