Hi, I'm dealing with random crashes when using pymongo in Python 3.7.3 in a Debian Buster.

This is the python backtrace:
(gdb) thread apply all py-bt

Thread 2 (Thread 0x7f9817d95700 (LWP 221)):
Traceback (most recent call first):
  File "/usr/local/lib/python3.7/dist-packages/gevent/_threading.py", line 80, in wait
    waiter.acquire() # Block on the native lock
  File "/usr/local/lib/python3.7/dist-packages/gevent/_threading.py", line 162, in get
    self._not_empty.wait()
  File "/usr/local/lib/python3.7/dist-packages/gevent/threadpool.py", line 270, in _worker
    task = task_queue.get()
  File "/usr/local/lib/python3.7/dist-packages/gevent/threadpool.py", line 254, in __trampoline
    g.switch()

Thread 1 (Thread 0x7f981fdfd740 (LWP 216)):
Traceback (most recent call first):
  <built-in method decode_all of module object at remote 0x7f981d41ca48>
  File "/usr/local/lib/python3.7/dist-packages/bson/__init__.py", line 1089, in _decode_all_selective
    return decode_all(data, codec_options)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/message.py", line 1616, in unpack_response
    self.payload_document, codec_options, user_fields)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1080, in _unpack_response
    legacy_response)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/server.py", line 131, in run_operation_with_response
    user_fields=user_fields)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/mongo_client.py", line 1366, in _cmd
    unpack_res)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
    return func(session, server, sock_info, slave_ok)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/mongo_client.py", line 1372, in _run_operation_with_response
    exhaust=exhaust)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1001, in __send_message
    address=self.__address)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1124, in _refresh
    self.__send_message(q)
  File "/usr/local/lib/python3.7/dist-packages/pymongo/cursor.py", line 1207, in next
    if len(self.__data) or self._refresh():
  File "/usr/local/lib/python3.7/dist-packages/pymongo/collection.py", line 1319, in find_one
    for result in cursor.limit(-1):
  File "/usr/local/lib/python3.7/dist-packages/gecoscc/userdb.py", line 119, in create_user
    user = self.collection.find_one({'email': email})
  File "/usr/local/lib/python3.7/dist-packages/gecoscc/commands/create_adminuser.py", line 95, in command
    {'is_superuser': self.options.is_superuser}
  File "/usr/local/lib/python3.7/dist-packages/gecoscc/management.py", line 90, in __call__
    self.command()
  File "/usr/local/lib/python3.7/dist-packages/gecoscc/management.py", line 48, in main
    command()
  File "/usr/local/bin/pmanage", line 10, in <module>
    sys.exit(main())
(gdb) 

And this is the builtin-code backtrace:

Core was generated by `/usr/bin/python3 /usr/local/bin/pmanage /opt/gecosccui/gecoscc.ini create_admin'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  PyErr_SetObject (exception=0x7ff9c0 <_PyExc_AttributeError.lto_priv.2311>, value=0x7f1ab49cb098) at ../Python/errors.c:101
101	        Py_INCREF(exc_value);
[Current thread is 1 (Thread 0x7f1abc823740 (LWP 370))]
(gdb) bt
#0  PyErr_SetObject (exception=<type at remote 0x7ff9c0>, value="type object 'dict' has no attribute '_type_marker'") at ../Python/errors.c:101
#1  0x000000000052c23b in PyErr_FormatV (vargs=0x7ffedff77c40, format=<optimized out>, exception=<type at remote 0x7ff9c0>) at ../Python/errors.c:852
#2  PyErr_Format (exception=<type at remote 0x7ff9c0>, format=<optimized out>) at ../Python/errors.c:852
#3  0x000000000058717d in type_getattro (type=<optimized out>, name=<optimized out>) at ../Objects/typeobject.c:3223
#4  0x000000000054baae in _PyObject_LookupAttr (result=<synthetic pointer>, name=<optimized out>, v=<type at remote 0x81a240>) at ../Objects/object.c:949
#5  builtin_getattr (self=<optimized out>, args=<optimized out>, nargs=<optimized out>) at ../Python/bltinmodule.c:1121
#6  0x00000000005cccc3 in _PyMethodDef_RawFastCallKeywords (method=0x89d160 <builtin_methods+544>, self=<module at remote 0x7f1abc52bc28>, args=0x1237208, nargs=<optimized out>, kwnames=<optimized out>)
    at ../Objects/call.c:651
#7  0x00000000005463e3 in _PyCFunction_FastCallKeywords (kwnames=0x0, nargs=3, args=0x1237208, func=<built-in method getattr of module object at remote 0x7f1abc52bc28>) at ../Objects/call.c:730
#8  call_function (kwnames=0x0, oparg=3, pp_stack=<synthetic pointer>) at ../Python/ceval.c:4568
#9  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3124
#10 0x00000000005cd68c in PyEval_EvalFrameEx (throwflag=0, 
    f=Frame 0x1237088, for file /usr/local/lib/python3.7/dist-packages/bson/codec_options.py, line 35, in _raw_document_class (document_class=<type at remote 0x81a240>)) at ../Python/ceval.c:547
#11 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#12 _PyFunction_FastCallKeywords (func=<optimized out>, stack=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ../Objects/call.c:408
#13 0x000000000054207c in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at ../Python/ceval.c:4616
#14 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3124
#15 0x000000000053f732 in PyEval_EvalFrameEx (throwflag=0, 
    f=Frame 0x17882e8, for file /usr/local/lib/python3.7/dist-packages/bson/__init__.py, line 1013, in decode_all (data=b'V\x00\x00\x00\x03cursor\x00=\x00\x00\x00\x04firstBatch\x00\x05\x00\x00\x00\x00\x12id\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02ns\x00\x13\x00\x00\x00gecoscc.adminusers\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\xf0?\x00', codec_options=<CodecOptions at remote 0x7f1ab4ac2208>, view=<memoryview at remote 0x7f1ab5375708>, data_len=86, docs=[], position=0, end=85)) at ../Python/ceval.c:547
#16 _PyEval_EvalCodeWithName (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7f1ab49de978, kwcount=<optimized out>, 
    kwstep=1, defs=0x7f1ab9e42220, defcount=1, kwdefs=0x0, closure=0x0, name='decode_all', qualname='decode_all') at ../Python/ceval.c:3930
#17 0x00000000005cd982 in _PyFunction_FastCallKeywords (func=<optimized out>, stack=0x7f1ab49de968, nargs=2, kwnames=<optimized out>) at ../Objects/call.c:433
#18 0x000000000054207c in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at ../Python/ceval.c:4616
#19 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at ../Python/ceval.c:3124
#20 0x00000000005cd68c in PyEval_EvalFrameEx (throwflag=0, 
    f=Frame 0x7f1ab49de7c8, for file /usr/local/lib/python3.7/dist-packages/bson/__init__.py, line 1089, in _decode_all_selective (data=b'V\x00\x00\x00\x03cursor\x00=\x00\x00\x00\x04firstBatch\x00\x05\x00\x00\x00\x00\x12id\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02ns\x00\x13\x00\x00\x00gecoscc.adminusers\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\xf0?\x00', codec_options=<CodecOptions at remote 0x7f1ab4ac2208>, fields={'cursor': {'firstBatch': 1, 'nextBatch': 1}})) at ../Python/ceval.c:547


As I understand the code is using "getattr" to ckeck if a dict contains an attribute called "_type_marker", and somehow when Python is formatting the exception finds that the stack has been corrupted.

What can be happening? How can I help to debug this?

Best regards!

> What can be happening? How can I help to debug this?

Try to run your application in the Python Debug Mode: https://docs.python.org/dev/library/devmode.html

The best is if you can run your application with a Python built in debug mode.

Usually, it's a bug in a 3rd party C extension.

See also: https://pythondev.readthedocs.io/debug_tools.html

Thank you Victor for your response.
I follow your advice but the output was an error in a different point of the code and I couldn't see any information about memory allocations.

But, I compiled Python 3.7 from source and modified the "Python/errors.c" code where the problem is detected in the following way:

void
PyErr_SetObject(PyObject *exception, PyObject *value)
{
    PyThreadState *tstate = PyThreadState_GET();
    PyObject *exc_value;
    PyObject *tb = NULL;
    _PyErr_StackItem *exc_info = NULL;


    if (exception != NULL &&
        !PyExceptionClass_Check(exception)) {
        PyErr_Format(PyExc_SystemError,
                     "exception %R not a BaseException subclass",
                     exception);
        return;
    }

    Py_XINCREF(value);
    exc_info = _PyErr_GetTopmostException(tstate);
    exc_value = exc_info->exc_value;
    if (exc_value != NULL && exc_value != Py_None) {
        /* Implicit exception chaining */
        printf("exc_value=%p\n", exc_value);
        printf("exc_info=%p\n", exc_info);
        printf("tstate=%p\n", tstate);
        printf("traceback=%p\n", exc_info->exc_traceback);
        printf("exc_value.ob_type=%p\n", exc_value->ob_type);
        Py_INCREF(exc_value);
       
In this way I had the pointer printed just before the error happening:

exc_value=0x73726573756e69
exc_info=0x7f83bfafacb8
tstate=0x5605dcd41330
traceback=0x6b6f0100
Segmentation fault (core dumped)

And by using gdb I printed the memory contents:

(gdb) x/32c 0x7f83bfafacb8
0x7f83bfafacb8:	111 'o'	115 's'	99 'c'	99 'c'	46 '.'	97 'a'	100 'd'	109 'm'
0x7f83bfafacc0:	105 'i'	110 'n'	117 'u'	115 's'	101 'e'	114 'r'	115 's'	0 '\000'
0x7f83bfafacc8:	0 '\000'	1 '\001'	111 'o'	107 'k'	0 '\000'	0 '\000'	0 '\000'	0 '\000'
0x7f83bfafacd0:	0 '\000'	0 '\000'	0 '\000'	-16 '\360'	63 '?'	0 '\000'	0 '\000'	0 '\000'

If you check the original post you will see:

#15 0x000000000053f732 in PyEval_EvalFrameEx (throwflag=0, 
    f=Frame 0x17882e8, for file /usr/local/lib/python3.7/dist-packages/bson/__init__.py, line 1013, in decode_all (data=b'V\x00\x00\x00\x03cursor\x00=\x00\x00\x00\x04firstBatch\x00\x05\x00\x00\x00\x00\x12id\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02ns\x00\x13\x00\x00\x00gecoscc.adminusers\x00\x00\x01ok\x00\x00\x00\x00\x00\x00\x00\xf0?\x00', codec_options=<CodecOptions at remote 0x7f1ab4ac2208>, view=<memoryview at remote 0x7f1ab5375708>, data_len=86, docs=[], position=0, end=85)) at ../Python/ceval.c:547

So, somehow the "oscc.adminusers\x00\x00\x01ok\x00" part of the message is written over the exc_info memory.

That makes me think that this is a problem in pymongo module. I will report this bug to them.

Thank you very much!

> That makes me think that this is a problem in pymongo module. I will report this bug to them.

Yeah, in 90% of cases, the bug comes from a third party C extensions. That's why Python 3.10 now dumps the list of 3rd party C extensions on a fatal error :-)

https://twitter.com/VictorStinner/status/1374736944022876169

This issue was resolved in https://jira.mongodb.org/browse/PYTHON-2621

The cause of the segfault was determined to be gevent 1.3.4 (2018) and/or greenlet 0.4.13 (2018). When the reporter upgraded to gevent==21.1.2 and greenlet==1.0 the segfault went away.