(Only 3.[56] tested)

my application listens on an SSL wrapped port, accepts socket and gets an incoming set of messages. connection time frame is several seconds but due to bad Comcast service frequently we have outages which hang connections. almost every day python crashes as follows:

Feb 07 02:36:11 ranger.blue-labs.org python[11816]: *** Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 ***
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: *** Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 ***
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in viola
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: ======= Backtrace: =========
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: ======= Backtrace: =========
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x5d8e)[0x7f62a8b39e2e]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x5d8e)[0x7f62a8b39e2e]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc5110)[0x7f62a8ada110]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc3475)[0x7f62a8ad8475]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3d6b)[0x7f62a8b37e0b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x930c5)[0x7f62a8aa80c5]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xee6)[0x7f62a8b34f86]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f62a8b33da7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1629a2)[0x7f62a8b779a2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpthread.so.0(+0x7454)[0x7f62a87ff454]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(clone+0x5f)[0x7f62a85427df]


Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc5110)[0x7f62a8ada110]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc3475)[0x7f62a8ad8475]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3d6b)[0x7f62a8b37e0b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x930c5)[0x7f62a8aa80c5]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xee6)[0x7f62a8b34f86]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f62a8b33da7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1629a2)[0x7f62a8b779a2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpthread.so.0(+0x7454)[0x7f62a87ff454]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(clone+0x5f)[0x7f62a85427df]


Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-797
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-798
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-794
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-796
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-795
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: cutils.cpython-35m-x86_64-linux-gnu.so (deleted)


i've restarted this under valgrind to debug it. valgrind's startup of python 3.6 is depressingly filled with invalid read of size n, conditional jump/move on uninitialized value, use of uninitialized value; mostly in libpython3.6m.

Python 3.6 got a new PYTHONMALLOC env var which should help you ;-)

Please try to get the Python traceback on the crash and use Python builtin memory debugger:

PYTHONMALLOC=debug python3 -X faulthandler ...


> i've restarted this under valgrind to debug it. valgrind's startup of python 3.6 is depressingly filled with invalid read of size n, (...)

Python memory allocator pymalloc causes false alarms. You can disable pymalloc to use Valgrind using:

PYTHONMALLOC=malloc valgrind python3 ...

does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto?

David Ford: "does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto?"

Nope, see:
http://bugs.python.org/issue18227#msg191610

Please try what I suggested. Buffer overflows may not crash immediatly, but PYTHONMALLOC=debug should help to detect some bugs in memory blocks allocated by Python.

yes, it's running now, hasn't crashed yet

2017-02-14 06:58:29 INFO connected: mustang.blue-labs.org/107.170.82.162:37188
2017-02-14 06:58:29 INFO No cert required for this host
*** Error in `/usr/bin/python': double free or corruption (out): 0x00007fb0f8003400 ***
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
Fatal Python error: Aborted

Thread 0x00007fb0effff700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 116 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/pytho2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
n3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb100de2700 (most recent call first):
  File "/var/bluelabs/python/_cams_thread2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 INFO disconnect: Thread-6
2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 INFO disconnect: Thread-5
ed_server.py", line 116 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 INFO disconnect: Thread-10
/usr/lib/python3.6/socketserver.py", line 361 in 2017-02-14 06:58:29 INFO disconnect: Thread-11
finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb1025e5700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 137 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb102de6700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 137 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fb1035e7700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback
  File "/usr/lib/python3.6/ssl.py", line 683 in do_handshake
  File "/usr/lib/python3.6/ssl.py", line 1061 in do_handshake
  File "/var/bluelabs/python/_cams_threaded_server.py", line 152 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11131a700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 299 in wait
  File "/usr/lib/python3.6/threading.py", line 551 in wait
  File "/var/bluelabs/bots/cams/camsbot.py", line 753 in start_message_queue_handler
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb111b1b700 (most recent call first):
  File "/var/bluelabs/python/irclib.py", line 230 in process_once
  File "/var/bluelabs/python/irclib.py", line 246 in process_forever
  File "/var/bluelabs/python/irclib.py", line 1228 in start
  File "/var/bluelabs/python/ircbot.py", line 258 in start
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11231c700 (most recent call first):
  File "/usr/lib/python3.6/selectors.py", line 376 in select
  File "/usr/lib/python3.6/socketserver.py", line 236 in serve_forever
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11a936400 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 1072 in _wait_for_tstate_lock
  File "/usr/lib/python3.6/threading.py", line 1056 in join
  File "/usr/lib/python3.6/threading.py", line 1290 in _shutdown
Aborted (core dumped)

You now have to test "PYTHONMALLOC=malloc valgrind python3 ...".

IMHO it's not a bug in Python, but in your code ussing OpenSSL:

  File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback

Please install debug symbols and provide a proper C backtrace of the invalid free call. The backtrace in your first comment is missing both line numbers, file and function names.

Please also provide more information about your system: platform, distro, architecture, OpenSSL version, exact Python version etc.

in process of creating a non-stripped python install. the default Arch Linux PKGBUILD uses xvfb-run at the end of the build process which is crashing.

my SNI callback is a function that chooses which SSL certificate to load and summarized is:

      6 def sni_callback(sock, servername, context):
[...]
     44     # load a specific crt/key
     45     store = crtstore[servername]
     46 
     47     context.load_cert_chain( certfile = store['certfile'],
     48                              keyfile  = store['keyfile'],
     49                              password = store['password'] )

Arch Linux (updated ~weekly but not often rebooted) on a Digital Ocean droplet.

Kernel 4.5.1-1-ARCH, x86_64, OpenSSL 1.0.2j, Python 3.6.0

Do you modify the SSLContext object in your SNI callback? That's the wrong way to do it. You have to create a SSLContext object for each certificate/key pair at startup and then change the socket's context attribute.

yes, i create a context and then when a request comes in, i load the appropriate key. this is how most [all?] SNI capable python server examples show how to do it, or did, at the time I wrote this module a few years ago. from that time, ~py 3.2 until now, it worked perfectly.

thank you for this information, i'll rewrite my callback accordingly. perhaps 18.2.5.3 in https://docs.python.org/3/library/ssl.html can be updated to inform developers how to change their socket.context in an SNI callback

> Do you modify the SSLContext object in your SNI callback? That's the wrong way to do it. You have to create a SSLContext object for each certificate/key pair at startup and then change the socket's context attribute.

Would it be possible to prevent bugs? For example, make SSLContext "read only" while the SNI callback is called? Or my question doesn't make sense? :-) I don't know well how OpenSSL works.

At least, we should warn users in the ssl documentation to mention such crash?

It's not a bug in Python's ssl module. If I understand David's approach correctly, then he is using the SNI callback the wrong way. By using it the wrong way he has discovered a threading bug in OpenSSL. There is some kind of race condition going on in which two threads free and replace the RSA private key at the same time.

I'm -1 to try to make the SSLContext object magically read-only.

David, which examples did you read? The documentation https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_servername_callback is pretty clear:

A typical use of this callback is to change the ssl.SSLSocket‘s SSLSocket.context attribute to a new object of type SSLContext representing a certificate chain that matches the server name.

Apache mod_ssl does it correctly, the first hit on stack overflow, too. https://stackoverflow.com/questions/5113333/how-to-implement-server-name-indication-sni

@Christian, that was years ago and there were few examples available of people that had tried to make an SNI capable server. in several cases, people were monkey patching to make a callback. .set_servername_callback() didn't formally show up in ssl mod until 3.4.

i disagree that the documentation is very clear about this. there's one direct reference to usage which is worded as "A typical use of this callback is to...". that doesn't imply 'very clear' :}

i cannot recall anything anywhere that says one must not modify the context after it is used. 18.2.3 implies that a context is a long lived object, longer lived than a socket. it's clear that several people including myself, have mistakenly tried modifying the context after it was built. until now, it worked, probably purely by coincidence.

as a matter of fact, i used to change more than just the key loaded. i changed the ciphers, options, and verify_flags when i knew the incoming IP was incapable of supporting modern ciphers and options. i also modified the verify paths to support a messed up corporate environment.

i suggest the 18.2.3 documentation be modified to be clear about:
a) if, when, and how a context can be modified after creation
b) an SNI example that changes contexts and explicitly indicates (a)

i don't see any value to not being clear about it, and it's disingenuous to leave nooses hiding in shadows for devs to hang themselves with :-]

David Ford: would you mind to directly propose a doc patch?

yes, after i've modified my tool and ensured i know the correct way of doing it. one of the problems i'm dealing with is not carting around a global variable holding prebuilt contexts. i find it unappealing.

OK, let's update the documentation with a better example. Are you still interested to contribute a simple example?