classification
Title: ssl: SNI callbacks should not modify context objects
Type: crash Stage:
Components: SSL Versions: Python 3.6, Python 3.5
process
Status: open Resolution: not a bug
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: David Ford (FirefighterBlu3), christian.heimes, haypo
Priority: normal Keywords:

Created on 2017-02-07 06:22 by David Ford (FirefighterBlu3), last changed 2017-02-16 04:41 by David Ford (FirefighterBlu3).

Messages (19)
msg287198 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-07 06:22
(Only 3.[56] tested)

my application listens on an SSL wrapped port, accepts socket and gets an incoming set of messages. connection time frame is several seconds but due to bad Comcast service frequently we have outages which hang connections. almost every day python crashes as follows:

Feb 07 02:36:11 ranger.blue-labs.org python[11816]: *** Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 ***
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: *** Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 ***
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in viola
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: ======= Backtrace: =========
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: ======= Backtrace: =========
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x70c4b)[0x7f62a84cac4b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x76fe6)[0x7f62a84d0fe6]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(+0x777de)[0x7f62a84d17de]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(CRYPTO_free+0x1d)[0x7f62a5ceaa6d]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(BN_clear_free+0x58)[0x7f62a5d4bbc8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(RSA_free+0x6f)[0x7f62a5d99a2f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(EVP_PKEY_free+0x4a)[0x7f62a5dcdcea]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(+0x163780)[0x7f62a5de5780]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x450)[0x7f62a5debf00]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_template_free+0x567)[0x7f62a5dec017]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x5d8e)[0x7f62a8b39e2e]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libcrypto.so.1.0.0(ASN1_item_free+0x1fb)[0x7f62a5dec2fb]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate+0xc4)[0x7f62a4ee3d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(SSL_CTX_use_certificate_chain_file+0x7f)[0x7f62a4ee490f]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xe77d)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyCFunction_Call+0xe9)[0x7f62a8ac4be9]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x78a1)[0x7f62a8b3b941]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_CallFunctionObjArgs+0xc2)[0x7f62a8a7dec2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0x9c46)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl_parse_clienthello_tlsext+0x648)[0x7f62a4ec9838]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_get_client_hello+0x6fa)[0x7f62a4eadeca]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libssl.so.1.0.0(ssl3_accept+0x66f)[0x7f62a4eb27af]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so(+0xc3a9)[0x7f6
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x843c)[0x7f62a8b3c4dc]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x5d8e)[0x7f62a8b39e2e]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc5110)[0x7f62a8ada110]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc3475)[0x7f62a8ad8475]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3d6b)[0x7f62a8b37e0b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x930c5)[0x7f62a8aa80c5]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xee6)[0x7f62a8b34f86]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f62a8b33da7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1629a2)[0x7f62a8b779a2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpthread.so.0(+0x7454)[0x7f62a87ff454]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(clone+0x5f)[0x7f62a85427df]


Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc5110)[0x7f62a8ada110]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0xc3475)[0x7f62a8ad8475]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x3d6b)[0x7f62a8b37e0b]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x930c5)[0x7f62a8aa80c5]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0xee6)[0x7f62a8b34f86]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalFrameEx+0x88ca)[0x7f62a8b3c96a]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1285e3)[0x7f62a8b3d5e3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_EvalCodeEx+0x23)[0x7f62a8b3d6c3]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x92fd8)[0x7f62a8aa7fd8]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x7ed14)[0x7f62a8a93d14]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyObject_Call+0x47)[0x7f62a8a7d2b7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(PyEval_CallObjectWithKeywords+0x47)[0x7f62a8b33da7]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpython3.5m.so.1.0(+0x1629a2)[0x7f62a8b779a2]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libpthread.so.0(+0x7454)[0x7f62a87ff454]
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: /usr/lib/libc.so.6(clone+0x5f)[0x7f62a85427df]


Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-797
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-798
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-794
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-796
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:645)
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO goodbye3 107.170.82.162
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: 2017-02-07 02:36:11 INFO disconnect: Thread-795
Feb 07 02:36:11 ranger.blue-labs.org python[11816]: cutils.cpython-35m-x86_64-linux-gnu.so (deleted)


i've restarted this under valgrind to debug it. valgrind's startup of python 3.6 is depressingly filled with invalid read of size n, conditional jump/move on uninitialized value, use of uninitialized value; mostly in libpython3.6m.
msg287220 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-07 09:47
Python 3.6 got a new PYTHONMALLOC env var which should help you ;-)

Please try to get the Python traceback on the crash and use Python builtin memory debugger:

PYTHONMALLOC=debug python3 -X faulthandler ...


> i've restarted this under valgrind to debug it. valgrind's startup of python 3.6 is depressingly filled with invalid read of size n, (...)

Python memory allocator pymalloc causes false alarms. You can disable pymalloc to use Valgrind using:

PYTHONMALLOC=malloc valgrind python3 ...
msg287260 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-07 22:16
does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto?
msg287262 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-07 22:51
David Ford: "does the builtin memory debugger handle things well such as when involving external libraries like ssl/libcrypto?"

Nope, see:
http://bugs.python.org/issue18227#msg191610

Please try what I suggested. Buffer overflows may not crash immediatly, but PYTHONMALLOC=debug should help to detect some bugs in memory blocks allocated by Python.
msg287263 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-07 22:53
yes, it's running now, hasn't crashed yet
msg287751 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-14 09:48
2017-02-14 06:58:29 INFO connected: mustang.blue-labs.org/107.170.82.162:37188
2017-02-14 06:58:29 INFO No cert required for this host
*** Error in `/usr/bin/python': double free or corruption (out): 0x00007fb0f8003400 ***
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
Fatal Python error: Aborted

Thread 0x00007fb0effff700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 116 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/pytho2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
2017-02-14 06:58:29 WARNING SSL client handshake has failed: EOF occurred in violation of protocol (_ssl.c:749)
n3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb100de2700 (most recent call first):
  File "/var/bluelabs/python/_cams_thread2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 INFO disconnect: Thread-6
2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 INFO disconnect: Thread-5
ed_server.py", line 116 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "2017-02-14 06:58:29 INFO goodbye3 107.170.82.162
2017-02-14 06:58:29 INFO disconnect: Thread-10
/usr/lib/python3.6/socketserver.py", line 361 in 2017-02-14 06:58:29 INFO disconnect: Thread-11
finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb1025e5700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 137 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb102de6700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 137 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Current thread 0x00007fb1035e7700 (most recent call first):
  File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback
  File "/usr/lib/python3.6/ssl.py", line 683 in do_handshake
  File "/usr/lib/python3.6/ssl.py", line 1061 in do_handshake
  File "/var/bluelabs/python/_cams_threaded_server.py", line 152 in handle
  File "/usr/lib/python3.6/socketserver.py", line 696 in __init__
  File "/usr/lib/python3.6/socketserver.py", line 361 in finish_request
  File "/usr/lib/python3.6/socketserver.py", line 639 in process_request_thread
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11131a700 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 299 in wait
  File "/usr/lib/python3.6/threading.py", line 551 in wait
  File "/var/bluelabs/bots/cams/camsbot.py", line 753 in start_message_queue_handler
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb111b1b700 (most recent call first):
  File "/var/bluelabs/python/irclib.py", line 230 in process_once
  File "/var/bluelabs/python/irclib.py", line 246 in process_forever
  File "/var/bluelabs/python/irclib.py", line 1228 in start
  File "/var/bluelabs/python/ircbot.py", line 258 in start
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11231c700 (most recent call first):
  File "/usr/lib/python3.6/selectors.py", line 376 in select
  File "/usr/lib/python3.6/socketserver.py", line 236 in serve_forever
  File "/usr/lib/python3.6/threading.py", line 864 in run
  File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x00007fb11a936400 (most recent call first):
  File "/usr/lib/python3.6/threading.py", line 1072 in _wait_for_tstate_lock
  File "/usr/lib/python3.6/threading.py", line 1056 in join
  File "/usr/lib/python3.6/threading.py", line 1290 in _shutdown
Aborted (core dumped)
msg287761 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-14 10:20
You now have to test "PYTHONMALLOC=malloc valgrind python3 ...".

IMHO it's not a bug in Python, but in your code ussing OpenSSL:

  File "/var/bluelabs/python/_cams_threaded_server.py", line 49 in sni_callback
msg287764 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-02-14 10:39
Please install debug symbols and provide a proper C backtrace of the invalid free call. The backtrace in your first comment is missing both line numbers, file and function names.
msg287765 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-02-14 10:41
Please also provide more information about your system: platform, distro, architecture, OpenSSL version, exact Python version etc.
msg287811 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-15 03:36
in process of creating a non-stripped python install. the default Arch Linux PKGBUILD uses xvfb-run at the end of the build process which is crashing.
msg287812 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-15 03:40
my SNI callback is a function that chooses which SSL certificate to load and summarized is:

      6 def sni_callback(sock, servername, context):
[...]
     44     # load a specific crt/key
     45     store = crtstore[servername]
     46 
     47     context.load_cert_chain( certfile = store['certfile'],
     48                              keyfile  = store['keyfile'],
     49                              password = store['password'] )
msg287813 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-15 03:46
Arch Linux (updated ~weekly but not often rebooted) on a Digital Ocean droplet.

Kernel 4.5.1-1-ARCH, x86_64, OpenSSL 1.0.2j, Python 3.6.0
msg287845 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-02-15 11:44
Do you modify the SSLContext object in your SNI callback? That's the wrong way to do it. You have to create a SSLContext object for each certificate/key pair at startup and then change the socket's context attribute.
msg287848 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-15 12:10
yes, i create a context and then when a request comes in, i load the appropriate key. this is how most [all?] SNI capable python server examples show how to do it, or did, at the time I wrote this module a few years ago. from that time, ~py 3.2 until now, it worked perfectly.

thank you for this information, i'll rewrite my callback accordingly. perhaps 18.2.5.3 in https://docs.python.org/3/library/ssl.html can be updated to inform developers how to change their socket.context in an SNI callback
msg287866 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-15 17:28
> Do you modify the SSLContext object in your SNI callback? That's the wrong way to do it. You have to create a SSLContext object for each certificate/key pair at startup and then change the socket's context attribute.

Would it be possible to prevent bugs? For example, make SSLContext "read only" while the SNI callback is called? Or my question doesn't make sense? :-) I don't know well how OpenSSL works.

At least, we should warn users in the ssl documentation to mention such crash?
msg287875 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-02-15 17:54
It's not a bug in Python's ssl module. If I understand David's approach correctly, then he is using the SNI callback the wrong way. By using it the wrong way he has discovered a threading bug in OpenSSL. There is some kind of race condition going on in which two threads free and replace the RSA private key at the same time.

I'm -1 to try to make the SSLContext object magically read-only.

David, which examples did you read? The documentation https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_servername_callback is pretty clear:

A typical use of this callback is to change the ssl.SSLSocket‘s SSLSocket.context attribute to a new object of type SSLContext representing a certificate chain that matches the server name.

Apache mod_ssl does it correctly, the first hit on stack overflow, too. https://stackoverflow.com/questions/5113333/how-to-implement-server-name-indication-sni
msg287901 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-15 23:36
@Christian, that was years ago and there were few examples available of people that had tried to make an SNI capable server. in several cases, people were monkey patching to make a callback. .set_servername_callback() didn't formally show up in ssl mod until 3.4.

i disagree that the documentation is very clear about this. there's one direct reference to usage which is worded as "A typical use of this callback is to...". that doesn't imply 'very clear' :}

i cannot recall anything anywhere that says one must not modify the context after it is used. 18.2.3 implies that a context is a long lived object, longer lived than a socket. it's clear that several people including myself, have mistakenly tried modifying the context after it was built. until now, it worked, probably purely by coincidence.

as a matter of fact, i used to change more than just the key loaded. i changed the ciphers, options, and verify_flags when i knew the incoming IP was incapable of supporting modern ciphers and options. i also modified the verify paths to support a messed up corporate environment.

i suggest the 18.2.3 documentation be modified to be clear about:
a) if, when, and how a context can be modified after creation
b) an SNI example that changes contexts and explicitly indicates (a)

i don't see any value to not being clear about it, and it's disingenuous to leave nooses hiding in shadows for devs to hang themselves with :-]
msg287906 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-02-16 00:07
David Ford: would you mind to directly propose a doc patch?
msg287913 - (view) Author: David Ford (FirefighterBlu3) (David Ford (FirefighterBlu3)) * Date: 2017-02-16 04:41
yes, after i've modified my tool and ensured i know the correct way of doing it. one of the problems i'm dealing with is not carting around a global variable holding prebuilt contexts. i find it unappealing.
History
Date User Action Args
2017-02-16 04:41:39David Ford (FirefighterBlu3)setmessages: + msg287913
2017-02-16 00:07:17hayposetmessages: + msg287906
2017-02-15 23:36:06David Ford (FirefighterBlu3)setmessages: + msg287901
2017-02-15 17:54:51christian.heimessetmessages: + msg287875
2017-02-15 17:33:07hayposettitle: [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0; ssl SNI callbacks should not modify context objects -> ssl: SNI callbacks should not modify context objects
2017-02-15 17:28:29hayposetmessages: + msg287866
2017-02-15 12:13:35David Ford (FirefighterBlu3)settitle: [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 -> [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0; ssl SNI callbacks should not modify context objects
2017-02-15 12:10:36David Ford (FirefighterBlu3)setresolution: not a bug
messages: + msg287848
2017-02-15 11:44:45christian.heimessetmessages: + msg287845
2017-02-15 03:46:44David Ford (FirefighterBlu3)setmessages: + msg287813
2017-02-15 03:40:53David Ford (FirefighterBlu3)setmessages: + msg287812
2017-02-15 03:36:47David Ford (FirefighterBlu3)setmessages: + msg287811
2017-02-14 10:41:20christian.heimessetmessages: + msg287765
2017-02-14 10:39:44christian.heimessetmessages: + msg287764
2017-02-14 10:20:21hayposetmessages: + msg287761
2017-02-14 09:48:49David Ford (FirefighterBlu3)setmessages: + msg287751
2017-02-07 22:53:36David Ford (FirefighterBlu3)setmessages: + msg287263
2017-02-07 22:51:09hayposetmessages: + msg287262
2017-02-07 22:16:02David Ford (FirefighterBlu3)setmessages: + msg287260
2017-02-07 09:47:15hayposetnosy: + haypo
messages: + msg287220
2017-02-07 07:49:54David Ford (FirefighterBlu3)settitle: Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0 -> [python] Error in `/usr/bin/python': free(): invalid size: 0x00007f628400d0e0
2017-02-07 06:22:31David Ford (FirefighterBlu3)create