classification
Title: corrupted size vs. prev_size
Type: crash Stage:
Components: SSL Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, kannanf9t
Priority: normal Keywords:

Created on 2020-10-27 08:22 by kannanf9t, last changed 2020-10-27 17:44 by kannanf9t.

Messages (12)
msg379743 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 08:22
Getting the below error from long running python processes
----------------------------------------------------------

2020-10-18T21:52:59.383Z ======= Backtrace: =========
2020-10-18T21:52:59.383Z /lib64/libc.so.6(+0x7f3e4)[0x7f7bd3e053e4]
2020-10-18T21:52:59.383Z /lib64/libc.so.6(+0x814db)[0x7f7bd3e074db]
2020-10-18T21:52:59.383Z /lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7f7bd1cfe80d]
2020-10-18T21:52:59.383Z /lib64/libssl.so.10(+0x41aa3)[0x7f7bd2134aa3]
2020-10-18T21:52:59.383Z /lib64/libssl.so.10(SSL_CTX_free+0x147)[0x7f7bd2136977]
2020-10-18T21:52:59.383Z /usr/local/lib/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so(+0xe50b)[0x7f7bcf9d750b]
2020-10-18T21:52:59.383Z /lib64/libpython3.7m.so.1.0(+0x101bab)[0x7f7bd4db1bab]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xe2416)[0x7f7bd4d92416]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0x101ca1)[0x7f7bd4db1ca1]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xca6d9)[0x7f7bd4d7a6d9]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0x1f6255)[0x7f7bd4ea6255]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0x19fede)[0x7f7bd4e4fede]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyObject_GC_Malloc+0x133)[0x7f7bd4e50943]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyObject_GC_New+0xd)[0x7f7bd4e5099d]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xe3fdf)[0x7f7bd4d93fdf]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(PyObject_GetIter+0x16)[0x7f7bd4d62d86]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x133a)[0x7f7bd4e0c55a]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(+0xc562b)[0x7f7bd4d7562b]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0xa16)[0x7f7bd4e0bc36]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0x27e)[0x7f7bd4e1602e]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyFunction_FastCallKeywords+0xda)[0x7f7bd4d6d4aa]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x4f34)[0x7f7bd4e10154]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0x27e)[0x7f7bd4e1602e]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyFunction_FastCallKeywords+0xda)[0x7f7bd4d6d4aa]
2020-10-18T21:52:59.384Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x4f34)[0x7f7bd4e10154]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0xbcdfa)[0x7f7bd4d6cdfa]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x4f34)[0x7f7bd4e10154]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0xbcdfa)[0x7f7bd4d6cdfa]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x79c)[0x7f7bd4e0b9bc]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0xbcdfa)[0x7f7bd4d6cdfa]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyFunction_FastCallDict+0x66)[0x7f7bd4d6cfc6]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyObject_Call_Prepend+0x7b)[0x7f7bd4d6ef0b]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0x107b81)[0x7f7bd4db7b81]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0x1038a2)[0x7f7bd4db38a2]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyObject_FastCallKeywords+0x99)[0x7f7bd4d6dc89]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x52c5)[0x7f7bd4e104e5]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0xbcdfa)[0x7f7bd4d6cdfa]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x4f34)[0x7f7bd4e10154]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(+0xbcdfa)[0x7f7bd4d6cdfa]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyFunction_FastCallDict+0x66)[0x7f7bd4d6cfc6]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x1fd8)[0x7f7bd4e0d1f8]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0x27e)[0x7f7bd4e1602e]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyFunction_FastCallKeywords+0xda)[0x7f7bd4d6d4aa]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x4c4)[0x7f7bd4e0b6e4]
2020-10-18T21:52:59.385Z /lib64/libpython3.7m.so.1.0(_PyEval_EvalCodeWithName+0x27e)[0x7f7bd4e1602e]
msg379744 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 08:24
python process getting killed after this error
msg379745 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 08:27
Implementation details
1). python long running stomp.py client (stomp+ssl)
2). main process thread used to create connection (while true sleep for holding the process)
3). listener thread receives message in another thread
msg379746 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 08:39
Implementation details
1). python long running stomp.py client (stomp+ssl)
2). main process thread used to create connection (while true sleep for holding the process)
3). listener thread receives message and processes it
msg379747 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-27 09:34
Could you please provide detailed information about your platform (OS, CPU, vendor/distribution, Python version, and OpenSSL version), a full stacktrace with debug symbols, and Python code to reproduce the error?

Why is the _ssl module in /usr/local/lib while the rest of your Python installation is in /lib64? Did you compile the module yourself?
msg379775 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 16:46
Thanks for the reply Christian Heimes
Environment details

1). os - centos 7
2). python - 3.7.3

python installed inside docker container and we are using AWS ECS with fargate for docker container deployment
while installing python37 we are copying below files

cp /usr/local/lib/libpython3.7m.so.1.0 /lib64/libpython3.7m.so.1.0
cp /usr/local/lib/python3.7 /usr/local/lib/python3.7
msg379776 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 16:47
we didn't compile the _ssl module
msg379777 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-27 17:01
Why are you copying the file around?

How did you install Python 3.7 in the first place. AFAIK CentOS 7 does not come with Python 3.7.

Python 3.7.3 is old and no longer supported. Can you upgrade to a more recent version of 3.7?
msg379778 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 17:12
This error happens randomly and we couldn't able to reproduce the error.
msg379779 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 17:15
we are installing python using wget
https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz
export LD_LIBRARY_PATH=/usr/local/Python3.7/lib
export PATH="/usr/local/bin:$PATH"
msg379781 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-27 17:33
There isn't much I can do for you. CentOS 7 has OpenSSL 1.0.2. There might be a threading issue in OpenSSL or Python. Both Python 3.7.3 and OpenSSL 1.0.2 are no longer supported. The minimum supported versions are OpenSSL 1.1.1 and Python 3.7.9.  You'd have to update to CentOS 8 to get a more recent OpenSSL version.
msg379785 - (view) Author: kannan (kannanf9t) Date: 2020-10-27 17:44
As you said we are using a threaded consumers in which main thread will create connection and listener thread will receive message.
We are passing main thread object to listener to acknowledge the message using connection object (conn.ack)

in connection class
self.conn = activemq stomp connection
self.listener_obj = ConsumerListener(self)

in listener class
self.conn = conn_obj_from_main_thread
self.conn.ack(msg_id, subscription)
History
Date User Action Args
2020-10-27 17:44:13kannanf9tsetmessages: + msg379785
2020-10-27 17:33:00christian.heimessetmessages: + msg379781
2020-10-27 17:15:04kannanf9tsetmessages: + msg379779
2020-10-27 17:12:45kannanf9tsetmessages: + msg379778
2020-10-27 17:01:06christian.heimessetmessages: + msg379777
2020-10-27 16:47:45kannanf9tsetmessages: + msg379776
2020-10-27 16:46:37kannanf9tsetmessages: + msg379775
2020-10-27 09:34:21christian.heimessetmessages: + msg379747
2020-10-27 08:39:33kannanf9tsetmessages: + msg379746
2020-10-27 08:27:27kannanf9tsetmessages: + msg379745
2020-10-27 08:24:40kannanf9tsetmessages: + msg379744
2020-10-27 08:22:55kannanf9tcreate