New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fatal error on thread creation in low memory condition #51793
Comments
Using my fuzzer (Fusil) on Python trunk, I got sometimes errors on Fatal Python error: PyEval_AcquireThread: NULL new thread state I'm sorry but I don't have example script to reproduce the bug. I How can I get more information on the bug? How can I reproduce it? |
Enable core dumps ( You can also try a debug build of Python which may give you more (as for how to reproduce the bug, it's your task to find it out :-)) |
Use a core dump: good idea! haypo> Using my fuzzer (Fusil) on Python trunk, I got sometimes I read the source code of the thread module. This error means that Said differently: in low memory condition, creating a new thread may exit the -- Sometimes, I get another error, similar to the "NULL new thread state" error: Fatal Python error: Couldn't create autoTLSkey mapping I guess that the reason is the same: memory allocation failed. It should be |
We just had this error on one of the buildbots: [...] I don't know it's related. |
No, I don't think that this issue is related, because it starts with |
Memory can be pre-allocated by thread_PyThread_start_new_thread() before |
Here I come with a patch! nirai idea was the good one: prealloc PyThreadState before creating the thread. Raise a MemoryError if the allocation fail, instead of raising a *fatal* Python error. Patch is quite simple and allow better error handling. |
The patch looks good; the line |
I'm not sure about autoTLSkey value. PyThreadState_New() calls _PyGILState_NoteThreadState() which checks that autoTLSkey is not zero, but I don't know if autoTLSkey have the right value with my preallocation patch. |
After some tests (printf fun), it looks ok. -- There is another problem in _testcapi: test_thread_state() calls directly PyThread_start_new_thread() and the thread function calls PyGILState_Ensure(). PyGILState_Ensure() would then require to create a new thread state (call PyThreadState_New()) because it was not done by PyThread_start_new_thread(). On low memory condition, we hit the same bug here. Only thread and _testcapi modules calls directly PyThread_start_new_thread(). _*test*capi is reserved to tests, so I consider that we don't care about bugs under low memory condition in this module. |
Running the tests in debug mode gives the following error: test_3_join_in_forked_from_thread (test.test_threading.ThreadJoinOnShutdown) ... Fatal Python error: Invalid thread state for this thread ====================================================================== Traceback (most recent call last):
File "/home/antoine/cpython/debug/Lib/test/test_threading.py", line 476, in test_3_join_in_forked_from_thread
self._run_and_join(script)
File "/home/antoine/cpython/debug/Lib/test/test_threading.py", line 412, in _run_and_join
self.assertEqual(data, "end of main\nend of thread\n")
AssertionError: 'end of main\n' != 'end of main\nend of thread\n' Ran 78 tests in 3.739s |
I tried all Lib/test/test_thread*py, but not in debug mode :-/ The problem is here: PyThreadState_New() -> _PyGILState_NoteThreadState() -> PyThread_set_key_value() -> find_key() and find_key() calls PyThread_get_thread_ident(), but the ident is not the right ident :-/ PyThreadState_New() should not call _PyGILState_NoteThreadState(), it should be done _in_ the thread. I wrote a new patch fixing the unit test. PyThreadState_New() is part of the public API, so I can't change its prototype. And _PyGILState_NoteThreadState() is a private function (use the "static" keyword). Because of that, I introduced two new functions:
I tried all Lib/test/test_thread*.py tests in debug mode and all test now pass ;-) |
Another problem with my patch! If initsite() calls PyGILState_Ensure(): assert(autoInterpreterState) fails because autoInterpreterState is NULL. _PyGILState_Init() have to be called before initsite(). I don't know where it should be called exactly. Why not just after the PyThreadState_New() (in Py_InitializeEx())? |
Note: I was using gdb to track a bug on a debug build (--with-pydebug). I used "pyo" macro which calls _PyObject_Dump(), and _PyObject_Dump() calls PyGILState_Ensure() => assertion error. |
Sum up of my patch:
|
I didn't test the patch but some comments:
|
Oh, and besides, you can use the -R option to regrtest to find out if there are any reference leaks (e.g. "-R 3:2:"). |
done
done
I'm unable to reproduce the bug related to this patch. I keep the patch somewhere in my disk, and I will open a new issue if I'm able to reproduce it :-)
I tried all test*thread*py. test_threadsignals leaks a reference... even without my patch, so I opened a new issue: bpo-7825. |
Commited: r78638 (trunk), r78639 (py3k), r78640 (3.1). Keep the issue open to remember me that I have to backport it to 2.6 (the branch is frozen waiting for the 2.6.5 final version). |
Backport done: r79199 (2.6). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: