Issue1720705
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Unsupported provider
Created on 2007-05-17 13:08 by ocean-city, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
bug.py | ocean-city, 2007-05-17 13:08 | The script to reproduce error | ||
bug.zip | ocean-city, 2007-05-17 14:49 | The script to reproduce error (version2) |
Messages (17) | |||
---|---|---|---|
msg32028 - (view) | Author: Hirokazu Yamamoto (ocean-city) * | Date: 2007-05-17 13:08 | |
I got some experience of crash on Tkinter, so I minimized triger code. (I'll attach it as 'bug.py') Same error happens on released python 2.5.1 binary, trunk built with VC6, and release25-maint built with VC6. (Not every time this error occurs, but frequently) # Here is error message. S:\python\tkinter>bug.py Traceback (most recent call last): File "S:\python\tkinter\bug.py", line 13, in <module> raise RuntimeError() RuntimeError Fatal Python error: PyImport_GetModuleDict: no module dictionary! This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information. # Here is stack trace. (Win2000SP4 + VC6) NTDLL! 77f9193c() PyImport_GetModuleDict() line 361 + 10 bytes import_submodule(_object * 0x1e3148b8 __Py_NoneStruct, char * 0x013fe440, char * 0x013fe440) line 2357 + 5 bytes load_next(_object * 0x1e3148b8 __Py_NoneStruct, _object * 0x1e3148b8 __Py_NoneStruct, char * * 0x013fe554, char * 0x013fe440, int * 0x013fe43c) line 2216 + 17 bytes import_module_level(char * 0x00000000, _object * 0x008e6620, _object * 0x1e3148b8 __Py_NoneStruct, _object * 0x1e3148b8 __Py_NoneStruct, int -1) line 1997 + 35 bytes PyImport_ImportModuleLevel(char * 0x009408cc, _object * 0x008e6620, _object * 0x1e3148b8 __Py_NoneStruct, _object * 0x1e3148b8 __Py_NoneStruct, int -1) line 2068 + 25 bytes builtin___import__(_object * 0x00000000, _object * 0x00b0aaa0, _object * 0x00000000) line 48 + 25 bytes PyCFunction_Call(_object * 0x008cf478, _object * 0x00b0aaa0, _object * 0x00000000) line 77 + 15 bytes PyObject_Call(_object * 0x008cf478, _object * 0x00b0aaa0, _object * 0x00000000) line 1860 + 15 bytes PyEval_CallObjectWithKeywords(_object * 0x008cf478, _object * 0x00b0aaa0, _object * 0x00000000) line 3434 PyEval_EvalFrameEx(_frame * 0x00a5d3e8, int 107) line 2065 fast_function(_object * 0x00000000, _object * * * 0x013fef00, int 1, int 1, int 10152296) line 3651 call_function(_object * * * 0x013fef00, int 0) line 3585 + 16 bytes PyEval_EvalFrameEx(_frame * 0x00a706b8, int 131) line 2269 PyEval_EvalCodeEx(PyCodeObject * 0x009e80e8, _object * 0x00a706b8, _object * 0x00000002, _object * * 0x00ac86a0, int 2, _object * * 0x00ac86a8, int 0, _object * * 0x00a1ccfc, int 1, _object * 0x00000000) line 2831 + 11 bytes fast_function(_object * 0x00000001, _object * * * 0x013ff418, int 2, int 2, int 0) line 3663 + 53 bytes call_function(_object * * * 0x013ff418, int 0) line 3585 + 16 bytes PyEval_EvalFrameEx(_frame * 0x00ac8528, int 131) line 2269 PyEval_EvalCodeEx(PyCodeObject * 0x009e4c88, _object * 0x00ac8528, _object * 0x00000002, _object * * 0x00a5b590, int 2, _object * * 0x00a5b598, int 0, _object * * 0x00a1cc1c, int 1, _object * 0x00000000) line 2831 + 11 bytes fast_function(_object * 0x00000001, _object * * * 0x013ff930, int 2, int 2, int 0) line 3663 + 53 bytes call_function(_object * * * 0x013ff930, int 0) line 3585 + 16 bytes PyEval_EvalFrameEx(_frame * 0x00a5b440, int 131) line 2269 PyEval_EvalCodeEx(PyCodeObject * 0x009e44a8, _object * 0x00a5b440, _object * 0x00000001, _object * * 0x00a5b224, int 1, _object * * 0x00a5b228, int 0, _object * * 0x00a1e9cc, int 2, _object * 0x00000000) line 2831 + 11 bytes fast_function(_object * 0x00000002, _object * * * 0x013ffe48, int 1, int 1, int 0) line 3663 + 53 bytes call_function(_object * * * 0x013ffe48, int 0) line 3585 + 16 bytes PyEval_EvalFrameEx(_frame * 0x00a5b0e0, int 131) line 2269 PyEval_EvalCodeEx(PyCodeObject * 0x009d5ce8, _object * 0x00a5b0e0, _object * 0x00000000, _object * * 0x008c104c, int 0, _object * * 0x00000000, int 0, _object * * 0x00000000, int 0, _object * 0x00000000) line 2831 + 11 bytes function_call(_object * 0x00a2b140, _object * 0x008c1038, _object * 0x00000000) line 522 + 64 bytes PyObject_Call(_object * 0x00a2b140, _object * 0x008c1038, _object * 0x00000000) line 1860 + 15 bytes PyEval_CallObjectWithKeywords(_object * 0x00a2b140, _object * 0x008c1038, _object * 0x00000000) line 3434 t_bootstrap(void * 0x008c8508) line 425 + 26 bytes bootstrap(void * 0x0022f8cc) line 179 + 7 bytes _threadstart(void * 0x009bff78) line 187 + 13 bytes KERNEL32! 77e5b396() |
|||
msg32029 - (view) | Author: Hirokazu Yamamoto (ocean-city) * | Date: 2007-05-17 14:49 | |
Sorry, Tkinter and urllib is not source of problem. I changed summary to "thread + import => crashes?". # I'll attach reproducable script as "bug.zip". please run main.py Probabry "import" in another thread causes crash, but I'll investigate more. File Added: bug.zip |
|||
msg32030 - (view) | Author: AndyShorts (andyshorts) | Date: 2007-06-19 23:40 | |
I have had a look at this using the latest trunk code built using VisualStudio 2005 on Windows XP SP2. I modified the source scripts from bug2.zip so that the code was wrapped in a function and added the same code as a function to main.py; I did this so as to see if the difference in calling method would affect it at all, i.e. if by removing the import of sub.py it would make the issue any better or worse. Running it in the debugger and modifying the debug output from the engine to include Win32 thread identifiers seems to suggest that the main thread is exiting and cleaning up the Python system modules and then the Win32 system is closing the other Python threads which are then trying to access the global Python modules and they can't find them and they exit. [Though as a /total/ newbie to Python I am only about 40-50% confident of this] And as to be expected the error does not happen all the time - I can run the same script and maybe get the error as listed maybe about 60-70% of the time. Interestingly in main.c there is a call to a function called "WaitForThreadShutdown" - but sadly it only looks to see if the Python threading module has been imported. I think extending this function to include the Python thread module might be the way to go - but I really don't know and I certainly haven't got to grips with the Python source yet. |
|||
msg60158 - (view) | Author: Christian Heimes (christian.heimes) * | Date: 2008-01-19 13:52 | |
For 2.6 I've fixed several places where C code was importing a Python module in a non-threadsafe way. In general a thread start must never be caused by an import and only the main thread should import modules. Imports within or caused-by a thread other than the main thread isn't safe. |
|||
msg60159 - (view) | Author: Christian Heimes (christian.heimes) * | Date: 2008-01-19 13:55 | |
Follow up: Can somebody please document the possible issue with threads and imports in large, friendly letters? |
|||
msg62842 - (view) | Author: Georg Brandl (georg.brandl) * | Date: 2008-02-23 23:25 | |
Documented in both thread and threading docs in r61029. |
|||
msg62879 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 06:18 | |
Reopening - I disagree with the assertion that this isn't expected to work. |
|||
msg62880 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 07:04 | |
Hmm, I can't reproduce any failure using the supplied zip file, even if I increase the number of threads to 382 (if I set it any higher, start_new_thread starts to fail). That's with both 2.5.1 and 2.6 SVN. I'll try again with the original tkinter/urllib script. |
|||
msg62881 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 07:10 | |
Nope, can't reproduce it with the original tkinter/urllib script either (again, Ubuntu's 2.5.1 release and SVN 2.6, with the thread count set at 350). Could it be a windows specific problem? Or require a non-hyperthreaded CPU? |
|||
msg62882 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 07:15 | |
Ah, never mind - just needed to run the test a few more times to get it to fail. |
|||
msg62884 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 07:40 | |
Ahah, found the real restrictions - while it turns out the OP's demonstration code is indeed broken, I think the warning Georg added to the docs based on Christian's comment above is *far* too strong (hence my somewhat annoyed message on the checkins list ;). The actual restriction is given in Py_Finalize(): all pending imports must be given a chance to finish before the interpreter is torn down. No new imports should be started after that point. The threading module takes care of this for you automatically by registering a system shut down handler that will call the join() method of all non-daemon threads before the interpreter starts to be torn down. As daemon threads aren't included in this, all of their imports must be completed before the thread is switched to daemon mode. With the thread module, a lot more is left up to the caller (one of the main reasons this module is being renamed to _thread in Py3k). You can make the warnings against using *this* module directly as strong as you want - doing so really can get you in big trouble. As for the exact nature of the bug in the OP's demonstration code: it takes no measures to ensure that all imports are complete before the interpreter is shut down (and in fact deliberately goes out of its way to break the restriction). While the approach Christian suggests (perform all your imports from the main thread) is certainly one way to obey the restriction, it is far from the only way (e.g. use the threading module, and have any daemon threads only flag themselves as such once they finish their imports). |
|||
msg62885 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 07:50 | |
Correction to previous comment: you can't safely perform imports from daemon threads at all, as you have to set their daemon status before you start them. So, to be able to safely import from a thread other than the main thread: - it must be created by the threading module, or otherwise guaranteed to be terminated when the interpreter exits - if created through the threading module, it must not be set to daemon mode |
|||
msg62886 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-24 07:55 | |
And one other restriction: the thread performing the import must not be spawned as a side effect of importing a module (as this will deadlock the interpreter) The effect of disobeying the two restrictions in my previous comment will be intermittent crashes at shutdown, such as those experienced by the original poster of this bug report. |
|||
msg63011 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-25 22:47 | |
Proposed wording for a warning pointing that threads and imports can get you in trouble. It might be better to squirrel off into a new subsection on threaded imports, rather than giving it too much prominence (since the recommended approach to creating subthreads should 'just work' - it's only daemon threads and using the thread module directly that can get you in trouble): """While the import machinery is thread safe, there are two key restrictions on threaded imports due to inherent limitations in the way that thread safety is provided. Firstly, other than in the main module, an import should not have the side effect of spawning a new thread and then waiting for that thread in any way. Failing to abide by this restriction can lead to a deadlock if the spawned thread directly or indirectly attempts to import a module. Secondly, all import attempts must be completed before the interpreter starts shutting itself down. This can be most easily achieved by only performing imports from non-daemon threads created through the threading module. Daemon threads and threads created directly with the thread module will require some other form of synchronization to ensure they do not attempt imports after system shutdown has commenced. Failure to abide by this restriction will lead to intermittent exceptions and crashes during interpreter shutdown (as the late imports attempt to access machinery which is no longer in a valid state).""" (If we were going to actually change anything in the code, it would be to ensure that breaking the second restriction could only lead to exceptions rather than crashes - I suspect that would be pretty messy though, as there are a bunch of PyImport_* calls that currently can't fail that would start having to be checked for error returns) |
|||
msg63012 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2008-02-25 22:49 | |
(oh, one thing - the 'crashes' mentioned in my proposed wording are due to explicit calls to PyErr_FatalError rather than a segfault or anything like that. The one specifically reported by the OP is due to sys.modules being gone by the time an import is attempted) |
|||
msg63498 - (view) | Author: Georg Brandl (georg.brandl) * | Date: 2008-03-13 07:22 | |
Added your remarks to threading docs in r61365. |
|||
msg67147 - (view) | Author: Adam Olsen (Rhamphoryncus) | Date: 2008-05-21 05:55 | |
The patch for issue 1856 should fix the potential crash, so we could eliminate that scary blurb from the docs. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:56:24 | admin | set | github: 44963 |
2008-12-12 02:15:54 | forest | set | nosy: + forest |
2008-05-21 05:56:04 | Rhamphoryncus | set | nosy:
+ Rhamphoryncus messages: + msg67147 |
2008-03-13 07:22:43 | georg.brandl | set | status: open -> closed resolution: fixed messages: + msg63498 |
2008-02-25 22:49:22 | ncoghlan | set | messages: + msg63012 |
2008-02-25 22:47:35 | ncoghlan | set | messages: + msg63011 |
2008-02-24 07:55:01 | ncoghlan | set | messages: + msg62886 |
2008-02-24 07:50:21 | ncoghlan | set | messages: + msg62885 |
2008-02-24 07:40:46 | ncoghlan | set | messages: + msg62884 |
2008-02-24 07:15:24 | ncoghlan | set | messages: + msg62882 |
2008-02-24 07:10:31 | ncoghlan | set | messages: + msg62881 |
2008-02-24 07:04:42 | ncoghlan | set | messages: + msg62880 |
2008-02-24 06:18:45 | ncoghlan | set | status: closed -> open nosy: + ncoghlan resolution: fixed -> (no value) messages: + msg62879 |
2008-02-23 23:25:44 | georg.brandl | set | status: open -> closed resolution: fixed messages: + msg62842 nosy: + georg.brandl |
2008-01-19 13:55:13 | christian.heimes | set | messages:
+ msg60159 components: + Documentation, - None |
2008-01-19 13:52:22 | christian.heimes | set | nosy:
+ christian.heimes messages: + msg60158 |
2007-05-17 13:08:39 | ocean-city | create |