New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FileIO destructor imports indirectly the io module at exit #63620
Comments
Since Python 3.3, when an unclosed file is destroyed, a warning is emitted. Good. The problem is when the warning is emitted after the io has been unloaded: in this case, the warning will try to reload the io module to display the Python line where the warning occurred. _Py_DisplaySourceLine() starts with: io = PyImport_ImportModuleNoBlock("io"); For example, I got the issue while the warning was emitted during PyImport_Cleanup(), see the full gdb backtrace below (the assertion failure was caused by a local modification, it's unrelated to this issue). show_warning() and/or _Py_DisplaySourceLine() should not try to display the Python line while Python is exiting. There is for example the flag _Py_Finalizing. If we really want to display the Python line/traceback, the io module should be kept alive until all warnings are emitted. But I don't think that it is possible. Or Python should try to unload other modules before the io module? (gdb) where |
By the way, the filename of the warning is not really useful. Example script test.py: f = open(__file__) At exit, Python displays: "sys" is not a real file, _Py_DisplaySourceLine() would not be able to display the Python line (the correct filename is test.py). Python traceback when the warning is emitted, according to gdb: (gdb) py-bt
Traceback (most recent call first):
<built-in method _dealloc_warn of _io.FileIO object at remote 0x7ffff1a04368>
<built-in method _dealloc_warn of _io.BufferedReader object at remote 0x7ffff1ae4838>
<built-in method close of _io.TextIOWrapper object at remote 0x7ffff1b97dc0> We are in PyImport_Cleanup(), called by Py_Finalize(). |
"show_warning() and/or _Py_DisplaySourceLine() should not try to display the Python line while Python is exiting. There is for example the flag _Py_Finalizing." Or PyImport_Import() should fail with an ImportError("block importing 'io' when python is exiting") (or a better message) when Python is exiting (if _Py_Finalizing is 1). |
Can you explain what the actual cause of the crash is? |
I modified locally my Python source code (I worked on a patch for PyFrame_LocalsToFast(), issue bpo-18408), the crash was a bug in my code, it's unrelated to the issue. I opened an issue because I'm surprised that Python tries to import a module which was just unloaded, during its finalization procedure. |
Ok, then I don't think it's really a bug. We might want to define import rules at shutdown but "fixing" this particular case doesn't sound important. |
New changeset 1bbedfb20932 by Victor Stinner in branch 'default': |
Here is a patch to not import again the io module in _Py_DisplaySourceLine() if the module was unloaded. A simpler patch is to just return (don't check sys.modules) if _Py_Finalizing is set. Note: I also created issue bpo-19424 to avoid conversion from/to UTF-8. |
Without this fix, the following script crashs with an assertion error (Objects/typeobject.c:740: type_call: Assertion `!PyErr_Occurred()' failed.): import warnings
warn = warnings.warn
class A:
def __del__(self):
warn("bla")
a=A() |
Again, I don't think this is a good patch. If you want to disable module imports during shutdown, it should be done inside the import machinery, not duplicated at every call site. |
New changeset 2d802765d31f by Victor Stinner in branch 'default': |
"Again, I don't think this is a good patch. If you want to disable module imports during shutdown, it should be done inside the import machinery, Ah sorry, I misunderstood your previous comment. I agree that traceback_finalize.patch is very specific. Here is a new patch "import_finalization.patch", which block importing new modules (and import again unloaded modules) during finalization. The full test suite pass with the patch, so at least no unit test tries to import a module during finalization.
I'm opening a thread on python-dev to discuss this tricky question. |
+ PyErr_SetString(PyExc_ImportError, "__import__ blocked Python finalization"); Oops, I forgot "during" word before Python finalization. "import" is maybe less surprising than "__import__". |
I misunderstood the gdb traceback. Display a warning does not reload the io module: in fact, the io module was unloaded and PyImport_ImportModuleLevelObject() already raises an ImportError in this case: ImportError("import of 'io' halted; None in sys.modules"). So nothing else need to be done, I'm now closing the issue. I created more specific issues about warnings, see issues bpo-19442 and bpo-19466. |
For the record, the python-dev thread about blocking imports is at <https://mail.python.org/pipermail/python-dev/2013-October/129907.html\>. It sounded like the conclusion was that there shouldn’t be a problem importing a module at finalization. I am investigating bpo-21049, and finding sys.meta_path is set to None at finalization, which is not handled well by the import code. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: