Issue23838
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2015-04-01 11:24 by serhiy.storchaka, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
linecache_memoryerror.patch | serhiy.storchaka, 2015-04-01 11:24 | review | ||
linecache_memoryerror_2.patch | serhiy.storchaka, 2015-04-01 12:45 | review | ||
linecache_memoryerror_3.patch | serhiy.storchaka, 2015-04-01 13:23 | review |
Messages (13) | |||
---|---|---|---|
msg239781 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-04-01 11:24 | |
When format a traceback of MemoryError, linecache tries to read all lines of source file and likely raises a MemoryError. For example: http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/9712/steps/test/logs/stdio Traceback (most recent call last): File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1280, in runtest_inner test_runner() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5340, in test_main support.run_unittest(*tests) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1808, in run_unittest _run_suite(suite) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1774, in _run_suite result = runner.run(suite) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 176, in run test(result) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run test(result) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run test(result) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 625, in __call__ return self.run(*args, **kwds) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 585, in run self._feedErrorsToResult(result, outcome.errors) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 518, in _feedErrorsToResult result.addError(test, exc_info) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 67, in addError super(TextTestResult, self).addError(test, err) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 17, in inner return method(self, *args, **kw) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 115, in addError self.errors.append((test, self._exc_info_to_string(err, test))) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 186, in _exc_info_to_string exctype, value, tb, limit=length, capture_locals=self.tb_locals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__ capture_locals=capture_locals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract f.line File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line self._line = linecache.getline(self.filename, self.lineno).strip() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline lines = getlines(filename, module_globals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines return updatecache(filename, module_globals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache lines = fp.readlines() MemoryError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 532, in main result = runtest(*args, **kwargs) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 967, in runtest display_failure=False) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1304, in runtest_inner msg = traceback.format_exc() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 161, in format_exc return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain)) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 115, in format_exception type(value), value, tb, limit=limit).format(chain=chain)) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__ capture_locals=capture_locals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract f.line File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line self._line = linecache.getline(self.filename, self.lineno).strip() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline lines = getlines(filename, module_globals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines return updatecache(filename, module_globals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache lines = fp.readlines() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/codecs.py", line 321, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) MemoryError Traceback (most recent call last): File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 170, in _run_module_as_main "__main__", mod_spec) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/__main__.py", line 3, in <module> regrtest.main_in_temp_cwd() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1564, in main_in_temp_cwd main() File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 738, in main raise Exception("Child error on {}: {}".format(test, result[1])) Exception: Child error on test_socket: make: *** [buildbottest] Error 1 program finished with exit code 2 Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read). I hesitate how to classify this issue. On one hand, it looks as new feature, on other hand it is related to testing on all versions. |
|||
msg239785 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-04-01 11:47 | |
> Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read). IMO it's a bad idea. You should not retry an operation on MemoryError but only do you best to limit errors. I don't think that it's correct to return an empty list for getlines() on MemoryError. So you maybe chose the wrong module: you should handle MemoryError in the traceback module, not in the linecache module. It's just fine to return an empty string for FrameSummary.line. It's common to not being able to display a line: because the .py was removed (only the .pyc is available), invalid line number, invalid filename ("<string>" or "<stdin>"), etc. If you are motivated to make the traceback module more reliable, you should maybe also enhance _Py_DisplaySourceLine() in Python/traceback.c. You can probably ignore almost all errors in _Py_DisplaySourceLine() ("except (Exception, MemoryError): pass"). |
|||
msg239788 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-04-01 11:56 | |
Serhiy: you look to be interested in fixing Python to not fail badly under low memory condition. You may take a look at #19817 which proposes a function to arbitrary limit the memory allocated to Python. I fixed a lot of issues found by failmalloc, but they are still serious issues. For example, unit tests crash on subinterpreter tests with failmalloc, and there is also the memory allocation failure when raising MemoryError: see #19835. |
|||
msg239790 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-04-01 12:02 | |
getlines() already returns an empty list on OSError. Clearing the cache on MemoryError will help to avoid other MemoryErrors. And after this there is a large chance the repeated reading will be successful. In any case it wouldn't make worse. If don't change linecache, then we should patch all places where linecache is used: traceback, idlelib, warnings, trace, debugger, etc. I intentionally made the fix at the low level, so it fixes all issues in one place. |
|||
msg239792 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-04-01 12:08 | |
2015-04-01 14:02 GMT+02:00 Serhiy Storchaka <report@bugs.python.org>: > getlines() already returns an empty list on OSError. Oh, I misread getlines(), you're right. > Clearing the cache on MemoryError will help to avoid other MemoryErrors. And after this there is a large chance I agree, but please avoid retrying on MemoryError. IMO clearing the cache and return [] is enough. > If don't change linecache, then we should patch all places where linecache is used: traceback, idlelib, warnings, trace, debugger, etc. I intentionally made the fix at the low level, so it fixes all issues in one place. I agree. |
|||
msg239798 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-04-01 12:45 | |
OK, here is a patch that just clear the cache and returns an empty result. It is hard to write reliable test, but I think we will see the result on OpenIndiana buildbot. The question is left: should the patch be applied only on 3.5 or all versions? |
|||
msg239800 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-04-01 13:04 | |
> It is hard to write reliable test Ah? Why not using something like mock.patch('mock.updatecache', side_effect=MemoryError)? |
|||
msg239801 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-04-01 13:07 | |
> The question is left: should the patch be applied only on 3.5 or all versions? linecache_memoryerror_2.patch looks good to me, but it would be better with an unit test. IMO Python 2.7 and 3.4 can be fixed too. |
|||
msg239802 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-04-01 13:08 | |
This test depends on implementation details. I prefer to avoid adding such tests for trivial changes. |
|||
msg239804 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-04-01 13:23 | |
On other side, linecache tests already use updatecache(). Here is a patch with a test. |
|||
msg239805 - (view) | Author: STINNER Victor (vstinner) * ![]() |
Date: 2015-04-01 13:28 | |
linecache_memoryerror_3.patch reviewed (comments on Rietveld). |
|||
msg239812 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2015-04-01 13:59 | |
New changeset d444496e714a by Serhiy Storchaka in branch '2.7': Issue #23838: linecache now clears the cache and returns an empty result on https://hg.python.org/cpython/rev/d444496e714a New changeset 88a0e6cd93c3 by Serhiy Storchaka in branch '3.4': Issue #23838: linecache now clears the cache and returns an empty result on https://hg.python.org/cpython/rev/88a0e6cd93c3 New changeset ab2326c801c2 by Serhiy Storchaka in branch 'default': Issue #23838: linecache now clears the cache and returns an empty result on https://hg.python.org/cpython/rev/ab2326c801c2 |
|||
msg239814 - (view) | Author: Serhiy Storchaka (serhiy.storchaka) * ![]() |
Date: 2015-04-01 14:01 | |
Thank you for your review Victor. Will look on OpenIndiana buildbot. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:14 | admin | set | github: 68026 |
2015-04-01 14:01:34 | serhiy.storchaka | set | status: open -> closed messages: + msg239814 assignee: serhiy.storchaka resolution: fixed stage: patch review -> resolved |
2015-04-01 13:59:10 | python-dev | set | nosy:
+ python-dev messages: + msg239812 |
2015-04-01 13:28:33 | vstinner | set | messages: + msg239805 |
2015-04-01 13:23:11 | serhiy.storchaka | set | files:
+ linecache_memoryerror_3.patch messages: + msg239804 |
2015-04-01 13:09:51 | serhiy.storchaka | set | type: resource usage versions: + Python 2.7, Python 3.4, Python 3.5 |
2015-04-01 13:08:58 | serhiy.storchaka | set | messages: + msg239802 |
2015-04-01 13:07:51 | vstinner | set | messages: + msg239801 |
2015-04-01 13:04:21 | vstinner | set | messages: + msg239800 |
2015-04-01 12:45:56 | serhiy.storchaka | set | files:
+ linecache_memoryerror_2.patch messages: + msg239798 |
2015-04-01 12:08:27 | vstinner | set | messages: + msg239792 |
2015-04-01 12:02:28 | serhiy.storchaka | set | messages: + msg239790 |
2015-04-01 11:56:04 | vstinner | set | messages: + msg239788 |
2015-04-01 11:47:52 | vstinner | set | messages: + msg239785 |
2015-04-01 11:24:24 | serhiy.storchaka | create |