Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

linecache and MemoryError #68026

Closed
serhiy-storchaka opened this issue Apr 1, 2015 · 13 comments
Closed

linecache and MemoryError #68026

serhiy-storchaka opened this issue Apr 1, 2015 · 13 comments
Assignees
@serhiy-storchaka
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@serhiy-storchaka
Copy link
Member

BPO 23838
Nosy @pitrou, @vstinner, @ezio-melotti, @voidspace, @serhiy-storchaka
Files
  • linecache_memoryerror.patch
  • linecache_memoryerror_2.patch
  • linecache_memoryerror_3.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-04-01.14:01:34.267>
created_at = <Date 2015-04-01.11:24:24.066>
labels = ['library', 'performance']
title = 'linecache and MemoryError'
updated_at = <Date 2015-04-01.14:01:34.266>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2015-04-01.14:01:34.266>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-04-01.14:01:34.267>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-04-01.11:24:24.066>
creator = 'serhiy.storchaka'
dependencies = []
files = ['38779', '38782', '38783']
hgrepos = []
issue_num = 23838
keywords = ['patch']
message_count = 13.0
messages = ['239781', '239785', '239788', '239790', '239792', '239798', '239800', '239801', '239802', '239804', '239805', '239812', '239814']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'ezio.melotti', 'michael.foord', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue23838'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']
    @serhiy-storchaka
    Copy link
    Member Author

    When format a traceback of MemoryError, linecache tries to read all lines of source file and likely raises a MemoryError. For example:

    http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/9712/steps/test/logs/stdio

    Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1280, in runtest_inner
    test_runner()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5340, in test_main
    support.run_unittest(*tests)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1808, in run_unittest
    _run_suite(suite)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1774, in _run_suite
    result = runner.run(suite)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 176, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 585, in run
    self._feedErrorsToResult(result, outcome.errors)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 518, in _feedErrorsToResult
    result.addError(test, exc_info)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 67, in addError
    super(TextTestResult, self).addError(test, err)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 17, in inner
    return method(self, *args, **kw)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 115, in addError
    self.errors.append((test, self._exc_info_to_string(err, test)))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 186, in _exc_info_to_string
    exctype, value, tb, limit=length, capture_locals=self.tb_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__
    capture_locals=capture_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract
    f.line
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines
    return updatecache(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache
    lines = fp.readlines()
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 532, in main
    result = runtest(*args, **kwargs)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 967, in runtest
    display_failure=False)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1304, in runtest_inner
    msg = traceback.format_exc()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 161, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 115, in format_exception
    type(value), value, tb, limit=limit).format(chain=chain))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__
    capture_locals=capture_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract
    f.line
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines
    return updatecache(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache
    lines = fp.readlines()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
MemoryError
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1564, in main_in_temp_cwd
    main()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_socket: 
make: *** [buildbottest] Error 1
program finished with exit code 2

    Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read).

    I hesitate how to classify this issue. On one hand, it looks as new feature, on other hand it is related to testing on all versions.

    @serhiy-storchaka serhiy-storchaka added the stdlib Python modules in the Lib dir label Apr 1, 2015
    @vstinner
    Copy link
    Member

    vstinner commented Apr 1, 2015

    Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read).

    IMO it's a bad idea. You should not retry an operation on MemoryError but only do you best to limit errors.

    I don't think that it's correct to return an empty list for getlines() on MemoryError. So you maybe chose the wrong module: you should handle MemoryError in the traceback module, not in the linecache module.

    It's just fine to return an empty string for FrameSummary.line. It's common to not being able to display a line: because the .py was removed (only the .pyc is available), invalid line number, invalid filename ("<string>" or "<stdin>"), etc.

    If you are motivated to make the traceback module more reliable, you should maybe also enhance _Py_DisplaySourceLine() in Python/traceback.c. You can probably ignore almost all errors in _Py_DisplaySourceLine() ("except (Exception, MemoryError): pass").

    @vstinner
    Copy link
    Member

    vstinner commented Apr 1, 2015

    Serhiy: you look to be interested in fixing Python to not fail badly under low memory condition. You may take a look at bpo-19817 which proposes a function to arbitrary limit the memory allocated to Python. I fixed a lot of issues found by failmalloc, but they are still serious issues. For example, unit tests crash on subinterpreter tests with failmalloc, and there is also the memory allocation failure when raising MemoryError: see bpo-19835.

    @serhiy-storchaka
    Copy link
    Member Author

    getlines() already returns an empty list on OSError. Clearing the cache on MemoryError will help to avoid other MemoryErrors. And after this there is a large chance the repeated reading will be successful. In any case it wouldn't make worse.

    If don't change linecache, then we should patch all places where linecache is used: traceback, idlelib, warnings, trace, debugger, etc. I intentionally made the fix at the low level, so it fixes all issues in one place.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 1, 2015

    2015-04-01 14:02 GMT+02:00 Serhiy Storchaka <report@bugs.python.org>:

    getlines() already returns an empty list on OSError.

    Oh, I misread getlines(), you're right.

    Clearing the cache on MemoryError will help to avoid other MemoryErrors. And after this there is a large chance

    I agree, but please avoid retrying on MemoryError. IMO clearing the
    cache and return [] is enough.

    If don't change linecache, then we should patch all places where linecache is used: traceback, idlelib, warnings, trace, debugger, etc. I intentionally made the fix at the low level, so it fixes all issues in one place.

    I agree.

    @serhiy-storchaka
    Copy link
    Member Author

    OK, here is a patch that just clear the cache and returns an empty result.

    It is hard to write reliable test, but I think we will see the result on OpenIndiana buildbot.

    The question is left: should the patch be applied only on 3.5 or all versions?

    @vstinner
    Copy link
    Member

    vstinner commented Apr 1, 2015

    It is hard to write reliable test

    Ah? Why not using something like mock.patch('mock.updatecache', side_effect=MemoryError)?

    @vstinner
    Copy link
    Member

    vstinner commented Apr 1, 2015

    The question is left: should the patch be applied only on 3.5 or all versions?

    linecache_memoryerror_2.patch looks good to me, but it would be better with an unit test.

    IMO Python 2.7 and 3.4 can be fixed too.

    @serhiy-storchaka
    Copy link
    Member Author

    This test depends on implementation details. I prefer to avoid adding such tests for trivial changes.

    @serhiy-storchaka serhiy-storchaka added the performance Performance or resource usage label Apr 1, 2015
    @serhiy-storchaka
    Copy link
    Member Author

    On other side, linecache tests already use updatecache(). Here is a patch with a test.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 1, 2015

    linecache_memoryerror_3.patch reviewed (comments on Rietveld).

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 1, 2015

    New changeset d444496e714a by Serhiy Storchaka in branch '2.7':
    Issue bpo-23838: linecache now clears the cache and returns an empty result on
    https://hg.python.org/cpython/rev/d444496e714a

    New changeset 88a0e6cd93c3 by Serhiy Storchaka in branch '3.4':
    Issue bpo-23838: linecache now clears the cache and returns an empty result on
    https://hg.python.org/cpython/rev/88a0e6cd93c3

    New changeset ab2326c801c2 by Serhiy Storchaka in branch 'default':
    Issue bpo-23838: linecache now clears the cache and returns an empty result on
    https://hg.python.org/cpython/rev/ab2326c801c2

    @serhiy-storchaka
    Copy link
    Member Author

    Thank you for your review Victor. Will look on OpenIndiana buildbot.

    @serhiy-storchaka serhiy-storchaka self-assigned this Apr 1, 2015
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @serhiy-storchaka serhiy-storchaka

    Labels
    performance Performance or resource usage stdlib Python modules in the Lib dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @serhiy-storchaka