classification
Title: Abort in PyErr_PrintEx() when no memory
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: serhiy.storchaka, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2017-06-30 14:19 by xdegaye, last changed 2017-10-24 14:44 by xdegaye. This issue is now closed.

Files
File name Uploaded Description Edit
memerr.py xdegaye, 2017-06-30 14:19
pyerr_printex.patch xdegaye, 2017-06-30 14:21
set-last-exception.diff serhiy.storchaka, 2017-07-02 05:21
Pull Requests
URL Status Linked Edit
PR 2526 merged xdegaye, 2017-07-01 13:43
PR 4107 merged xdegaye, 2017-10-24 14:10
Messages (9)
msg297408 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-06-30 14:19
To reproduce the abort (the set_nomemory() function is being added to _testcapi in issue 30695):

$ ./python memerr.py
python: Objects/call.c:89: _PyObject_FastCallDict: Assertion `!PyErr_Occurred()' failed.
Aborted (core dumped)

The corresponding backtrace:

#0  0x00007ffff7131670 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff7132d00 in abort () from /usr/lib/libc.so.6
#2  0x00007ffff712a45a in __assert_fail_base () from /usr/lib/libc.so.6
#3  0x00007ffff712a4d2 in __assert_fail () from /usr/lib/libc.so.6
#4  0x00000000004628ce in _PyObject_FastCallDict (
    callable=<built-in method excepthook of module object at remote 0x7ffff70ab8d8>,
    args=args@entry=0x7fffffffe330, nargs=nargs@entry=3, kwargs=kwargs@entry=0x0)
    at Objects/call.c:89
#5  0x0000000000426004 in PyErr_PrintEx (set_sys_last_vars=set_sys_last_vars@entry=1)
    at Python/pythonrun.c:641
#6  0x00000000004263ac in PyErr_Print () at Python/pythonrun.c:510
#7  0x0000000000427586 in PyRun_SimpleFileExFlags (fp=fp@entry=0x987620, filename=<optimized out>,
    filename@entry=0x7ffff6febcd0 "memerr.py", closeit=closeit@entry=1,
    flags=flags@entry=0x7fffffffe4cc) at Python/pythonrun.c:403
#8  0x0000000000427673 in PyRun_AnyFileExFlags (fp=fp@entry=0x987620,
    filename=0x7ffff6febcd0 "memerr.py", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe4cc)
    at Python/pythonrun.c:82
#9  0x0000000000437217 in run_file (fp=fp@entry=0x987620, filename=0x91a190 L"memerr.py",
    p_cf=p_cf@entry=0x7fffffffe4cc) at Modules/main.c:341
#10 0x0000000000437bae in Py_Main (argc=argc@entry=2, argv=argv@entry=0x918020)
    at Modules/main.c:901
#11 0x000000000041d7fa in main (argc=2, argv=0x7fffffffe6c8) at ./Programs/python.c:102
msg297411 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-06-30 14:21
With pyerr_printex.patch we get the following correct results (rc=120 is set by Py_Main() after Py_FinalizeEx() returns with an error):

$ ./python memerr.py 5 12
set_nomemory(0, 5)
result = _PythonRunResult(rc=1, out=b'', err=b'MemoryError\n\nDuring handling of the above exception, another exception occurred:\n\nMemoryError')
=============================
set_nomemory(0, 6)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nMemoryError\n\nDuring handling of the above exception, another exception occurred:\n\nMemoryError')
=============================
set_nomemory(0, 7)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nMemoryError\n\nDuring handling of the above exception, another exception occurred:\n\nMemoryError')
=============================
set_nomemory(0, 8)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : MemoryError()\ntype    : MemoryError\nrefcount: 1\naddress : 0x7f7ae1be3eb0\nlost sys.stderr')
=============================
set_nomemory(0, 9)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7f8aead15eb0\nlost sys.stderr')
=============================
set_nomemory(0, 10)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7f7b489d1eb0\nlost sys.stderr')
=============================
set_nomemory(0, 11)
result = _PythonRunResult(rc=1, out=b'', err=b'sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7ff97365eeb0\nlost sys.stderr')
=============================
set_nomemory(0, 12)
result = _PythonRunResult(rc=120, out=b'', err=b"sys.excepthook is missing\nobject  : \ntype    : MemoryError\nrefcount: 1\naddress : 0x7f559c9e0eb0\nlost sys.stderr\nException ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>\nMemoryError")
=============================
msg297414 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-30 14:51
pyerr_printex.patch LGTM, please convert it to a GitHub PR.
msg297483 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-01 17:05
Is it good to ignore errors of setting sys attributes? Can we prevent these errors?
msg297495 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-07-01 19:58
There are just 6 cases left where the return code of _PySys_SetObjectId() is ignored:

Python/pylifecycle.c|689 col 5| _PySys_SetObjectId(&PyId_stderr, pstderr);
Python/pylifecycle.c|1211 col 9| _PySys_SetObjectId(&PyId_stderr, pstderr);
Python/pylifecycle.c|1657 col 5| _PySys_SetObjectId(&PyId_stdin, std);
Python/pylifecycle.c|1666 col 5| _PySys_SetObjectId(&PyId_stdout, std);
./Python/pythonrun.c|104 col 9| _PySys_SetObjectId(&PyId_ps1, v = PyUnicode_FromString(">>> "));
./Python/pythonrun.c|109 col 9| _PySys_SetObjectId(&PyId_ps2, v = PyUnicode_FromString("... "));

Error should be handled there in the same way errors are handled within the functions where they are invoked (returning -1, NULL or aborting). This should be the purpose of another issue as it is acceptable I think that there is no test case for those changes.

I am leaving the day after tomorrow for few weeks, without internet access (sailing to the mid-atlantic), and will create then a new issue if no one else does that first.
msg297508 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-02 05:21
I tried to set last_type, last_value and last_traceback to None at startup, and this also fixes a crash. But this changes behavior (these attributes became set), and deleting them doesn't guaranties that following setting will be successful, especially with new compact dict implementation.
msg304809 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-10-23 15:03
Removing 2.7 as the problem cannot be reproduced here. PyEval_CallObjectWithKeywords() does not assert on PyErr_Occurred().
msg304818 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-10-23 16:08
New changeset 66caacf2f0d6213b049a3097556e28e30440b900 by xdegaye in branch 'master':
bpo-30817: Fix PyErr_PrintEx() when no memory (#2526)
https://github.com/python/cpython/commit/66caacf2f0d6213b049a3097556e28e30440b900
msg304913 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-10-24 14:42
New changeset d5d79545b73110b2f4c2b66d150409514e2ca8e0 by xdegaye in branch '3.6':
[3.6] bpo-30817: Fix PyErr_PrintEx() when no memory (GH-2526). (#4107)
https://github.com/python/cpython/commit/d5d79545b73110b2f4c2b66d150409514e2ca8e0
History
Date User Action Args
2017-10-24 14:44:13xdegayesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-10-24 14:42:38xdegayesetmessages: + msg304913
2017-10-24 14:10:46xdegayesetpull_requests: + pull_request4077
2017-10-23 16:08:43xdegayesetmessages: + msg304818
2017-10-23 15:03:35xdegayesetmessages: + msg304809
versions: - Python 2.7
2017-10-23 13:26:26xdegayesetstage: patch review
versions: - Python 3.5
2017-07-02 05:21:22serhiy.storchakasetfiles: + set-last-exception.diff

messages: + msg297508
2017-07-01 19:58:34xdegayesetmessages: + msg297495
2017-07-01 17:05:47serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg297483
2017-07-01 13:43:57xdegayesetpull_requests: + pull_request2591
2017-06-30 14:51:16vstinnersetnosy: + vstinner
messages: + msg297414
2017-06-30 14:21:16xdegayesetfiles: + pyerr_printex.patch
keywords: + patch
messages: + msg297411
2017-06-30 14:19:31xdegayecreate