classification
Title: Core dump in f-string with formatting errors
Type: crash Stage: resolved
Components: Interpreter Core Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: eric.smith, martin.panter, petr.viktorin, python-dev
Priority: normal Keywords:

Created on 2016-02-05 01:21 by petr.viktorin, last changed 2016-02-05 23:24 by eric.smith. This issue is now closed.

Messages (8)
msg259609 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2016-02-05 01:21
Evaluating the expression f"{(lambda: 0):x}" crashes Python.

$ ./python
Python 3.6.0a0 (default, Feb  5 2016, 02:14:48) 
[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f"{(lambda: 0):x}"  
Fatal Python error: Python/ceval.c:3576 object at 0x7f6b42f21338 has negative ref count -2604246222170760230
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
Aborted (core dumped)
msg259613 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-05 01:39
I had to recompile with “--with-pydebug” to get the crash. I know f-strings don’t support the lambda syntax very well, but I can also make it crash without using lambda:

>>> f"{ {1: 2}:x}"
Fatal Python error: Python/ceval.c:3576 object at 0x7fa32ab030c8 has negative ref count -1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
Aborted (core dumped)
msg259665 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-02-05 15:14
The problem has to do with refcounting when an error occurs. Adjusting the title accordingly.

I'm not sure yet if the problem is in PyObject_Format(), or in handling errors in the eval loop when processing FORMAT_VALUE opcodes. I'm slowly tracking it down. It doesn't happen with format(), so I suspect it's in FORMAT_VALUE.

Here's an example showing the error with a large, non-cached int. You need to call it twice to trigger the refcount problem.

>>> f'{1000:j}'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: Unknown format code 'j' for object of type 'int'
>>> f'{1000:j}'
Fatal Python error: Objects/tupleobject.c:233 object at 0x7f904006b360 has negative ref count -2604246222170760230

Current thread 0x00007f9041278700 (most recent call first):
Aborted (core dumped)
msg259689 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-05 20:39
I have zero experience with with ceval.c but I have a theory after looking over revision 1ddeb2e175df. I suspect “value” is a borrowed reference from TOP(), but at <https://hg.python.org/cpython/diff/1ddeb2e175df/Python/ceval.c#l1.35> and <https://hg.python.org/cpython/diff/1ddeb2e175df/Python/ceval.c#l1.54> you are calling DECREF on it without calling SET_TOP() if there is an error.

I suspected FORMAT_VALUE, because the crash doesn’t happen when you call format() directly.
msg259697 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-02-05 22:41
Yes, that's my thinking as well. I'll have a patch soon-ish.
msg259698 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-02-05 22:55
Now I think it's not a refcount problem.

I'm going to switch to POP/PUSH instead of TOP/SET_TOP. I believe the problem is that I'm not adjusting the top of stack if there's an error.
msg259700 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-02-05 23:23
New changeset 9095a5787a82 by Eric V. Smith in branch 'default':
Fix issue 26287: While handling FORMAT_VALUE opcode, the top of stack was being corrupted if an error occurred in PyObject_Format().
https://hg.python.org/cpython/rev/9095a5787a82
msg259701 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2016-02-05 23:24
Thanks for the report and the debugging help.
History
Date User Action Args
2016-02-05 23:24:01eric.smithsetstatus: open -> closed
resolution: fixed
messages: + msg259701

stage: resolved
2016-02-05 23:23:15python-devsetnosy: + python-dev
messages: + msg259700
2016-02-05 22:55:04eric.smithsetmessages: + msg259698
title: Core dump in f-string with formatting errors due to refcount bug -> Core dump in f-string with formatting errors
2016-02-05 22:41:15eric.smithsetmessages: + msg259697
2016-02-05 20:39:59martin.pantersetmessages: + msg259689
2016-02-05 15:14:44eric.smithsetmessages: + msg259665
title: Core dump in f-string with lambda and format specification -> Core dump in f-string with formatting errors due to refcount bug
2016-02-05 09:33:10eric.smithsetassignee: eric.smith
2016-02-05 01:40:00martin.pantersetnosy: + martin.panter
messages: + msg259613

components: + Interpreter Core
type: crash
2016-02-05 01:21:25petr.viktorincreate