classification
Title: _pickle doesn't handle recursion limits properly
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, amaury.forgeotdarc, belopolsky, georg.brandl, pitrou, terry.reedy
Priority: normal Keywords: patch

Created on 2011-01-23 06:06 by terry.reedy, last changed 2011-01-23 17:26 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
frl_pickle1.py terry.reedy, 2011-01-23 06:06 Shows variant behavior
picklerec.patch pitrou, 2011-01-23 15:32
Messages (4)
msg126874 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-01-23 06:06
Tool/scripts/find_recursionlimit.py includes test_cpickle() which, like the other test_xxx functions, is supposed to raise a RuntimeError when the recursion limit is reached. It appears to work correctly on 3.1 and I presume previously. On 3.2, test_cpickle() hangs. Here is much reduced code that shows the behavior:

import itertools
import io
import _pickle

# extracted from 'def test_cpickle' and condensed:
l = None
for n in itertools.count():
    try:
        raise KeyError
    except KeyError:
        for i in range(100):
            l = [l]
        print(n,i)
    _pickle.Pickler(io.BytesIO(), protocol=-1).dump(l)

The added print line prints 0,99 1,99, ... indefinitely. If the recursive list l is added to the print function, the attempt to create repr(l) raises a runtime error at n = 9. If we remove the try-except part:

l = None
for n in itertools.count():
    for i in range(100):
        l = [l]
    print(n,i)
    _pickle.Pickler(io.BytesIO(), protocol=-1).dump(l)

*pickle* now raises a RuntimeError, as expected in the original context, at n=4!

1. I do not actually know which behavior is buggy. I suppose the next step would be to capture and not toss the pickle output to see what is the difference.

2. At least for the present, I think the call to test_cpickle should be commented out in find_recursionlimit.py.

There seems to be other pickle and recursive structure issues, like #9269, but I did not see any the same as this.
msg126875 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-01-23 06:24
3.2rc1 on WinXP. I got hanging behavior with both interpreter and IDLE.
msg126891 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-23 15:32
Here is a patch. I also strength the recursion limit testing script by testing both recursion through dicts and through lists.
msg126894 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-01-23 17:26
Committed in r88147 (3.2) and r88148 (3.1).
History
Date User Action Args
2011-01-23 17:26:43pitrousetstatus: open -> closed
nosy: georg.brandl, terry.reedy, amaury.forgeotdarc, belopolsky, pitrou, alexandre.vassalotti
messages: + msg126894

resolution: fixed
stage: patch review -> resolved
2011-01-23 15:32:24pitrousetfiles: + picklerec.patch


keywords: + patch
stage: patch review
title: Bizarre pickle -- exception interaction bug -> _pickle doesn't handle recursion limits properly
nosy: + alexandre.vassalotti, amaury.forgeotdarc, georg.brandl
versions: + Python 3.1
messages: + msg126891
components: + Extension Modules
type: behavior -> crash
2011-01-23 06:24:29terry.reedysetmessages: + msg126875
2011-01-23 06:06:07terry.reedycreate