Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected "maximum recursion depth exceeded" in IDLE shell with objects that cannot be pickled #45221

Closed
taleinat opened this issue Jul 19, 2007 · 8 comments
Closed

Unexpected "maximum recursion depth exceeded" in IDLE shell with objects that cannot be pickled #45221

taleinat opened this issue Jul 19, 2007 · 8 comments
Labels
topic-IDLE

Comments

@taleinat
Copy link
Contributor

BPO 1757057
Nosy @terryjreedy, @kbkaiser, @taleinat, @ned-deily, @serwy

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-07-09.21:12:50.969>
created_at = <Date 2007-07-19.18:17:37.000>
labels = ['expert-IDLE']
title = 'Unexpected "maximum recursion depth exceeded" in IDLE shell with objects that cannot be pickled'
updated_at = <Date 2012-07-09.21:12:50.918>
user = 'https://github.com/taleinat'

bugs.python.org fields:

activity = <Date 2012-07-09.21:12:50.918>
actor = 'roger.serwy'
assignee = 'none'
closed = True
closed_date = <Date 2012-07-09.21:12:50.969>
closer = 'roger.serwy'
components = ['IDLE']
creation = <Date 2007-07-19.18:17:37.000>
creator = 'taleinat'
dependencies = []
files = []
hgrepos = []
issue_num = 1757057
keywords = []
message_count = 8.0
messages = ['32530', '84318', '84412', '84425', '84436', '116705', '116715', '165126']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'kbk', 'taleinat', 'gpolo', 'ned.deily', 'roger.serwy', 'BreamoreBoy']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1757057'
versions = ['Python 2.7']
@taleinat
Copy link
Contributor Author

This bug manifests only when running with a subprocess.

Trying to display an instance of BeautifulSoup's NavigableString class, this is the result:

RuntimeError: maximum recursion depth exceeded

See http://mail.python.org/pipermail/idle-dev/2007-July/002600.html for details (including how to recreate the error).

Diagnosis: The problem arises when trying to pickle such instances - pickle enters an endless loop and reaches the max recursion limit (eventually). This happens regardless of the protocol used.

IDLE is probably trying to pickle them because their class, NavigableString, inherits from unicode, so isinstance(<NavigableString instance>, basestring) return True.

Possibly related to SF bug bpo-1581183: "pickle protocol 2 failure on int subclass"
http://sourceforge.net/tracker/index.php?funchttp://sourceforge.net/tracker/index.php?func=detail&aid=1581183&group_id=5470&atid=105470=detail&aid=1512695&group_id=5470&atid=105470

Fix: IDLE should check for any exception when trying to pickle, not just pickle.PicklingError, to avoid such errors. If pickle doesn't work, for whatever reason, IDLE can still work around it with str() and repr().

I'll post a bug report for Pickle as well.

@gpolo
Copy link
Mannequin

gpolo mannequin commented Mar 28, 2009

I can't seem to reproduce this here. I've tried both python 2.4.5 and
2.5.2 using beautifulsoup 3.0.7. I also used the sample html attached in
that email as well part of the code that is supposed to cause the
problem, and I can also run pickle on the body's contents without
getting a RuntimeError.

Any chance you can retry it ? And if you do reproduce it then include
the instructions here instead of pointing to a mail list, please.

@taleinat
Copy link
Contributor Author

To recreate use BeautifulSoup 3.0.4 and run the following:

>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup("<html>aa</html")
>>> x = soup.find('html').contents[0]
>>> x
u'aa'
>>> print x

Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    print x
RuntimeError: maximum recursion depth exceeded

This is caused by a bug in BeautifulSoup which was fixed in version
3.0.5. The bug manifests when trying to pickle an instance of the
NavigableString class.

In the above scenario, IDLE has the subprocess pickle the object and
send it to the parent process. Since the problem is with the pickling,
turning the object into a string in the subprocess (instead of sending
it as-is to the parent process) avoids generating the error:

>>> print str(x)
aa
>>> print repr(x)
u'aa'


To verify that pickle is the culprit:
>>> import pickle
>>> pickle.dumps(x)
(very long traceback...)
RuntimeError: maximum recursion depth exceeded

Like I said in my first post, IMO IDLE should check for any exception
(not just pickle.PicklingError) when trying to pickle an object for
sending to the parent process. If pickle doesn't work, for whatever
reason, IDLE can still try to work around it with str() and/or repr().

(I tried this with Python 2.5 but I've tested this in the past with 2.6
as well. I haven't tried it with 3.0 or 2.7 yet.)

@gpolo
Copy link
Mannequin

gpolo mannequin commented Mar 29, 2009

IMO IDLE should check for any exception
(not just pickle.PicklingError) when trying to pickle an object for
sending to the parent process. If pickle doesn't work, for whatever
reason, IDLE can still try to work around it with str() and/or repr().

Do you have some specific suggestion on how to do so ? As I'm seeing
it rpc.SocketIO.putmessage would return a custom exception when a
PicklingError doesn't happen, which then
PyShell.ModifiedInterpreter.runcode could handle and decide to run the
code object locally. Sounds unpleasant.

@taleinat
Copy link
Contributor Author

Sending a code object back to the parent process and having it deal with
the situation sounds very unpleasant indeed! I think a completely
different type of solution may be possible.

In general, I can't think of any reason for IDLE to pickle "user
objects" from the subprocess and send them to the parent process; it
should merely send back the output (as strings), with special cases for
exceptions and such. By "user objects" I mean objects "inside the
interpreter", as opposed to those used by IDLE itself.

I'll have to unwind the spaghetti in rpc.py, run.py and PyShell.py a bit
more to propose a specific set of changes; I hope to get to that tomorrow.

@BreamoreBoy
Copy link
Mannequin

BreamoreBoy mannequin commented Sep 17, 2010

Can we close this given "This is caused by a bug in BeautifulSoup which was fixed in version 3.0.5." from msg84412?

@BreamoreBoy BreamoreBoy mannequin added invalid labels Sep 17, 2010
@ned-deily
Copy link
Member

While BeautifulSoup may have been fixed, the issue here still points to an underlying problem in IDLE being vulnerable to pickling errors. The given test case still fails in 2.7 (I didn't try to construct a test for Python 3).

@ned-deily ned-deily removed the invalid label Sep 17, 2010
@ned-deily ned-deily changed the title IDLE + BeautifulSoup = Error Unexpected "maximum recursion depth exceeded" in IDLE shell with objects that cannot be pickled Sep 17, 2010
@ned-deily ned-deily removed the invalid label Sep 17, 2010
@ned-deily ned-deily changed the title IDLE + BeautifulSoup = Error Unexpected "maximum recursion depth exceeded" in IDLE shell with objects that cannot be pickled Sep 17, 2010
@serwy
Copy link
Mannequin

serwy mannequin commented Jul 9, 2012

I think that bpo-13532 fixes this issue, as only strings are now pickled from the subprocess to the IDLE front-end for writing to stdout and stderr. This should address the pickling vulnerability.

I downloaded BeautifulSoup 3.0.4 and did not see this problem when using IDLE 2.7.1, and then against the latest dev version from the repo.

I am closing this issue as being out of date.

@serwy serwy mannequin closed this as completed Jul 9, 2012
@serwy serwy mannequin closed this as completed Jul 9, 2012
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic-IDLE
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@taleinat @ned-deily