Created on 2007-07-19 18:17 by taleinat, last changed 2012-07-09 21:12 by roger.serwy. This issue is now closed.
| Messages (8) | |||
|---|---|---|---|
| msg32530 - (view) | Author: Tal Einat (taleinat)  |
Date: 2007-07-19 18:17 | |
This bug manifests only when running with a subprocess. Trying to display an instance of BeautifulSoup's NavigableString class, this is the result: RuntimeError: maximum recursion depth exceeded See http://mail.python.org/pipermail/idle-dev/2007-July/002600.html for details (including how to recreate the error). Diagnosis: The problem arises when trying to pickle such instances - pickle enters an endless loop and reaches the max recursion limit (eventually). This happens regardless of the protocol used. IDLE is probably trying to pickle them because their class, NavigableString, inherits from unicode, so isinstance(<NavigableString instance>, basestring) return True. Possibly related to SF bug #1581183: "pickle protocol 2 failure on int subclass" http://sourceforge.net/tracker/index.php?funchttp://sourceforge.net/tracker/index.php?func=detail&aid=1581183&group_id=5470&atid=105470=detail&aid=1512695&group_id=5470&atid=105470 Fix: IDLE should check for any exception when trying to pickle, not just pickle.PicklingError, to avoid such errors. If pickle doesn't work, for whatever reason, IDLE can still work around it with str() and repr(). I'll post a bug report for Pickle as well. |
|||
| msg84318 - (view) | Author: Guilherme Polo (gpolo) * |
Date: 2009-03-28 18:06 | |
I can't seem to reproduce this here. I've tried both python 2.4.5 and 2.5.2 using beautifulsoup 3.0.7. I also used the sample html attached in that email as well part of the code that is supposed to cause the problem, and I can also run pickle on the body's contents without getting a RuntimeError. Any chance you can retry it ? And if you do reproduce it then include the instructions here instead of pointing to a mail list, please. |
|||
| msg84412 - (view) | Author: Tal Einat (taleinat) |
Date: 2009-03-29 15:59 | |
To recreate use BeautifulSoup 3.0.4 and run the following:
>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup("<html>aa</html")
>>> x = soup.find('html').contents[0]
>>> x
u'aa'
>>> print x
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
print x
RuntimeError: maximum recursion depth exceeded
This is caused by a bug in BeautifulSoup which was fixed in version
3.0.5. The bug manifests when trying to pickle an instance of the
NavigableString class.
In the above scenario, IDLE has the subprocess pickle the object and
send it to the parent process. Since the problem is with the pickling,
turning the object into a string in the subprocess (instead of sending
it as-is to the parent process) avoids generating the error:
>>> print str(x)
aa
>>> print repr(x)
u'aa'
To verify that pickle is the culprit:
>>> import pickle
>>> pickle.dumps(x)
(very long traceback...)
RuntimeError: maximum recursion depth exceeded
Like I said in my first post, IMO IDLE should check for any exception
(not just pickle.PicklingError) when trying to pickle an object for
sending to the parent process. If pickle doesn't work, for whatever
reason, IDLE can still try to work around it with str() and/or repr().
(I tried this with Python 2.5 but I've tested this in the past with 2.6
as well. I haven't tried it with 3.0 or 2.7 yet.)
|
|||
| msg84425 - (view) | Author: Guilherme Polo (gpolo) * |
Date: 2009-03-29 19:49 | |
> IMO IDLE should check for any exception > (not just pickle.PicklingError) when trying to pickle an object for > sending to the parent process. If pickle doesn't work, for whatever > reason, IDLE can still try to work around it with str() and/or repr(). > Do you have some specific suggestion on how to do so ? As I'm seeing it rpc.SocketIO.putmessage would return a custom exception when a PicklingError doesn't happen, which then PyShell.ModifiedInterpreter.runcode could handle and decide to run the code object locally. Sounds unpleasant. |
|||
| msg84436 - (view) | Author: Tal Einat (taleinat) |
Date: 2009-03-29 22:01 | |
Sending a code object back to the parent process and having it deal with the situation sounds very unpleasant indeed! I think a completely different type of solution may be possible. In general, I can't think of any reason for IDLE to pickle "user objects" from the subprocess and send them to the parent process; it should merely send back the output (as strings), with special cases for exceptions and such. By "user objects" I mean objects "inside the interpreter", as opposed to those used by IDLE itself. I'll have to unwind the spaghetti in rpc.py, run.py and PyShell.py a bit more to propose a specific set of changes; I hope to get to that tomorrow. |
|||
| msg116705 - (view) | Author: Mark Lawrence (BreamoreBoy) | Date: 2010-09-17 18:31 | |
Can we close this given "This is caused by a bug in BeautifulSoup which was fixed in version 3.0.5." from msg84412? |
|||
| msg116715 - (view) | Author: Ned Deily (ned.deily) * |
Date: 2010-09-17 19:32 | |
While BeautifulSoup may have been fixed, the issue here still points to an underlying problem in IDLE being vulnerable to pickling errors. The given test case still fails in 2.7 (I didn't try to construct a test for Python 3). |
|||
| msg165126 - (view) | Author: Roger Serwy (roger.serwy) * |
Date: 2012-07-09 21:12 | |
I think that issue13532 fixes this issue, as only strings are now pickled from the subprocess to the IDLE front-end for writing to stdout and stderr. This should address the pickling vulnerability. I downloaded BeautifulSoup 3.0.4 and did not see this problem when using IDLE 2.7.1, and then against the latest dev version from the repo. I am closing this issue as being out of date. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2012-07-09 21:12:50 | roger.serwy | set | status: open -> closed nosy: + roger.serwy messages: + msg165126 resolution: out of date |
| 2012-07-09 04:43:32 | terry.reedy | set | nosy:
+ terry.reedy |
| 2010-09-17 19:32:29 | ned.deily | set | status: pending -> open title: IDLE + BeautifulSoup = Error -> Unexpected "maximum recursion depth exceeded" in IDLE shell with objects that cannot be pickled nosy: + kbk, ned.deily versions: + Python 2.7 messages: + msg116715 resolution: invalid -> (no value) |
| 2010-09-17 18:31:19 | BreamoreBoy | set | status: open -> pending nosy: + BreamoreBoy messages: + msg116705 resolution: invalid |
| 2009-03-29 22:01:33 | taleinat | set | messages: + msg84436 |
| 2009-03-29 19:49:27 | gpolo | set | messages: + msg84425 |
| 2009-03-29 15:59:21 | taleinat | set | messages: + msg84412 |
| 2009-03-28 18:06:41 | gpolo | set | nosy:
+ gpolo messages: + msg84318 |
| 2007-07-19 18:17:37 | taleinat | create | |