classification
Title: copy of an asyncore dispatcher causes infinite recursion
Type: behavior Stage: test needed
Components: Extension Modules Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: giampaolo.rodola Nosy List: amaury.forgeotdarc, giampaolo.rodola, josiahcarlson, stutzbach, vstinner, xdegaye
Priority: normal Keywords: patch

Created on 2011-10-04 20:41 by xdegaye, last changed 2014-06-27 21:08 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
infinite_recursion_asyncore.patch xdegaye, 2011-10-05 11:16
Messages (10)
msg144925 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2011-10-04 20:41
A regression occurs in python 3.2 when doing a copy of an asyncore
dispatcher.

$ python3.1
Python 3.1.2 (r312:79147, Apr  4 2010, 17:46:48) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore, copy
>>> copy.copy(asyncore.dispatcher())
<asyncore.dispatcher at 0x7fcfb3590e90>


$ python3.2
Python 3.2 (r32:88445, Jun 18 2011, 20:30:18) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore, copy
>>> copy.copy(asyncore.dispatcher())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/copy.py", line 97, in copy
    return _reconstruct(x, rv, 0)
  File "/usr/local/lib/python3.2/copy.py", line 291, in _reconstruct
    if hasattr(y, '__setstate__'):
  File "/usr/local/lib/python3.2/asyncore.py", line 410, in __getattr__
    retattr = getattr(self.socket, attr)
  ....
  File "/usr/local/lib/python3.2/asyncore.py", line 410, in __getattr__
    retattr = getattr(self.socket, attr)
  File "/usr/local/lib/python3.2/asyncore.py", line 410, in __getattr__
    retattr = getattr(self.socket, attr)
RuntimeError: maximum recursion depth exceeded while calling a Python object


This occurs after the 'copy' module has created the new instance with
__new__(). This new instance does not have the 'socket' attribute,
hence the infinite recursion.

Adding the following methods to the dispatcher class, fixes the infinite
recursion:

    def __getstate__(self):
        state = self.__dict__.copy()
        return state

    def __setstate__(self, state):
        self.__dict__.update(state)

But it does not explain why the recursion occurs in 3.2 and not in
3.1.
msg144948 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2011-10-05 11:16
The infinite recursion occurs also when running python 3.2 with the
extension modules copy, copyreg and asyncore from python 3.1. So it
seems this regression is not caused by a modification in these
modules.

Anyway, the bug is in asyncore. The attached patch fixes it and is
more robust than adding the __getstate__ and __setstate__ methods to
dispatcher.

The patch includes a test case.
msg144954 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2011-10-05 16:21
About why the asyncore bug shows up in python 3.2:

The simple test below is ok with python 3.1 but triggers a
"RuntimeError: maximum recursion depth exceeded..." with python 3.2:

$ python3.1
Python 3.1.2 (r312:79147, Apr  4 2010, 17:46:48) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class C:
...   def __getattr__(self, attr):
...     return getattr(self.foo, attr)
... 
>>> c = C()
>>> hasattr(c, 'bar')
False
>>> 

For the reasoning behind this change made in python 3.2, see issue
9666 and the mail
http://mail.python.org/pipermail/python-dev/2010-August/103178.html
msg144962 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-05 17:20
So, in 3.1 hasattr(y, '__setstate__') *did* recurse and hit the limit, but the exception was caught and hasattr returned False?

I think I prefer the new behavior...
The patch looks good, I would simply have raised AttributeError(name) though.
msg144964 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2011-10-05 17:32
IMO, patch should only be applied to Python 3.2.
For 3.3 we finally have the chance to get rid of the dispatcher.__getattr__ aberration (see issue 8483) so I say let's just remove it and fix this issue as a consequence.
msg144969 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-10-05 19:55
Let's add the test to 3.3 nonetheless.
msg144970 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2011-10-05 21:00
> So, in 3.1 hasattr(y, '__setstate__') *did* recurse and hit the limit,
> but the exception was caught and hasattr returned False?

This is right.


> I think I prefer the new behavior...
> The patch looks good, I would simply have raised AttributeError(name)
> though.

It is fine with me to raise AttributeError(name).
Note that when raising AttributeError('socket'), the user gets
notified of the exceptions on both 'socket' and 'name'.
For example with the patch applied:

$ python3
Python 3.2 (r32:88445, Jun 18 2011, 20:30:18) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore
>>> a = asyncore.dispatcher()
>>> del a.socket
>>> a.foo
Traceback (most recent call last):
  File "asyncore.py", line 415, in __getattr__
    retattr = getattr(self.socket, attr)
  File "asyncore.py", line 413, in __getattr__
    % self.__class__.__name__)
AttributeError: dispatcher instance has no attribute 'socket'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "asyncore.py", line 418, in __getattr__
    %(self.__class__.__name__, attr))
AttributeError: dispatcher instance has no attribute 'foo'
msg179136 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2013-01-05 19:36
Same problem in 3.4.

$ ./python
Python 3.4.0a0 (default:e1bee8b09828, Jan  5 2013, 20:29:00)
[GCC 4.3.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncore
>>> a = asyncore.dispatcher()
>>> del a.socket
>>> a.foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./Lib/asyncore.py", line 410, in __getattr__
    retattr = getattr(self.socket, attr)
  ...
  File "./Lib/asyncore.py", line 410, in __getattr__
    retattr = getattr(self.socket, attr)
RuntimeError: maximum recursion depth exceeded while calling a Python object
>>>
msg179137 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2013-01-05 19:56
asyncore's __getattr__ horror was scheduled for removal a long ago (3.1 as per issue8483).
We can safely remove it for 3.4 and fix the RuntimeError exception above for all the earlier Python versions which are affected.
msg221736 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-27 21:08
This issue has been fixed in Python 3.5 by this change:
---
changeset:   90495:2cceb8cb552b
parent:      90493:d1a03834cec7
user:        Giampaolo Rodola' <g.rodola@gmail.com>
date:        Tue Apr 29 02:03:40 2014 +0200
files:       Lib/asyncore.py Lib/test/test_asyncore.py Misc/NEWS
description:
fix isuse #13248: remove previously deprecated asyncore.dispatcher __getattr__ cheap inheritance hack.
---

If I understdood correctly, for backward compatibility (and limit risk of regressions), this fix cannot be done in Python 3.4.

I close the issue, copy.copy(asyncore.dispatcher()) doesn't crash anymore.
History
Date User Action Args
2014-06-27 21:08:25vstinnersetstatus: open -> closed
versions: + Python 3.5, - Python 3.2
nosy: + vstinner

messages: + msg221736

resolution: fixed
2013-01-05 19:56:58giampaolo.rodolasetassignee: giampaolo.rodola
2013-01-05 19:56:48giampaolo.rodolasetmessages: + msg179137
2013-01-05 19:36:31xdegayesetmessages: + msg179136
2011-10-05 21:00:24xdegayesetmessages: + msg144970
2011-10-05 19:55:48amaury.forgeotdarcsetmessages: + msg144969
2011-10-05 17:32:54giampaolo.rodolasetmessages: + msg144964
2011-10-05 17:20:55amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg144962
2011-10-05 16:21:25xdegayesetmessages: + msg144954
2011-10-05 11:16:13xdegayesetfiles: + infinite_recursion_asyncore.patch
keywords: + patch
messages: + msg144948
2011-10-05 04:30:49ezio.melottisetnosy: + josiahcarlson, giampaolo.rodola, stutzbach

stage: test needed
2011-10-04 20:41:19xdegayecreate