New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Special method lookup fails on uninitialized types #71093
Comments
This is an annoying heisenbug; it seems that some objects cannot be formatted until you explicitly do obj.format. For example Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> format(object.__reduce__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type method_descriptor doesn't define __format__
>>> format(object.__reduce__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type method_descriptor doesn't define __format__
>>> object.__reduce__.__format__
<built-in method __format__ of method_descriptor object at 0x7f67563ed0e0>
>>> format(object.__reduce__)
"<method '__reduce__' of 'object' objects>" I can replicate this in 2.7.9, .10 and .11 on Ubuntu and Debian, though it works on Windows Python, works in 2.6.6, and Pythons 3 wherever I've tried, but I've heard this also failing on Python 3. |
s/explicitly do/explicitly access/ |
Proposed patch makes method descriptors types to be explicitly initialized as in 3.x. |
There is similar issue on 3.x: >>> import array
>>> it = iter(array.array('i'))
>>> format(it)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type arrayiterator doesn't define __format__
>>> type(it).__format__
<method '__format__' of 'object' objects>
>>> format(it)
'<arrayiterator object at 0xb703f4ec>' |
A number of other types are not initialized until you request an attribute. Here is larger patch for 2.7 that makes 38 types to be explicitly initialized. |
An alternative way is just call PyType_Ready from _PyType_Lookup if type->tp_mro is NULL. Here is a patch against 2.7 that restores the solution from bpo-551412, but returns NULL if type->tp_mro is still NULL after calling PyType_Ready. I found one place in tests when this is happened (CIOTest.test_IOBase_finalize in test_io). |
Serhiy, I'm happy to help, but I'm not sure what you're asking me to do. Decide between different patches? I can't even repro the issue. |
Added a check for Py_TPFLAGS_READYING to prevent recursive calling. |
The problem is that format() fails for instances of some classes, because the type still is not initialized. The simplest example -- list iterator. >>> format(iter([]))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type listiterator doesn't define __format__ After forcing type initialization (for example by getting any type's attribute), format() becomes working. >>> type(iter([])).foo
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: type object 'listiterator' has no attribute 'foo'
>>> format(iter([]))
'<listiterator object at 0xb708d0ec>' I afraid that format() is just one example, and there are other functions or operators that don't work or work incorrectly if the type was not initialized. init_types-2.7.patch adds explicit initialization of 38 types (I didn't check that all of them need this, but I suppose they do). This is large patch, and I'm not sure that it fixes all types. Other way is to try to initialize the type just in _PyType_Lookup if it is not initialized. This is simple change, but a comment in _PyType_Lookup warns me. I found that this solution was already applied as an attempt to fix bpo-551412, but then reverted. Since you seem to be the most knowledgeable with this code, I'm asking you what was wrong with this approach and how we can fix this. Python 3.x also suffers from this bug, but it is reproduced with less types. For example it isn't reproduced for list iterator. I don't know why. |
I can reproduce the bug in 3.5.0+ Ubuntu with list iterator, if I execute python with -S: % python3.5 -S
Python 3.5.0+ (default, Oct 11 2015, 09:05:38)
[GCC 5.2.1 20151010] on linux
>>> format(iter([]))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Type list_iterator doesn't define __format__ Thus here it depends on the stuff that site does or doesn't do. Iterating over a list iterator does *not* trigger the initialization. Printing it doesn't help either, or anything else that does not touch the non-magic attributes. I am not even sure what the site.py and such are doing to the list iterator class to trigger the initialization. |
And to the other things failing, I was trying to find out which of the magic method ops fail, and for that tried to find out the % python3.5 -S
Python 3.5.0+ (default, Oct 11 2015, 09:05:38)
[GCC 5.2.1 20151010] on linux
>>> dir(iter([]))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: object does not provide __dir__ |
Sadly it's been a very long time since I wrote that code and I don't recall |
Is there a way to have format() try to force the initialization, by explicitly doing the equivalent of obj.__format__, at least for types, instead of raising the TypeError? |
But the problem isn't limited to format()... Why would format() be special? |
There is one test (ClassPropertiesAndMethods.test_mutable_bases_with_failing_mro in test_descr) that crashes with the code from bpo-551412 because _PyType_Lookup() is recursive called from PyType_Ready(). Is this the reason? My patch prevents recursive calls. Here is minimal example (for Python 3): class M(type):
def mro(self):
hasattr(self, 'foo')
return type.mro(self)
class C(metaclass=M):
pass When class C is created, C.mro() is called while C still is not ready. Resolving an attribute calls _PyType_Lookup() which calls PyType_Ready() which calls mro() etc. |
Probably. |
I am not an expert on PyType internals, so I am wondering why is the PyType_Ready'ing done implicitly at all? |
Because the data structure that defines a type is just data, and at some |
Could it be possible to to make the debug build absolutely abort on any usage of PyType's that are not readied, usage including instantiating them. Then, instead of changing all Likewise the C-API docs for PyType_Ready should perhaps say "This must be called on all type objects to finish their initialization." instead of "should" |
Similar bug just was introduced in bpo-21124. |
Yet one similar bug: bpo-11702. |
Serhiy -- please do what do you think we should do. At this point I'm open to just about anything, but I don't feel comfortable creating or reviewing patches any more. |
Yet one demonstration of this bug: $ ./python -IS
>>> import operator
>>> operator.length_hint(iter("abc"))
0
>>> import collections.abc
>>> operator.length_hint(iter("abc"))
3 |
New changeset bbaf6c928526 by Serhiy Storchaka in branch '3.5': New changeset 3119f08802a5 by Serhiy Storchaka in branch '2.7': New changeset 888a26fac9d2 by Serhiy Storchaka in branch '3.6': New changeset d24f1467a297 by Serhiy Storchaka in branch 'default': |
(Just to save time for anyone interested) Anyway, with regard to the disconcerting comment: I realize adding such a flag is really a big deal, but maybe it's worth catching sneaky bugs caused by Python's equivalent of Use-After-Free bugs? |
All of the examples for python 3 are working now: >>> import array
>>> it = iter(array.array('i'))
>>> format(it)
'<array.arrayiterator object at 0x10598f7a0>'
>>> format(iter([]))
'<list_iterator object at 0x10598f890>'
>>> import operator
>>> operator.length_hint(iter("abc"))
03 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: