This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: dir() does not return the list of valid attributes for the object
Type: behavior Stage:
Components: Interpreter Core Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, serhiy.storchaka, tim.peters, veky
Priority: normal Keywords:

Created on 2020-03-28 19:47 by serhiy.storchaka, last changed 2022-04-11 14:59 by admin.

Messages (11)
msg365227 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-28 19:47
Due to the difference in the code of __getattr__ and __dir__ for object and type dir() does not return the list of valid attributes for the object. It can return a name which is not a valid attribute and miss a name of the valid attribute.

1. It does not support metaclasses.

class M(type):
    x = 1

class A(metaclass=M):
    pass

assert hasattr(A, 'x')
assert 'x' not in dir(A)

2. It does not use __mro__, but uses __bases__ recursively.

class M(type):
    def mro(cls):
        if cls.__name__ == 'A':
            return cls, B
        return cls,

class B(metaclass=M):
    x = 1

class A(metaclass=M):
    pass

assert hasattr(A, 'x')
assert 'x' not in dir(A)

3. It uses the __dict__ attribute instead of the instance dict (they can be different).

class A:
    @property
    def __dict__(self):
        return {'x': 1}

assert not hasattr(A(), 'x')
assert 'x' in dir(A())

4. It uses the __class__ attribute instead of type().

class B:
    y = 2

class A:
    x = 1
    @property
    def __class__(self):
        return B

assert hasattr(A, 'x')
assert not hasattr(A, 'y')
assert hasattr(A(), 'x')
assert not hasattr(A(), 'y')
assert 'x' in dir(A)
assert 'y' not in dir(A)
assert 'x' not in dir(A())
assert 'y' in dir(A())

4.1. As a side effect dir() creates an instance dictionary if it was not initialized yet (for memory saving).

It is possible to make these implementations of __dir__() returning exactly what the corresponding __getattr__() accepts, not more and not less. The code will even be much simpler. But is it what we want?
msg365271 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-29 18:02
I think you're going too far for some of these.

> 1. metaclasses

This is reasonable.

> 2. __mro__

This is also reasonable. (I wonder if that part of the dir() implementation predates __mro__?)  I'm not sure about honoring mro() in the metaclass, but it may be useful.

> 3. __dict__

I think this is probably a feature -- I can't think of a reason to override __dict__ as a property except when implementing some sort of proxy class, and then it's likely that there's a matching overrid of __getattr__.

> 4. __class__

Again, I think this is a feature. For example, PEP 585 overrides __class__:

>>> t = list[int]
>>> type(t)
<class 'types.GenericAlias'>
>>> t.__class__
<class 'type'>

> 4.1 auto-creation of instance dict

This seems an accident of implementation, and if you can avoid it, that's better. (Though what use case of dir() currently suffers from memory overhead due to this?)
msg365272 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-29 18:06
FWIW it might be a good idea to look into how PEP 585 could benefit from the improvements to dir(). Currently, dir(list) and dir(list[int]) are quite different -- only the former shows list methods like append and insert.

See https://github.com/gvanrossum/cpython/tree/pep585
msg365411 - (view) Author: Vedran Čačić (veky) * Date: 2020-03-31 19:35
Guido wrote:

>>> t = list[int]

A few years ago, I tried to explain to you that it's much more intuitive to use builtin types instead of their alter egos from typing (List etc.) for [] operator, you said it's not important. It's funny that you've implicitly acknowledged what I said through this typo. ;-)
msg365416 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-31 20:04
> >>> t = list[int]
>
> A few years ago, I tried to explain to you that it's much more intuitive
> to use builtin types instead of their alter egos from typing (List etc.)
> for [] operator, you said it's not important. It's funny that you've
> implicitly acknowledged what I said through this typo. ;-)

It wasn't a typo. See PEPE 585. Its time for this change *now*. It wasn't *then*.
msg365417 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-31 20:19
5. dir(module) does not contain module type attributes (in contrary to dir() for regular object).

>>> import keyword
>>> dir(keyword)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'iskeyword', 'kwlist']
>>> sorted(object.__dir__(keyword))
['__all__', '__builtins__', '__cached__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__file__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__loader__', '__lt__', '__name__', '__ne__', '__new__', '__package__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__spec__', '__str__', '__subclasshook__', 'iskeyword', 'kwlist']

Seems dir() was initially designed for old-style objects and many changes in object model were passed its.

On one hand, it would look logical if dir() returned the list of all names that can be used as attributes. But my concerns are that it may add a "noise", names which can be legally used as attribute names, but which are rarely accessed as instance attribute instead of type attributes. This applies to 1 (metaclasses) and 5 (module type).

If left out 1 and 5 and implement all other options (and I remind that it simplifies the code), it will only break one test for unittest.mock. I'm exploring what's going on there.
msg365419 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-31 20:24
For me, one of the most annoying things about dir() is that it gives all the dunders. There are many dunders that are just always there (__class__, mostly __dict__, __doc__, __name__ etc.). I wish it would just not give dunders that are inherited from object or type. Though it's probably too late for that (people's code might break).
msg365420 - (view) Author: Vedran Čačić (veky) * Date: 2020-03-31 20:33
Guido:

First, wow!

Second, now it's even more glaring that list[str] has repr 'list[str]', while list doesn't have a repr 'list'. :-( Is it possible that the time for _that_ change will also come some day? :-)

Third, my problem with dir is not dunders, but dunders that really shouldn't be there. For example, __le__ when type doesn't support ordering. I understand how that happens, but it doesn't mean it isn't a bug.
msg365423 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-31 20:40
Vedran, please stay on topic for this issue.

FWIW I agree that it would be best if dir() showed only those dunders that are significant. E.g. __eq__ should only be shown if it is overridden by a subclass.

Serhiy did you see my feedback on 3 and 4?
msg365426 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-03-31 21:05
If using __class__ and __dict__ attribuites is a feature, it does not work for __getattr__(). I think it is confusing if __dir__() and __getattr__() are not consistent. I'll try to deal with unittes.mock and then will look how this will work with other classes that override __class__ or __dict__. Also we need to check all classes with custom __getattr__ for the matter of custom __dir__.

I think that it might be better if dir() for instance excluded names which are not accessed as via instance, e.g. non-descriptor class attributes and class methods. But it may be not easy to do.
msg365428 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-31 21:33
At least for overriding __dict__, I would presume there would be a matching override for __getattribute__ (not __getattr__).

Ditto for __class__, e.g. look at GenericAlias in the pep585 implementation PR. (Here __class__ is not in the list of exceptions so it returns the attribute from list.)
History
Date User Action Args
2022-04-11 14:59:28adminsetgithub: 84279
2020-03-31 21:33:43gvanrossumsetmessages: + msg365428
2020-03-31 21:05:18serhiy.storchakasetmessages: + msg365426
2020-03-31 20:40:26gvanrossumsetmessages: + msg365423
2020-03-31 20:33:08vekysetmessages: + msg365420
2020-03-31 20:24:42gvanrossumsetmessages: + msg365419
2020-03-31 20:19:01serhiy.storchakasetmessages: + msg365417
2020-03-31 20:04:10gvanrossumsetmessages: + msg365416
2020-03-31 19:35:51vekysetnosy: + veky
messages: + msg365411
2020-03-29 18:06:38gvanrossumsetmessages: + msg365272
2020-03-29 18:02:58gvanrossumsetmessages: + msg365271
2020-03-28 19:47:28serhiy.storchakacreate