classification
Title: Rlcompleter.Completer does not use __dir__ magic method
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: tab-completition on instances repeatedly accesses attribute/descriptors values
View: 25590
Assigned To: docs@python Nosy List: BreamoreBoy, carlj, eric.araujo, georg.brandl, ggenellina, martin.panter
Priority: normal Keywords: patch

Created on 2009-01-26 01:28 by carlj, last changed 2016-05-13 13:20 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
rlcompleter.patch ggenellina, 2009-01-27 03:58
Messages (10)
msg80556 - (view) Author: Carl Johnson (carlj) Date: 2009-01-26 01:28
The documentation at http://docs.python.org/library/rlcompleter.html
claims that

Completer.complete(text, state)¶

   Return the state*th completion for *text.

   If called for text that doesn’t include a period character ('.'), it
will complete from names currently defined in __main__, __builtin__ and
keywords (as defined by the keyword module).

   If called for a dotted name, it will try to evaluate anything without
obvious side-effects (functions will not be evaluated, but it can
generate calls to __getattr__()) up to the last part, and find matches
for the rest via the dir() function. Any exception raised during the
evaluation of the expression is caught, silenced and None is returned.

In other words, it claims to use dir(obj) as part of the tab completion
process. This is not true (using Python 2.6.1 on OS X):

>>> class B(object):
...  def __dir__(self): return dir(u"") #Makes B objects look like strings
...
>>> b = B()
>>> dir(b)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__',
'__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__',
'__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__',
'__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__',
'__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__',
'__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__',
'_formatter_field_name_split', '_formatter_parser', 'capitalize',
'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find',
'format', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit',
'islower', 'isnumeric', 'isspace', 'istitle', 'isupper', 'join',
'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex',
'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines',
'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']
>>> c = rlcompleter.Completer()
>>> c.complete("b.", 0) #Notice that it does NOT return __add__
u'b.__class__('
>>> c.matches #Notice that this list is completely different from the
list given by dir(b)
[u'b.__class__(', u'b.__delattr__(', u'b.__doc__', u'b.__format__(',
u'b.__getattribute__(', u'b.__hash__(', u'b.__init__(', u'b.__new__(',
u'b.__reduce__(', u'b.__reduce_ex__(', u'b.__repr__(',
u'b.__setattr__(', u'b.__sizeof__(', u'b.__str__(',
u'b.__subclasshook__(', u'b.__class__(', u'b.__class__(',
u'b.__delattr__(', u'b.__dict__', u'b.__dir__(', u'b.__doc__',
u'b.__format__(', u'b.__getattribute__(', u'b.__hash__(',
u'b.__init__(', u'b.__module__', u'b.__new__(', u'b.__reduce__(',
u'b.__reduce_ex__(', u'b.__repr__(', u'b.__setattr__(',
u'b.__sizeof__(', u'b.__str__(', u'b.__subclasshook__(',
u'b.__weakref__', u'b.__class__(', u'b.__delattr__(', u'b.__doc__',
u'b.__format__(', u'b.__getattribute__(', u'b.__hash__(',
u'b.__init__(', u'b.__new__(', u'b.__reduce__(', u'b.__reduce_ex__(',
u'b.__repr__(', u'b.__setattr__(', u'b.__sizeof__(', u'b.__str__(',
u'b.__subclasshook__(']

Suggested course of action: 

* Change the documentation for Python 2.6/3.0.
* Update Completer to use __dir__ in Pythons 2.7/3.1 and revert the
documentation.

See
http://mail.python.org/pipermail/python-dev/2009-January/thread.html#85471
msg80615 - (view) Author: Gabriel Genellina (ggenellina) Date: 2009-01-27 03:58
This is not a bug in rlcompleter; __dir__ is returning bogus items, and 
rlcompleter checks whether there exist actually an attribute with such 
name.

Defining __getattr__ (or __getattribute__) and a matching __dir__ works 
fine:

>>> class B(object):
...   def __dir__(self):
...     return dir(object) + ["xa","xb","xc"]
...   def __getattr__(self, name):
...     if name in ["xa","xb","xc"]:
...       return None
...     raise AttributeError, name
...
>>> b = B()
>>> import rlcompleter
>>> c = rlcompleter.Completer()
>>> c.complete("b.", 0)
'b.__class__('
>>> c.matches
['b.__class__(', 'b.__delattr__(', 'b.__doc__', 'b.__format__(', 
'b.__getattribute__(', 
...
'b.xa', 'b.xb', 'b.xc', 'b.__class__(', 'b.__class__(', 
...]
>>> c.complete("b.x", 0)
'b.xa'
>>> c.matches
['b.xa', 'b.xb', 'b.xc']

Now, looking at this I saw there *is* a bug in rlcompleter, as it may 
return many duplicate items:

>>> c.complete("b.__c", 0)
'b.__class__('
>>> c.matches
['b.__class__(', 'b.__class__(', 'b.__class__(', 'b.__class__(']

The attached patch fixes that.
msg80619 - (view) Author: Carl Johnson (carlj) Date: 2009-01-27 04:39
It seems to me that it isn't tab completion's place to out think the
__dir__ method. A) Because the documentation doesn't tell you that it
does (although you are warned that it may call some stuff) and B)
because if someone set up a __dir__ method, they probably are listing
the things that they want listed for a particular reason. I think that
it would be less confusing for rlcompleter to follow the __dir__ method
when it exists and only do its own poking and prodding when it does not.
msg80623 - (view) Author: Gabriel Genellina (ggenellina) Date: 2009-01-27 06:09
This is what rlcompleter does; it uses dir() to find out what names to 
return.
Or do you mean that it should not iterate along __bases__ because this 
has already been done by dir()?
msg80625 - (view) Author: Carl Johnson (carlj) Date: 2009-01-27 06:15
I think that checking to see which things really exist with
getattr/hasattr made sense back in the days before the __dir__, since in
those days the real API for an object could diverge wildly from what was
reported by dir(object), but nowadays, if someone goes to the trouble of
defining the __dir__ method, then we should just trust that as being
"the API" and not do any other checking.
msg80626 - (view) Author: Gabriel Genellina (ggenellina) Date: 2009-01-27 06:46
The check is made to decide whether the attribute is a method or not 
(because methods get a "(" appended) -- for names that fail to exist, 
one could just omit the "(" and include the name anyway.

rlcompleter does nothing special with __dir__, it always uses dir() 
only.
msg80627 - (view) Author: Carl Johnson (carlj) Date: 2009-01-27 06:53
Ah, I see. It does a dir(obj) then tests things to see which are
callable and while it is at that, it removes the names that don't really
exist according to getattr.

Actually, can we go back to the Python 2.5 behavior? I really hate those
auto-added parentheses. For one thing, it screws it up when you do
"help(name<TAB>". Am I missing some really obvious switch that would
turn the behavior back to the old style of ignoring the
callable/non-callable thing?
msg80630 - (view) Author: Gabriel Genellina (ggenellina) Date: 2009-01-27 07:20
The current behaviour is actually a requested feature: see #449227

I see your point, it may be annoying sometimes -- but calling a method 
is far more common than just getting a reference to it, so I think the 
current behaviour is fine (I'm talking about the added "(", not the 
repeated entries, nor the unneeded __bases__ recursion, nor the deleted 
entries)
msg221649 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-06-26 21:17
Patched code produces identical output to unpatched code.  This doesn't really surprise me as word is reused within the for loop.
msg265474 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-13 13:20
Issue 14782 is open for complaints about the opening brackets.

I think this report was mainly about attributes that fail the hasattr() test. In Issue 25590 we were conservative and only fixed it in 3.6+.
History
Date User Action Args
2016-05-13 13:20:46martin.pantersetcomponents: + Library (Lib), - Documentation, Extension Modules
2016-05-13 13:20:23martin.pantersetstatus: open -> closed

superseder: tab-completition on instances repeatedly accesses attribute/descriptors values
versions: + Python 3.6, - Python 2.6, Python 3.0
nosy: + martin.panter

messages: + msg265474
resolution: duplicate
stage: resolved
2014-06-26 21:17:48BreamoreBoysetnosy: + BreamoreBoy
messages: + msg221649
2010-10-29 10:07:21adminsetassignee: georg.brandl -> docs@python
2010-02-16 05:57:29eric.araujosetnosy: + eric.araujo
2009-01-27 07:20:44ggenellinasetmessages: + msg80630
2009-01-27 06:53:43carljsetmessages: + msg80627
2009-01-27 06:46:08ggenellinasetmessages: + msg80626
2009-01-27 06:15:38carljsetmessages: + msg80625
2009-01-27 06:09:03ggenellinasetmessages: + msg80623
2009-01-27 04:39:40carljsetmessages: + msg80619
2009-01-27 03:58:49ggenellinasetfiles: + rlcompleter.patch
nosy: + ggenellina
messages: + msg80615
keywords: + patch
2009-01-26 01:28:57carljcreate