New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pydoc fails with codecs #69012
Comments
Pydoc fails with the codecs module in 3.5+. All works in 3.4. $ ./python -m pydoc codecs
Traceback (most recent call last):
File "/home/serhiy/py/cpython-3.5/Lib/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/serhiy/py/cpython-3.5/Lib/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 2648, in <module>
cli()
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 2613, in cli
help.help(arg)
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 1895, in help
elif request: doc(request, 'Help on %s:', output=self._output)
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 1632, in doc
pager(render_doc(thing, title, forceload))
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 1625, in render_doc
return title % desc + '\n\n' + renderer.document(object, name)
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 370, in document
if inspect.ismodule(object): return self.docmodule(*args)
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 1160, in docmodule
contents.append(self.document(value, key, name))
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 372, in document
if inspect.isroutine(object): return self.docroutine(*args)
File "/home/serhiy/py/cpython-3.5/Lib/pydoc.py", line 1345, in docroutine
signature = inspect.signature(object)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 2988, in signature
return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 2738, in from_callable
follow_wrapper_chains=follow_wrapped)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 2229, in _signature_from_callable
skip_bound_arg=skip_bound_arg)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 2061, in _signature_from_builtin
return _signature_fromstr(cls, func, s, skip_bound_arg)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 2009, in _signature_fromstr
p(name, default)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 1991, in p
default_node = RewriteSymbolics().visit(default_node)
File "/home/serhiy/py/cpython-3.5/Lib/ast.py", line 245, in visit
return visitor(node)
File "/home/serhiy/py/cpython-3.5/Lib/ast.py", line 310, in generic_visit
new_node = self.visit(old_value)
File "/home/serhiy/py/cpython-3.5/Lib/ast.py", line 245, in visit
return visitor(node)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 1978, in visit_Attribute
return wrap_value(value)
File "/home/serhiy/py/cpython-3.5/Lib/inspect.py", line 1965, in wrap_value
raise RuntimeError()
RuntimeError |
This is related to Arguments Clinic and Larry's implementation of signature parsing for built-in functions. This particular bug is caused by 'codecs.encode' & 'codecs.decode' functions with the AC signatures defined as follows: _codecs.encode "encoding" argument's default is a method call, and _signature_fromstr fails to recognize method calls appropriately. The attached patch fixes the problem by rendering such default values as "<method_name()>". I don't think we should evaluate the method call anyways, because it can cause strange side effects and can be just plain wrong -- like in this issue -- we shouldn't render 'encoding="utf-8"' just because that's how docs.python.org server is configured. Anyways, the patch isn't pretty, but does fix the problem with minimal code. Another option would be to fix codecs.encode and codecs.decode signatures to "encoding: None" and edit documentation accordingly. Assigning this issue to Larry for his review. |
This is a legitimate problem and I'd definitely like it fixed.
Attached is a tweaked version of the patch that sidesteps the quote marks and the angle brackets, by substituting in an object with a custom repr. Yury, if my change to your patch looks good to you, please go ahead and check it in. That way it won't slow down 3.5.0rc1. Thanks! |
How with more complex expressions, like "sys.getdefaultencoding() or 'utf-8'". |
Larry, Serhiy, After giving this some thought I think that my initial patch is a wrong approach here -- inspect module should not be touched to fix this issue. With this patch applied, signature object for codecs.encode would have a Parameter with a bogus default value, breaking functions like 'BoundArguments.apply_defaults()' etc. In other words, whatever AC puts in 'signature.parameters['encoding'].default' must be an object that will be accepted by the function. codecs.encode, if implemented in Python, would look like: def encode(obj, encoding=None, errors='strict'):
if encoding is None:
encoding = sys.getdefaultencoding()
... And that's how the signature should be defined for the C version (because that's what is actually happening in C code as well!) The new patch changes the AC specs from _codecs.encode to _codecs.encode (docstring is updated too). This change, by the way, is in accordance with PEP-436: The values supplied for these [default] parameters must be compatible with ast.literal_eval. |
Can we do better? How about a new field on the Parameter object, "symbolic_default_value", which shows you the expression used to compute the value? We could then set default_value to the result of the expression, pydoc could print the symbolic expression, and codecs encode/decode could be more straightforward. |
Maybe it's a good idea, but I'm -1 on pushing new APIs to 3.5 without some careful consideration at this super late stage (and it's not going to be a small change btw). Let's just fix the codecs module. |
I just tried every module, simulating pydoc on it. codecs is the only one that failed. So, I can live with just changing codecs for now. But let's do it properly in 3.6. Go ahead and check in for 3.5--or, if you don't get it done before I want to tag the release, I'll do it. Explain one thing to me, though--how would docs.python.org get "utf-8" for the encoding argument to encode/decode? The docs are built from .rst files, not from introspection on a live interpreter. |
sys.getdefaultencoding() always returns "utf-8" in 3.x (it left for compatibility with 2.x). I suggested to set defaults to literal "utf-8". This matches documentation and signatures of str.encode() and bytes.decode(). |
That's a fine change for 3.5. For 3.6 I want to solve the general problem, at which point we can switch back to calling sys.getdefaultencoding() if we like. Serhiy, can you make a patch a post it here? I want to tag 3.5.0rc1 in one or two hours. |
The patch is easy. In future we should unify docstrings for codecs.encode() and codecs.decode() with str.encode(), bytes.decode() and like. |
Please use encoding='utf-8' as definition for codecs.encode() and codecs.decode(). There is no adjustable default encoding in Python 3 anymore. For Python 3.6 this should probably be fixed everywhere. |
Please change "Default encoding is" to "The default encoding is". Apart from that, LGTM, please check in! |
New changeset bdd1df816f84 by Serhiy Storchaka in branch '3.5': New changeset ad65cad76331 by Serhiy Storchaka in branch 'default': |
Fixed. Thanks, Serhiy! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: