classification
Title: ast module doesn't support optional fields of Parser/Python.asdl
Type: enhancement Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, mbussonn, takluyver, vstinner
Priority: normal Keywords:

Created on 2017-02-22 16:48 by vstinner, last changed 2017-03-24 23:50 by inada.naoki. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 249 merged inada.naoki, 2017-02-23 10:09
PR 267 merged mbussonn, 2017-02-24 00:58
Messages (6)
msg288372 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-22 16:48
The _ast.AST constructor requires a tuple with the same number of items than the fields list.

Example with Raise:

ASDL: Raise(expr? exc, expr? cause)

>>> import _ast
>>> _ast.Raise()
<_ast.Raise object at 0x7fc2ca7dee90>
>>> _ast.Raise(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Raise constructor takes either 0 or 2 positional arguments
>>> _ast.Raise(1, 2)
<_ast.Raise object at 0x7fc2ca7def60>

The cause field is optional in the ASDL, but required in the _ast module.

A tradeoff would be to add the minimum and maximum number of fields to _ast.AST.

This issue should prevent API breakage caused by the new docstring attribute added by issue #29463.
msg288375 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2017-02-22 16:54
Duplicating my comment of 29463 who was in a race condition with this issue:

----
thank you for your work on the AST, I know many developers are looking forward to improvement and stabilisation with the hope of having it stable (and documented) in the stdlib at some point. 

The recent change in PR 46 now change (at least) the constructor of `ast.Module` to take a second mandatory parameter (the docstring). 

I know the ast is autogenerated and "use at your own risk". But IPython for example, use `mod = ast.Module([nodes])`, with the second mandatory parameter added to Module that make it incompatible with current Python 3.7. 
Well it's long until it's released, and we can patch things, but I'm sure we are not the only one in this case, and we'd like older version of IPython to still be compatible with Python 3.7, so if  `Module()`'s second parameter (the docstring) could be optional, that would be great. 

I would be happy if it raise a deprecation warning that it will be required in the future.

I'm of course speaking about `Module` because that's the first error I encountered, but I'm guessing it applies to other changed AST nodes. 

Thanks.
---
msg288414 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2017-02-23 06:21
AST type doesn't have any information about optional fields.  And I don't know it's worth enough to add it.
When AST is instantiated by keyword arguments, there are no check about absence of some fields.
I suppose we can just loosen positional arguments too.

current:

    if (PyTuple_GET_SIZE(args) > 0) {
        if (numfields != PyTuple_GET_SIZE(args)) {
            PyErr_Format(PyExc_TypeError, "%.400s constructor takes %s"
                         "%zd positional argument%s",
                         Py_TYPE(self)->tp_name,
                         numfields == 0 ? "" : "either 0 or ",
                         numfields, numfields == 1 ? "" : "s");

will be:

    if (PyTuple_GET_SIZE(args) > 0) {
        if (numfields > PyTuple_GET_SIZE(args)) {
            PyErr_Format(PyExc_TypeError, "%.400s constructor takes at most "
                         "%zd positional argument%s",
                         Py_TYPE(self)->tp_name,
                         numfields, numfields == 1 ? "" : "s");
msg288442 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2017-02-23 11:21
Now trailing optional fields are optional arguments of AST type.
msg288443 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2017-02-23 11:38
@mbussonn With PR 249, "import os" and "%timeit" works fine.
msg290420 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2017-03-24 23:50
New changeset 4c78c527d215c37472145152cb0e95f196cdddc9 by INADA Naoki in branch 'master':
bpo-29622: Make AST constructor to accept less than enough number of positional arguments (GH-249)
https://github.com/python/cpython/commit/4c78c527d215c37472145152cb0e95f196cdddc9
History
Date User Action Args
2017-03-24 23:50:02inada.naokisetmessages: + msg290420
2017-02-24 00:58:16mbussonnsetpull_requests: + pull_request240
2017-02-23 17:48:48inada.naokisetstatus: open -> closed
resolution: fixed
stage: resolved
2017-02-23 11:38:10inada.naokisetmessages: + msg288443
2017-02-23 11:21:35inada.naokisetmessages: + msg288442
2017-02-23 10:09:22inada.naokisetpull_requests: + pull_request216
2017-02-23 06:21:42inada.naokisetmessages: + msg288414
2017-02-22 21:12:12takluyversetnosy: + takluyver
2017-02-22 16:54:54mbussonnsetnosy: + mbussonn
messages: + msg288375
2017-02-22 16:48:32vstinnercreate