classification
Title: Apparent regression in 3.8-dev: 'TypeError: required field "type_ignores" missing from Module'
Type: behavior Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, benjamin.peterson, brett.cannon, davidism, gvanrossum, njs, yselivanov
Priority: normal Keywords:

Created on 2019-02-04 05:05 by njs, last changed 2019-05-20 15:18 by gvanrossum. This issue is now closed.

Messages (5)
msg334807 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-02-04 05:05
Travis provides a "3.8-dev" python, which is updated regularly to track cpython master. When running our tests on this Python, specifically version

  python: 3.8.0a0 (heads/master:f75d59e, Feb  3 2019, 07:27:24) 

we just started getting tracebacks:

  TypeError                                 Traceback (most recent call last)
  /opt/python/3.8-dev/lib/python3.8/codeop.py in __call__(self, source, filename, symbol)
      131 
      132     def __call__(self, source, filename, symbol):
  --> 133         codeob = compile(source, filename, symbol, self.flags, 1)
      134         for feature in _features:
      135             if codeob.co_flags & feature.compiler_flag:
  TypeError: required field "type_ignores" missing from Module

(Full log: https://travis-ci.org/python-trio/trio/jobs/488312057)

Grepping through git diffs for 'type_ignores' suggests that this is probably related to bpo-35766.

I haven't dug into this in detail, but it seems to be happening on tests using IPython. The lack of further traceback suggests to me that the exception is happening inside IPython's guts (it has some hacks to try to figure out which parts of the traceback are in user-defined code versus its own internal code, and tries to hide the latter when printing tracebacks). The crash is in codeop.Compile.__call__, and IPython does create ast.Module objects and pass them to codeop.Compile.__call__:

https://github.com/ipython/ipython/blob/512d47340c09d184e20811ca46aaa2f862bcbafe/IPython/core/interactiveshell.py#L3199-L3200

Maybe ast.Module needs to default-initialize the new type_ignores field, or compile() needs to be tolerant of it being missing?
msg334808 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-04 05:32
Yeah, this definitely changed. When we updated typed_ast to Python 3.7 recently, the mypy project looked into making certain arguments to AST certain nodes optional, but the code responsible for those arguments is too weird to easily make that work (it's all generated), so we decided to leave it alone.

IMO the AST in general can't be considered a stable API like other stdlib APIs, since when the syntax changes, well, the AST changes, and sometimes that means that nodes change. For *consumers* of the AST we generally manage to keep things backwards compatible, but for producers, there just is no guarantee.

I think the best way forward is for your code to check the Python version and if it's >= 3.8, pass an empty list as the second argument to Module().

If you look at the changes to Python.asdl in the PR you found, you'll see what other nodes have changed.
msg334810 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-02-04 07:06
Oh, that's not my code, it's the core of IPython's REPL :-). I just filed a bug with them, referencing this one: https://github.com/ipython/ipython/issues/11590
msg342862 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-05-19 15:58
there's other optional fields in the ast, type ignores don't seem essential to the `Module`, could those be made optional as well?
msg342913 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-05-20 15:18
> [T]here's other optional fields in the ast, type ignores don't seem essential to the `Module`, could those be made optional as well?

I think you're referring to the `?` syntax in `Python.asdl`.  But the `type_ignores` attribute is already a list (using `*`) and AFAICT you cannot combine `?` and `*`.  You have to provide an empty list.
History
Date User Action Args
2019-05-20 15:18:56gvanrossumsetmessages: + msg342913
2019-05-19 16:49:15davidismsetnosy: + davidism
2019-05-19 15:58:29Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg342862
2019-02-06 00:30:41gvanrossumsetstatus: open -> closed
resolution: not a bug
stage: resolved
2019-02-04 07:06:27njssetmessages: + msg334810
2019-02-04 05:32:42gvanrossumsetmessages: + msg334808
2019-02-04 05:05:47njscreate