New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add endline and endcolumn to every AST node #77597
Comments
Some Python tools (in particular I am interested in type checkers) will benefit from knowing where a given expression ends to indicate/highlight location of an error in the source code. Other tools and IDEs may have also some other benefits. Currently such tools need to use some hacks and/or custom parser to find the end line and end column of an expression, while it would be more straightforward to just add this information to every AST node by CPythons own parser. This will increase memory usage, but expectation is that this effect will be minor. What do you think? |
I like this. This can help to generate more meaningful error messages at compile and run time. |
bpo-22616 is a bit different (proposing line/column ranges instead of just endline/endcolumn). I am happy to close this one in favor of bpo-22616 if we agree that we will go with endline/endcolumn. I can't guarantee, but likely I will work on this during winter holidays, unless there are some objections, or unless someone will do this before :-) |
FYI, I started working on this. I will have PR ready end of next week. Serhiy, I don't think we should keep both this and bpo-22616 open. Which one would you prefer to close? |
This idea seems reasonable. Most of the AST nodes have a span and it would be nice to know what that is. I'm sure we will find use cases though I doubt that many compiler syntax errors would benefit (since a syntax error means that we don't have a completely matched grammar rule). BTW, does this information have to be added by the parser or could there be am AST module function that deduces the end locations from the start location of next sibling node or from the parent node? If so, it may still be possible get the benefit without slowing down or complicating the parser logic. Do we know what other languages do (carry the full span info in the AST or deduce the span after the fact)? I don't know what the best practices are in this regard. One other thought: We should be careful not to impair existing AST capabilities that support code rewriting and movement (i.e. refactoring tools). The copy_location() and fix_missing_locations() functions would need to be updated as well. |
It is up to you Ivan. The end location can not be deduces from the start location of next sibling node or from the parent node. For example, the AST for the expression "foo.bar.baz" does not contain information for the end location of "foo.bar". |
This is mostly useful for code analysis tools and IDEs.
There may be some other ways to do it, but currently I am adding both
I didn't do real research, but quick browsing shows carrying the end position info is quite common. But more importantly as Serhiy noted, deducing might be just not possible in some situations.
OK, I will close the other one as superseded by this issue. |
I strongly support this feature, because my IDE (https://thonny.org) needs to highlight AST nodes in the source code. There would be many interested parties if you count the stars of this project: https://github.com/gristlabs/asttokens |
Buildbots are green, this can be now closed. |
Re-opening to track renaming of potentially public PyNode_AddChild() and PyParser_AddToken(). |
So what was the conclusion about PyCode_New()? Situation is quite similar here (except that these functions are used less often). Should we just document the changes in What's New? |
It seems we're keeping the new PyCode_New() signature, so we can do the same here. Just document it; a versionchanged (if there isn't one yet) and a mention in What's New sound appropriate. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: