New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
async/await parser issues #68807
Comments
If I understand the reference manual correctly, these should probably >>> async def f():
... def g(): pass
... async = 10
...
>>> async def f():
... def async():
... pass
...
>>> async def f(): async = 10
...
>>> async def f():
... def await(): pass
...
>>> And this should perhaps be accepted: >>> async def f():
... return lambda await: await
File "<stdin>", line 2
return lambda await: await
^
SyntaxError: invalid syntax This, too: >>> async def f():
... async def g(): pass
... await z
File "<stdin>", line 3
await z
^
SyntaxError: invalid syntax |
Sorry for not responding earlier, I'm on vacation and don't check email too often. While investigating what can be done in tokenizer.c to fix some of the bugs that Stefan pointed out, I discovered a simpler approach: instead of checking what kind of function we have on top of the stack, I now have a counter of nested async functions. This aligns nicely with what Nick said on python-dev once: we essentially treat 'async def' as a future import, everything inside it parses differently. In other words, before this patch: async def foo():
def bar():
async = 1 was legal, now it's a syntax error, as 'async' will be parsed as 'ASYNC' token. This simplifies the implementation of tokenizer.c hacks, fixes all of the bugs that Stefan posted here, *and* enables one-line 'async' functions: async def foo(): await bar() # legal with the patch It will also help with migrating the code to Python 3.7 when async/await will become proper keywords. And, with this patch we can effectively remove the shortcomings section from the PEP-492 (https://www.python.org/dev/peps/pep-0492/#transition-period-shortcomings). Please review the attached patch, it would be great if we can commit it before 3.5beta4. |
Haven't reviewed the patch, but this approach sounds great (in fact I had |
Good news :) I guess this means we can also remove the sentence I added at <https://docs.python.org/3.5/reference/compound_stmts.html#coroutine-function-definition\>. |
An updated patch is attached. I had to implement a little bit more sophisticated tracking of one-line functions to fix parsing of things like def foo():
async def f(): pass
async def f(): pass
async = 1 I hope that test_coroutine.py now covers all possible legal and illegal async/await syntax.
Yes, I'm a bit surprised myself ;) I guess one of the reasons why I tried to do more in tokenizer is that at the time of me hacking the tokenizer, compile.c wasn't quite ready (specifically, catching async for/async with/await outside of async functions).
Right! |
Patch & test cases look good to me. I'm so used to thinking of the tokenisation phase as a linear token stream |
New changeset 9da080ecadb2 by Yury Selivanov in branch '3.5': New changeset 987b72921a0c by Yury Selivanov in branch 'default': |
Thanks Nick! I've committed the patch with a few more unittests and a couple of additional comments in tokenizer.(c|h). |
New changeset e4e01488afff by Yury Selivanov in branch '3.5': New changeset 6f4e0c462daf by Yury Selivanov in branch 'default': |
This is a very nice solution! I'm just curious if the 'ctx' is still needed: It looks like IOW, a single indent_level variable that follows all INDENTs/DEDENTs [This is in no way urgent, please do not feel obliged to respond |
This is a wonderful idea, that's the way it should be done in Please see the new patch. I love the simplicity of it, no |
Updated patch attached (rebased; minor optimization). I'll commit this patch in a few hours to make sure it lands in 3.5beta4. |
New changeset d03f86e41066 by Yury Selivanov in branch '3.5': New changeset 3f8048926690 by Yury Selivanov in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: