classification
Title: Multiple context expressions do not support parentheses for continuation across lines
Type: enhancement Stage: patch review
Components: Interpreter Core Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: pablogsal Nosy List: Anthony Sottile, Jeffrey.Kintscher, Julian, Terry Davis, barry, benjamin.peterson, eric.araujo, ezio.melotti, georg.brandl, ishimoto, lukasz.langa, ncoghlan, pablogsal, r.david.murray, serhiy.storchaka, steven.daprano, thautwarm, ulope
Priority: normal Keywords: patch

Created on 2011-08-19 02:10 by Julian, last changed 2019-07-04 05:06 by Jeffrey.Kintscher.

Messages (21)
msg142411 - (view) Author: Julian Berman (Julian) * Date: 2011-08-19 02:10
Using multiple `with` statements across multiple lines does not support using parens to break them up:


with (open("a_really_long_foo") as foo,
      open("a_really_long_bar") as bar):
    pass

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "demo.py", line 19
    with (open("a_really_long_foo") as foo,
                                    ^
SyntaxError: invalid syntax


Also, without convoluting things, import also does not support doing so, and is the only other example I can think of of a compound statement that forces you to either be redundant or bite your teeth and use \, despite the fact that PEP 328 gave us parens for from imports.

(I did not find a discussion as to why import didn't grow it as well, so please correct me as I'm sure it must have been discussed before).

It's understandably a lot rarer to need multiple lines when importing, but it'd be nice if all compound statements uniformly allowed the same continuation syntax.
msg142549 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-20 18:22
I agree it’s unfortunate that we have to use backslashes to have multi-line with statements.
msg142550 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-08-20 18:27
> is the only other example I can think of 

One similar example would be "raise" in Python 2.

> all compound statements uniformly allowed the same continuation syntax.

This is not true: only "import-as" allows this syntax.  All other uses of parentheses for continuation are continuations of *expressions*.
msg142631 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-08-21 16:30
Following on to Georg's comment about expressions, as a workaround, note
that:

  with (
     open('abc')) as foo:

works.
msg142658 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-08-22 02:18
As Georg noted, only individual expressions get parentheses based continuations automatically. For statement level use of comma separation, it's decided on a case-by-cases basis as to whether we think it is a legitimate usage based on our style guidelines.

That's why 'from location import (name1, name2)' is allowed, but 'import (name1, name2)' is not: we explicitly advise against importing too many modules in a single import statement, but importing multiple names from a single location is often a useful thing to do.

However, while the multiple context expression use case is reasonable, there may be a grammar ambiguity problem in this case, since (unlike from-import) with statements allow arbitrary subexpressions.
msg142750 - (view) Author: Julian Berman (Julian) * Date: 2011-08-22 21:28
>  we explicitly advise against importing too many modules in a single import statement, but importing multiple names from a single location is often a useful thing to do.

Cool. I imagined this had to do with it.

> there may be a grammar ambiguity problem in this case, since (unlike from-import) with statements allow arbitrary subexpressions.

Sorry, can you possibly clarify where the ambiguity might come in?
msg180512 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2013-01-24 06:40
In Python 3.3, we have contextlib.ExitStack() for multiple contexts. 
So, perhaps we can close this issue?
msg180514 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-24 07:58
If this can't be fixed I think it should be at least documented in the FAQs.
msg180515 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-24 08:01
There is a discussion about this on Python-Ideas:

http://comments.gmane.org/gmane.comp.python.ideas/17597
http://mail.python.org/pipermail/python-ideas/2012-October/017610.html
msg236083 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2015-02-16 01:09
This has come up on Python-Ideas again:

http://permalink.gmane.org/gmane.comp.python.ideas/32002

https://mail.python.org/pipermail/python-ideas/2015-February/032000.html
msg236121 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-02-17 02:40
Let's just Won't Fix this.  Use a contextlib.ExitStack.
msg293660 - (view) Author: Ulrich Petri (ulope) * Date: 2017-05-14 21:20
So this would basically be:

    with ExitStack() as stack:
        cm1 = stack.enter_context(a_long_name.with_a_long_method())
        cm2 = stack.enter_context(another_long_variable.with_a_long_method())

Seems like a very non-obvious and inelegant solution...
msg326637 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2018-09-28 13:50
This was closed without enough explanation. Suggesting people should use ExitStack due to a Python grammar deficiency is suboptimal to say the least.

This problem is coming back to users of Black due to Black's removal of backslashes. It's the only piece of our grammar where backslashes are required for readability which shows there's something wrong.

The syntax ambiguity that Nick is raising fortunately shouldn't be a problem because a single tuple is an invalid context manager. In other contexts if the organizational parentheses are matched by the with-statement and not by the underlying `test`, that's also fine since they were organizational this doesn't make the `test` invalid.

Pablo has a working patch for this, we intend to fix this wart for Python 3.8.
msg326639 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-09-28 14:48
Especially since the dynamic flexibility of ExitStack comes at a genuine runtime cost when unwinding the resource stack.

I also (very!) belatedly noticed that I never answered Julian's request for clarification about the potential grammar ambiguity, so going into detail about that now:

The first item in the grammar after the 'with' keyword is a 'test' node, which can already start with a parenthesis, which means a naive attempt at allowing grouping parentheses will likely fail to generate a valid LL(1) parser.

That doesn't mean a more sophisticated change isn't possible (and Pablo has apparently implemented one) - it just means that the required grammar update is going to be more complicated than just changing:

    with_stmt: 'with' with_item (',' with_item)*  ':' suite

to be:

    with_stmt: 'with' (with_items | '(' with_items ')') ':' suite
    with_items: with_item (',' with_item)*

(That would need too much lookahead to decide whether an opening parenthesis belongs to the first 'with_item' in 'with_items' or if it's starting the alternative multi-line grouping construct)
msg326645 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-09-28 16:36
The Python grammar is already not LL(1) strictly. Take for example the production for "argument":

argument: ( test [comp_for] | test '=' test | '**' test | '*' test )

obviously the first sets of test and test are the same and is ambiguous, but the NDFAs are still able to produce DFAs that can generate a concrete syntax tree that allows the AST generation to disambiguate that the second test is a NAME and not any other thing. 

The rule with_stmt: 'with' ( with_item (',' with_item)* | '(' with_item (',' with_item)* [','] ')' ) ':' suite

will generate a similar scenario. The NDFAs will generate DFAs that will ultimately allow us to just skip the more external group of parenthesis when generating the nodes. This makes valid all these expressions:

         with (manager() as x, manager() as y):
             pass
         with (manager() as x, manager() as y,):
             pass
         with (manager()):
             pass
         with (manager() as x):
             pass
         with (((manager()))):
             pass
         with ((((manager()))) as x):

but not this one:

         with (((manager()))) as x:

the reason is that it assigns the first LPAR to the second production and it fails when searching for the one that is at the end. I think this limitation is OK.

If you want to play with that. here is a prototype of the implementation with some tests:

https://github.com/pablogsal/cpython/tree/parenthesized_with
msg326648 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-09-28 17:04
The DFA for the rule

with_stmt: 'with' ( with_item (',' with_item)* | '(' with_item (',' with_item)* [','] ')' ) ':' suite

is:

DFA for with_stmt                                                                                                                                 [512/2103]
  State 0
    'with' -> 1
  State 1
    '(' -> 2
    with_item -> 3
  State 2
    with_item -> 4
  State 3
    ',' -> 5
    ':' -> 6
  State 4
    ')' -> 7
    ',' -> 8
  State 5
    with_item -> 3
  State 6
    suite -> 10
  State 7
    ':' -> 6
  State 8
    ')' -> 7
    with_item -> 4
  State 9
    ',' -> 5
    ':' -> 6
  State 10 (final)
  State 11
    ')' -> 7
    ',' -> 8

It works because the transition from State 1 into a "(" is going to prioritize the path:

0 -> 1 -> "(" -> 2

instead if 

0 -> 1 -> with_item -> 3
msg326727 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-09-30 14:52
Now that I think about this a bit better, this may be actually a problem as:

         with (yield something) as x:

is a more than valid use case that will be broken with the simple grammar rule :(
msg327875 - (view) Author: thautwarm (thautwarm) * Date: 2018-10-17 08:52
How about:

with_stmt: 'with' (with_items | '(' with_items ')') ':' suite
ignored: INDENT | NEWLINE | DEDENT
with_items: with_item (ignored* ',' ignored* with_item)*
msg346137 - (view) Author: Terry Davis (Terry Davis) Date: 2019-06-20 17:00
I'd like to re-raise this issue. Should I cross-post to "discuss.python.org - Ideas"?
msg346157 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-20 18:43
What do you mean with re-raise? The issue is not closed. If you have some proposal to overcome the limitations, the best approach is to comment here in the issue.
msg347025 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-07-01 14:48
Reviewing the thread, we never actually commented on thautwarm's proposal in https://bugs.python.org/issue12782#msg327875 that aims to strip out any INDENT, NEWLINE, and DEDENT tokens that appear between the opening "with" keyword and the statement header terminating ":".

The problem with that is that line continuations are actually handled by the tokenizer, *not* the compiler, and the tokenizer already switches off the INDENT/NEWLINE/DEDENT token generation based on the following rules:

* tracking opening & closing of triple-quoted strings 
* tracking opening & closing of parentheses ("()"), brackets ("[]"), and braces ("{}")
* detecting a backslash immediately followed by a newline

By design, the tokenizer is generally unaware of which NAME tokens are actually keywords - it's only aware of async & await at the moment as part of the backwards compatibility dance that allowed those to be gradually converted to full keywords over the course of a couple of releases.

Hence why INDENT/NEWLINE/DEDENT never appear inside expressions in the Grammar: the tokenization rules mean that those tokens will never appear in those locations.

And it isn't simply a matter of making the tokenizer aware of the combination of "with" and ":" as a new pairing that ignores linebreaks between them, as ":" can appear in many subexpressions (e.g. lambda functions, slice notation, and the new assignments expressions), and it's only the full parser that has enough context to tell which colon is the one that actually ends the statement header.

Thus the design requirement is to come up with a grammar rule that allows this existing code to continue to compile and run correctly:

```
>>> from contextlib import nullcontext
>>> with (nullcontext()) as example:
...     pass
... 
>>> 
```

While also enabling new code constructs like the following:


    with (nullcontext() as example):
        pass

    with (nullcontext(), nullcontext()):
        pass

    with (nullcontext() as example, nullcontext()):
        pass

    with (nullcontext(), nullcontext() as example):
        pass

    with (nullcontext() as example1, nullcontext() as example2):
        pass

If we can get the Grammar to allow those additional placements of parentheses, then the existing tokenizer will take care of the rest.
History
Date User Action Args
2019-07-04 05:06:43Jeffrey.Kintschersetnosy: + Jeffrey.Kintscher
2019-07-01 14:48:03ncoghlansetmessages: + msg347025
2019-06-20 18:43:35pablogsalsetmessages: + msg346157
2019-06-20 17:00:42Terry Davissetnosy: + Terry Davis
messages: + msg346137
2018-10-17 08:52:31thautwarmsetnosy: + thautwarm
messages: + msg327875
2018-10-12 16:48:19Anthony Sottilesetnosy: + Anthony Sottile
2018-09-30 14:52:25pablogsalsetmessages: + msg326727
2018-09-30 14:50:42pablogsalsetpull_requests: - pull_request9036
2018-09-30 14:45:48pablogsalsetkeywords: + patch
pull_requests: + pull_request9036
2018-09-28 17:04:44pablogsalsetmessages: + msg326648
2018-09-28 16:36:05pablogsalsetmessages: + msg326645
2018-09-28 14:48:19ncoghlansetmessages: + msg326639
2018-09-28 13:50:56lukasz.langasetassignee: pablogsal

nosy: + pablogsal
2018-09-28 13:50:45lukasz.langasetstatus: closed -> open

versions: + Python 3.8, - Python 3.4
nosy: + lukasz.langa

messages: + msg326637
resolution: wont fix ->
stage: patch review
2017-05-14 21:20:47ulopesetnosy: + ulope
messages: + msg293660
2015-02-17 02:40:51barrysetstatus: open -> closed
resolution: wont fix
2015-02-17 02:40:13barrysetmessages: + msg236121
2015-02-16 01:09:40steven.dapranosetnosy: + steven.daprano
messages: + msg236083
2013-01-24 08:01:18serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg180515
2013-01-24 07:58:36ezio.melottisetmessages: + msg180514
2013-01-24 06:40:31ishimotosetnosy: barry, georg.brandl, ishimoto, ncoghlan, benjamin.peterson, ezio.melotti, eric.araujo, r.david.murray, Julian
messages: + msg180512
2012-10-31 17:14:59ezio.melottisetnosy: + barry

versions: + Python 3.4, - Python 3.3
2012-08-09 23:52:19ishimotosetnosy: + ishimoto
2011-08-22 21:28:12Juliansetmessages: + msg142750
2011-08-22 02:18:02ncoghlansetmessages: + msg142658
2011-08-21 16:30:40r.david.murraysetnosy: + r.david.murray
messages: + msg142631
2011-08-20 19:01:26ezio.melottisetnosy: + ezio.melotti
2011-08-20 18:27:28georg.brandlsetnosy: + georg.brandl
messages: + msg142550
2011-08-20 18:22:56eric.araujosetnosy: + eric.araujo
messages: + msg142549
2011-08-19 13:10:48pitrousetnosy: + ncoghlan, benjamin.peterson

type: behavior -> enhancement
versions: - Python 2.7
2011-08-19 02:10:17Juliancreate