Title: Multiple context expressions do not support parentheses for continuation across lines
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pablogsal Nosy List:
Priority: normal Keywords: patch

Created on 2011-08-19 02:10 by Julian, last changed 2021-11-04 14:27 by eryksun. This issue is now closed.

Messages (26)
msg142411 - (view) Author: Julian Berman (Julian) * Date: 2011-08-19 02:10
Using multiple `with` statements across multiple lines does not support using parens to break them up:

with (open("a_really_long_foo") as foo,
      open("a_really_long_bar") as bar):

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "", line 19
    with (open("a_really_long_foo") as foo,
SyntaxError: invalid syntax

Also, without convoluting things, import also does not support doing so, and is the only other example I can think of of a compound statement that forces you to either be redundant or bite your teeth and use \, despite the fact that PEP 328 gave us parens for from imports.

(I did not find a discussion as to why import didn't grow it as well, so please correct me as I'm sure it must have been discussed before).

It's understandably a lot rarer to need multiple lines when importing, but it'd be nice if all compound statements uniformly allowed the same continuation syntax.
msg142549 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-08-20 18:22
I agree it’s unfortunate that we have to use backslashes to have multi-line with statements.
msg142550 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-08-20 18:27
> is the only other example I can think of 

One similar example would be "raise" in Python 2.

> all compound statements uniformly allowed the same continuation syntax.

This is not true: only "import-as" allows this syntax.  All other uses of parentheses for continuation are continuations of *expressions*.
msg142631 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-08-21 16:30
Following on to Georg's comment about expressions, as a workaround, note

  with (
     open('abc')) as foo:

msg142658 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-08-22 02:18
As Georg noted, only individual expressions get parentheses based continuations automatically. For statement level use of comma separation, it's decided on a case-by-cases basis as to whether we think it is a legitimate usage based on our style guidelines.

That's why 'from location import (name1, name2)' is allowed, but 'import (name1, name2)' is not: we explicitly advise against importing too many modules in a single import statement, but importing multiple names from a single location is often a useful thing to do.

However, while the multiple context expression use case is reasonable, there may be a grammar ambiguity problem in this case, since (unlike from-import) with statements allow arbitrary subexpressions.
msg142750 - (view) Author: Julian Berman (Julian) * Date: 2011-08-22 21:28
>  we explicitly advise against importing too many modules in a single import statement, but importing multiple names from a single location is often a useful thing to do.

Cool. I imagined this had to do with it.

> there may be a grammar ambiguity problem in this case, since (unlike from-import) with statements allow arbitrary subexpressions.

Sorry, can you possibly clarify where the ambiguity might come in?
msg180512 - (view) Author: Atsuo Ishimoto (ishimoto) * Date: 2013-01-24 06:40
In Python 3.3, we have contextlib.ExitStack() for multiple contexts. 
So, perhaps we can close this issue?
msg180514 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-01-24 07:58
If this can't be fixed I think it should be at least documented in the FAQs.
msg180515 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-24 08:01
There is a discussion about this on Python-Ideas:
msg236083 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2015-02-16 01:09
This has come up on Python-Ideas again:
msg236121 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2015-02-17 02:40
Let's just Won't Fix this.  Use a contextlib.ExitStack.
msg293660 - (view) Author: Ulrich Petri (ulope) * Date: 2017-05-14 21:20
So this would basically be:

    with ExitStack() as stack:
        cm1 = stack.enter_context(a_long_name.with_a_long_method())
        cm2 = stack.enter_context(another_long_variable.with_a_long_method())

Seems like a very non-obvious and inelegant solution...
msg326637 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2018-09-28 13:50
This was closed without enough explanation. Suggesting people should use ExitStack due to a Python grammar deficiency is suboptimal to say the least.

This problem is coming back to users of Black due to Black's removal of backslashes. It's the only piece of our grammar where backslashes are required for readability which shows there's something wrong.

The syntax ambiguity that Nick is raising fortunately shouldn't be a problem because a single tuple is an invalid context manager. In other contexts if the organizational parentheses are matched by the with-statement and not by the underlying `test`, that's also fine since they were organizational this doesn't make the `test` invalid.

Pablo has a working patch for this, we intend to fix this wart for Python 3.8.
msg326639 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-09-28 14:48
Especially since the dynamic flexibility of ExitStack comes at a genuine runtime cost when unwinding the resource stack.

I also (very!) belatedly noticed that I never answered Julian's request for clarification about the potential grammar ambiguity, so going into detail about that now:

The first item in the grammar after the 'with' keyword is a 'test' node, which can already start with a parenthesis, which means a naive attempt at allowing grouping parentheses will likely fail to generate a valid LL(1) parser.

That doesn't mean a more sophisticated change isn't possible (and Pablo has apparently implemented one) - it just means that the required grammar update is going to be more complicated than just changing:

    with_stmt: 'with' with_item (',' with_item)*  ':' suite

to be:

    with_stmt: 'with' (with_items | '(' with_items ')') ':' suite
    with_items: with_item (',' with_item)*

(That would need too much lookahead to decide whether an opening parenthesis belongs to the first 'with_item' in 'with_items' or if it's starting the alternative multi-line grouping construct)
msg326645 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-09-28 16:36
The Python grammar is already not LL(1) strictly. Take for example the production for "argument":

argument: ( test [comp_for] | test '=' test | '**' test | '*' test )

obviously the first sets of test and test are the same and is ambiguous, but the NDFAs are still able to produce DFAs that can generate a concrete syntax tree that allows the AST generation to disambiguate that the second test is a NAME and not any other thing. 

The rule with_stmt: 'with' ( with_item (',' with_item)* | '(' with_item (',' with_item)* [','] ')' ) ':' suite

will generate a similar scenario. The NDFAs will generate DFAs that will ultimately allow us to just skip the more external group of parenthesis when generating the nodes. This makes valid all these expressions:

         with (manager() as x, manager() as y):
         with (manager() as x, manager() as y,):
         with (manager()):
         with (manager() as x):
         with (((manager()))):
         with ((((manager()))) as x):

but not this one:

         with (((manager()))) as x:

the reason is that it assigns the first LPAR to the second production and it fails when searching for the one that is at the end. I think this limitation is OK.

If you want to play with that. here is a prototype of the implementation with some tests:
msg326648 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-09-28 17:04
The DFA for the rule

with_stmt: 'with' ( with_item (',' with_item)* | '(' with_item (',' with_item)* [','] ')' ) ':' suite


DFA for with_stmt                                                                                                                                 [512/2103]
  State 0
    'with' -> 1
  State 1
    '(' -> 2
    with_item -> 3
  State 2
    with_item -> 4
  State 3
    ',' -> 5
    ':' -> 6
  State 4
    ')' -> 7
    ',' -> 8
  State 5
    with_item -> 3
  State 6
    suite -> 10
  State 7
    ':' -> 6
  State 8
    ')' -> 7
    with_item -> 4
  State 9
    ',' -> 5
    ':' -> 6
  State 10 (final)
  State 11
    ')' -> 7
    ',' -> 8

It works because the transition from State 1 into a "(" is going to prioritize the path:

0 -> 1 -> "(" -> 2

instead if 

0 -> 1 -> with_item -> 3
msg326727 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-09-30 14:52
Now that I think about this a bit better, this may be actually a problem as:

         with (yield something) as x:

is a more than valid use case that will be broken with the simple grammar rule :(
msg327875 - (view) Author: thautwarm (thautwarm) * Date: 2018-10-17 08:52
How about:

with_stmt: 'with' (with_items | '(' with_items ')') ':' suite
with_items: with_item (ignored* ',' ignored* with_item)*
msg346137 - (view) Author: Terry Davis (Terry Davis) Date: 2019-06-20 17:00
I'd like to re-raise this issue. Should I cross-post to " - Ideas"?
msg346157 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-06-20 18:43
What do you mean with re-raise? The issue is not closed. If you have some proposal to overcome the limitations, the best approach is to comment here in the issue.
msg347025 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-07-01 14:48
Reviewing the thread, we never actually commented on thautwarm's proposal in that aims to strip out any INDENT, NEWLINE, and DEDENT tokens that appear between the opening "with" keyword and the statement header terminating ":".

The problem with that is that line continuations are actually handled by the tokenizer, *not* the compiler, and the tokenizer already switches off the INDENT/NEWLINE/DEDENT token generation based on the following rules:

* tracking opening & closing of triple-quoted strings 
* tracking opening & closing of parentheses ("()"), brackets ("[]"), and braces ("{}")
* detecting a backslash immediately followed by a newline

By design, the tokenizer is generally unaware of which NAME tokens are actually keywords - it's only aware of async & await at the moment as part of the backwards compatibility dance that allowed those to be gradually converted to full keywords over the course of a couple of releases.

Hence why INDENT/NEWLINE/DEDENT never appear inside expressions in the Grammar: the tokenization rules mean that those tokens will never appear in those locations.

And it isn't simply a matter of making the tokenizer aware of the combination of "with" and ":" as a new pairing that ignores linebreaks between them, as ":" can appear in many subexpressions (e.g. lambda functions, slice notation, and the new assignments expressions), and it's only the full parser that has enough context to tell which colon is the one that actually ends the statement header.

Thus the design requirement is to come up with a grammar rule that allows this existing code to continue to compile and run correctly:

>>> from contextlib import nullcontext
>>> with (nullcontext()) as example:
...     pass

While also enabling new code constructs like the following:

    with (nullcontext() as example):

    with (nullcontext(), nullcontext()):

    with (nullcontext() as example, nullcontext()):

    with (nullcontext(), nullcontext() as example):

    with (nullcontext() as example1, nullcontext() as example2):

If we can get the Grammar to allow those additional placements of parentheses, then the existing tokenizer will take care of the rest.
msg363755 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-09 17:17
If we introduce a PEG-based parser, we can do this without hacking the tokenizer. See

I'd propose to aim for Python 3.10 (if the PEG parser happens).
msg372383 - (view) Author: thautwarm (thautwarm) * Date: 2020-06-25 18:36
I can confirm Guido's words, now parentheses for continuation across lines are already supported.

Even without parentheses, multiline with items can be supported. I just implemented it here:

  from contextlib import contextmanager

  def f(x):
        yield x

# Ok
  with f('c') as a,
       f('a') as b:

# Ok
  with f('c') as a,
       f('a') as b,
       f('a') as c:

  with f('c') as a,
       f('a') as b,
       f('a') as c:
     x = 1 + 1

  # message:
    File "/home/thaut/github/cpython/../", line 49
      x = 1 + 1
    IndentationError: unindent does not match any outer indentation 

  with f('c') as a,
       f('a') as b,
       f('a') as c:
          x = 1 + 1

  File "/home/thaut/github/cpython/../", line 49
    x = 1 + 1
  IndentationError: unexpected indent

The grammar is:

    | ...
    | 'with' a=(',' [NEWLINE ~ INDENT?]).with_item+ ':' tc=[TYPE_COMMENT] NEWLINE b=statements DEDENT {
        _Py_With(a, b, NEW_TYPE_COMMENT(p, tc), EXTRA) }
    | ...

The restriction here is, since the second 'with_item', until the end of 'statements', the expression and statements have to keep the same indentation.

    with item1,
    The indentation of 'item2', ..., 'block' should be the same.

This implementation leverages the new PEG and how the lexer deals with indent/dedent.
msg372384 - (view) Author: thautwarm (thautwarm) * Date: 2020-06-25 18:38
Maybe you should close this.
msg372385 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-25 19:11
This is already implemented in master with the new PEG parser so closing this.
msg372388 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-06-25 21:32
But it is undocumented and doesn’t work with -X oldparser.
--Guido (mobile)
Date User Action Args
2021-11-04 14:27:18eryksunsetmessages: - msg405694
2021-11-04 14:27:04eryksunsetnosy: - terry.reedy, ahmedsayeed1982
components: + Interpreter Core, - IDLE
2021-11-04 12:09:42ahmedsayeed1982setversions: + Python 3.8
nosy: + ahmedsayeed1982, terry.reedy, - gvanrossum, barry, georg.brandl, ishimoto, ncoghlan, benjamin.peterson, ezio.melotti, eric.araujo, steven.daprano, r.david.murray, lukasz.langa, Julian, serhiy.storchaka, ulope, Anthony Sottile, pablogsal, thautwarm, BTaskaya, Terry Davis, Jeffrey.Kintscher, jack1142

messages: + msg405694

components: + IDLE, - Interpreter Core
2020-06-25 21:32:18gvanrossumsetmessages: + msg372388
2020-06-25 19:11:20pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg372385

stage: patch review -> resolved
2020-06-25 18:38:11thautwarmsetmessages: + msg372384
2020-06-25 18:36:35thautwarmsetmessages: + msg372383
2020-03-09 17:17:12gvanrossumsetversions: - Python 3.8
2020-03-09 17:17:00gvanrossumsetnosy: + gvanrossum
messages: + msg363755
2020-02-07 18:36:25jack1142setnosy: + jack1142
2020-01-07 09:08:20BTaskayasetnosy: + BTaskaya
2019-07-04 05:06:43Jeffrey.Kintschersetnosy: + Jeffrey.Kintscher
2019-07-01 14:48:03ncoghlansetmessages: + msg347025
2019-06-20 18:43:35pablogsalsetmessages: + msg346157
2019-06-20 17:00:42Terry Davissetnosy: + Terry Davis
messages: + msg346137
2018-10-17 08:52:31thautwarmsetnosy: + thautwarm
messages: + msg327875
2018-10-12 16:48:19Anthony Sottilesetnosy: + Anthony Sottile
2018-09-30 14:52:25pablogsalsetmessages: + msg326727
2018-09-30 14:50:42pablogsalsetpull_requests: - pull_request9036
2018-09-30 14:45:48pablogsalsetkeywords: + patch
pull_requests: + pull_request9036
2018-09-28 17:04:44pablogsalsetmessages: + msg326648
2018-09-28 16:36:05pablogsalsetmessages: + msg326645
2018-09-28 14:48:19ncoghlansetmessages: + msg326639
2018-09-28 13:50:56lukasz.langasetassignee: pablogsal

nosy: + pablogsal
2018-09-28 13:50:45lukasz.langasetstatus: closed -> open

versions: + Python 3.8, - Python 3.4
nosy: + lukasz.langa

messages: + msg326637
resolution: wont fix -> (no value)
stage: patch review
2017-05-14 21:20:47ulopesetnosy: + ulope
messages: + msg293660
2015-02-17 02:40:51barrysetstatus: open -> closed
resolution: wont fix
2015-02-17 02:40:13barrysetmessages: + msg236121
2015-02-16 01:09:40steven.dapranosetnosy: + steven.daprano
messages: + msg236083
2013-01-24 08:01:18serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg180515
2013-01-24 07:58:36ezio.melottisetmessages: + msg180514
2013-01-24 06:40:31ishimotosetnosy: barry, georg.brandl, ishimoto, ncoghlan, benjamin.peterson, ezio.melotti, eric.araujo, r.david.murray, Julian
messages: + msg180512
2012-10-31 17:14:59ezio.melottisetnosy: + barry

versions: + Python 3.4, - Python 3.3
2012-08-09 23:52:19ishimotosetnosy: + ishimoto
2011-08-22 21:28:12Juliansetmessages: + msg142750
2011-08-22 02:18:02ncoghlansetmessages: + msg142658
2011-08-21 16:30:40r.david.murraysetnosy: + r.david.murray
messages: + msg142631
2011-08-20 19:01:26ezio.melottisetnosy: + ezio.melotti
2011-08-20 18:27:28georg.brandlsetnosy: + georg.brandl
messages: + msg142550
2011-08-20 18:22:56eric.araujosetnosy: + eric.araujo
messages: + msg142549
2011-08-19 13:10:48pitrousetnosy: + ncoghlan, benjamin.peterson

type: behavior -> enhancement
versions: - Python 2.7
2011-08-19 02:10:17Juliancreate