Basically, nested() doesn't seem to be consistent with explicitly nested
with-statements when an exception is thrown in a CM's __enter__.

Consider a pair of nested CMs, the inner __enter__ raises an exception
trapped by the outer. In the situation of explicitly nested
with-statements, the inner's block will not be ran as an exception was
raised prior to the block. The outer traps the exception and code
continues to flow after the outer's block because the exception was
*not* raised.
Currently, if contextlib.nested() is used in such a situation, it blows
up with a RuntimeError("generator didn't yield").

See the attached file for a set of naive variations and their resulting
exceptions or lack thereof in the situation of explicitly nested CMs.

Despite the RuntimeError raised by nested(), I'm not sure it's
*currently* possible for an implementation of nested() to be *totally*
consistent with explicitly nested with-statements. It would seem that an
additional facility(AbortBlock exception?) would be needed to
communicate that the block should not actually be ran, and that no
exception should be raised as it was consumed by an "outer" CM.


If this is considered to be the intended behavior, I would think the
doc-string on contextlib.nested should be updated to document the
inconsistency as it currently states that nested() and nested with
statements are equivalent. Based on the results of the attached file in
Python 3.0, they are clearly not.

Cheers folks; lovin' CMs. And, of course, apologies if this has already
been discussed. :P

That's an interesting problem... since one of the things I like about
CMs is the ability to turn pretty much any block of code into a context
manager by substituting a yield statement at the appropriate point, not
having the ability to skip the yield is actually something of a hole in
the design.

This will need a bit of python-dev discussion, and may even turn into a
PEP (assuming we do go ahead with adding a SkipBlock exception and
modifying the with statement to check for it being raised by __enter__()
methods).

I'll be working on a PEP for 2.7/3.1 to try to get the semantics of with
statement changed as suggested.

http://mail.python.org/pipermail/python-dev/2009-February/086470.html

PEP 377 (http://www.python.org/dev/peps/pep-0377/) has now been submitted.

Attached simple benchmark script to check for any slowdowns introduced
by the planned with statement changes.

First draft of patch attached.

Limitations of this version of the patch:
- includes some unrelated marshal.c changes (improved error messages)
- no tests of the new functionality
- contextlib not changed yet
- no documentation changes

The new semantics are in place though, and the existing test_with and
test_contextlib tests all still pass.

Removed first draft of patch - it was fundamentally flawed (it didn't
clean up the stack properly when the statement body was skipped)

Second draft attached, this time with tests and sans segmentation faults :)

Limitations of this version of the patch:
- still includes some unrelated marshal.c changes (improved error messages)
- no documentation changes yet

Note that the semantics in the current patch aren't quite correct, since
the __exit__ attribute is retrieved inside the scope of the outer
try/except block. Updated patch with the correct semantics coming soon.

Just downloaded v2 and tried it out against Python 2.7a0 (trunk:70381M,
Mar 14 2009, 23:12:51).

output of the "nested_issue.py" script with patch:

jwp@torch[]:org/python/trunk 0% /src/build/py/bin/python ./nested_issue.py
()
[try_with_nested]
Skipping statement body
()
SUCCESS! i guess..
()
[try_with_nested_class]
Skipping statement body
()
SUCCESS! i guess..
()
[try_with_nested_statements]
()
SUCCESS! i guess..
()
[try_with_nested_statements_class]
()
SUCCESS! i guess..

I'm going to play with it a bit more, but it looks pretty solid already...

To see what is wrong with v2, try something like:

  class Bizarre():
    def __getattr__(self, attr):
        raise SkipStatement

  with Bizarre():
    print "Not gonna happen, but no exception either!"

Version 3 of patch attached
- now functionally complete with the correct semantics
- compiler package not updated yet
- documentation not updated yet

Unfortunately, getting the semantics right involves moving
context.__enter__ and context.__exit__ to temporary variables instead of
being able to leave them on the stack - that makes for something of a
speed hit.

Updated micro benchmark to also give numbers for direct try/finally
execution

I'm definitely not happy with the effect the current patch has on with
statement execution speed - I think I'll try to come up with a patch
that adds a new SETUP_WITH opcode and see how that performs.

I tested the attached script against v2.

It further identifies consistencies between nested with-statements and
nested() that should exist(and do in v2). It answers the question: what
is the effect of a SkipStatement exception on another, outer CM?

with nested(outer(), trap(), fail()):
 ...

That is, validate that outer()'s __exit__ invocation is not given an
exception.

This is probably an obvious effect of the patch, but I think it merited
a test.

Reclassifying as a documentation bug, since PEP 377 was rejected by Guido.

Lowering priority due to implementation of explicit syntax for multiple
context managers in a single with statement

R.I.P contextlib.nested (it is gone in 3.2 following the deprecation in 3.1). The issue is obscure enough that I don't see much value in updating the documentation for the versions that still contain it in deprecated form.

Reopening as a reminder to myself that I have a new PEP I want to write in this area. The idea is essentially a lighter weight alternative to PEP 377 that adds an optional __entered__ method to the context management protocol along the following lines:

_v = cm.__enter__()
try:
  if hasattr(cm, "__entered__"):
    VAL = cm.__entered__(_v)
  else:
    VAL = _v
  # do stuff
finally:
  cm.__exit__(*exception_status)

Providing a second, optional method that is executed *inside* the body will let CMs do things they can't do now (like skip the body of the with statement) without significantly affecting the behaviour of normal CMs. Notably, GeneratorContextManager will be able to use this to more gracefully handle the case where the generator doesn't yield a value.

I plan to flesh this out into a python-ideas post (and likely a subsequent PEP) at some point in the next few months.

Superseded by issue 13585 (which will add an improved dynamic context management API)