AsyncExitStack.enter_async_context() is mishandling exception context #88760

belm0 · 2021-07-09T23:59:54Z

BPO	44594
Nosy	@ncoghlan, @belm0, @njsmith, @miss-islington, @belm0
PRs	bpo-44594: fix (Async)ExitStack handling of __context__ #27089 [3.10] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) #28730 [3.9] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) #28731

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-14.14:05:30.292>
created_at = <Date 2021-07-09.23:59:53.594>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'AsyncExitStack.enter_async_context() is mishandling exception __context__'
updated_at = <Date 2022-01-14.14:05:30.291>
user = 'https://github.com/belm0'

bugs.python.org fields:

activity = <Date 2022-01-14.14:05:30.291>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-14.14:05:30.292>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2021-07-09.23:59:53.594>
creator = 'John Belmonte'
dependencies = []
files = []
hgrepos = []
issue_num = 44594
keywords = ['patch']
message_count = 8.0
messages = ['397230', '397271', '397284', '397285', '397289', '403123', '403197', '403198']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'jbelmonte', 'njs', 'David Hoyes', 'miss-islington', 'John Belmonte']
pr_nums = ['27089', '28730', '28731']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44594'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

belm0 · 2021-07-09T23:59:54Z

Over at the Trio project, we have evidence that AsyncExitStack.enter_async_context(foo()) is not actually equivalent to async with foo() regarding raised exception context.

The symptom is a very long, unhelpful tracebacks because the __context__ of raised exceptions is not set to the expected object.

python-trio/trio#2001

I can't speak to this solution myself, but njsmith suggests this amendment to contextlib:

    saved_context = exc_details[1].__context__
        try:
            raise exc_details[1]
        finally:
            exc_details[1].__context__ = saved_context

belm0 · 2021-07-12T00:56:20Z

demonstrating the difference for async case:

    import contextlib
    import trio
    
    async def background():
        assert False
    
    async def main1():
        async with trio.open_nursery() as nursery:
            nursery.start_soon(background)
            await trio.sleep_forever()
    
    async def main2():
        async with contextlib.AsyncExitStack() as stack:
            nursery = await stack.enter_async_context(trio.open_nursery())
            nursery.start_soon(background)
            await trio.sleep_forever()
    
    try:
        trio.run(main1)
    except BaseException as e:
        print('main1, context:', e.__context__)
    
    try:
        trio.run(main2)
    except BaseException as e:
        print('main2, context:', e.__context__)

main1, context: None
main2, context: Cancelled

belm0 · 2021-07-12T06:31:57Z

To clarify the problem case, I believe the discrepancy is seen when raising exceptions as follows:

exc = foo()
try:
    raise exc
finally:
    exc.__context__ = None

(From my understanding, Trio has valid use cases for doing this since it wants to control complex exception chaining, and this is beyond the scope of __suppress_context__.)

Neither ExitStack nor AsyncExcitStack are preserving the None context in the case above.

=== with statement ===

  Traceback (most recent call last):
    File "exit_stack_test.py", line 251, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException

=== enter_context() ===

  Traceback (most recent call last):
    File "exit_stack_test.py", line 240, in my_cm
      yield
    File "exit_stack_test.py", line 259, in <module>
      assert False
  AssertionError

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "exit_stack_test.py", line 259, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 524, in __exit__
      raise exc_details[1]
    File "/.../python3.7/contextlib.py", line 509, in __exit__
      if cb(*exc_details):
    File "/.../python3.7/contextlib.py", line 377, in _exit_wrapper
      return cm_exit(cm, exc_type, exc, tb)
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException

belm0 · 2021-07-12T06:33:46Z

[reposting the example, with source]

example:
class MyException(Exception):
pass

  @contextmanager
  def my_cm():
      try:
          yield
      except BaseException:
          exc = MyException()
          try:
              raise exc
          finally:
              exc.__context__ = None

  print('\n=== `with` statement ===')
  try:
      with my_cm():
          assert False
  except BaseException as e:
      traceback.print_exc()

print('\n=== enter_context() ===')
try:
with ExitStack() as stack:
stack.enter_context(my_cm())
assert False
except BaseException as e:
traceback.print_exc()

output:
=== with statement ===

  Traceback (most recent call last):
    File "exit_stack_test.py", line 251, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException

=== enter_context() ===

  Traceback (most recent call last):
    File "exit_stack_test.py", line 240, in my_cm
      yield
    File "exit_stack_test.py", line 259, in <module>
      assert False
  AssertionError

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "exit_stack_test.py", line 259, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 524, in __exit__
      raise exc_details[1]
    File "/.../python3.7/contextlib.py", line 509, in __exit__
      if cb(*exc_details):
    File "/.../python3.7/contextlib.py", line 377, in _exit_wrapper
      return cm_exit(cm, exc_type, exc, tb)
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException

belm0 · 2021-07-12T07:48:36Z

cc: ncoghlan for help with ExitStack exception context

njsmith · 2021-10-04T06:50:03Z

New changeset e6d1aa1 by John Belmonte in branch 'main':
bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089)
e6d1aa1

miss-islington · 2021-10-05T06:21:47Z

New changeset 872b1e5 by John Belmonte in branch '3.10':
[3.10] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28730)
872b1e5

miss-islington · 2021-10-05T06:37:31Z

New changeset 7c2a040 by John Belmonte in branch '3.9':
[3.9] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28731)
7c2a040

belm0 mannequin added the stdlib Python modules in the Lib dir label Jul 9, 2021

belm0 mannequin added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes type-bug An unexpected behavior, bug, or error labels Oct 1, 2021

iritkatriel removed 3.7 (EOL) end of life 3.8 only security fixes labels Jan 14, 2022

iritkatriel closed this as completed Jan 14, 2022

iritkatriel removed the 3.8 only security fixes label Jan 14, 2022

iritkatriel closed this as completed Jan 14, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

This was referenced Feb 24, 2023

Trio + standard contextlib tools can create an exception __context__ cycle -> interpreter hang python-trio/trio#2577

Closed

(Async)ExitStack can lose __context__ when its __exit__ runs without an active exception #102201

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AsyncExitStack.enter_async_context() is mishandling exception context #88760

AsyncExitStack.enter_async_context() is mishandling exception context #88760

belm0 mannequin commented Jul 9, 2021

belm0 mannequin commented Jul 9, 2021

belm0 mannequin commented Jul 12, 2021

belm0 mannequin commented Jul 12, 2021

belm0 mannequin commented Jul 12, 2021

belm0 mannequin commented Jul 12, 2021

njsmith commented Oct 4, 2021

miss-islington commented Oct 5, 2021

miss-islington commented Oct 5, 2021

AsyncExitStack.enter_async_context() is mishandling exception __context__ #88760

AsyncExitStack.enter_async_context() is mishandling exception __context__ #88760

Comments

belm0 mannequin commented Jul 9, 2021

belm0 mannequin commented Jul 9, 2021

belm0 mannequin commented Jul 12, 2021

belm0 mannequin commented Jul 12, 2021

belm0 mannequin commented Jul 12, 2021

belm0 mannequin commented Jul 12, 2021

njsmith commented Oct 4, 2021

miss-islington commented Oct 5, 2021

miss-islington commented Oct 5, 2021

AsyncExitStack.enter_async_context() is mishandling exception context #88760

AsyncExitStack.enter_async_context() is mishandling exception context #88760