This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: AsyncExitStack.enter_async_context() is mishandling exception __context__
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: David Hoyes, John Belmonte, jbelmonte, miss-islington, ncoghlan, njs
Priority: normal Keywords: patch

Created on 2021-07-09 23:59 by John Belmonte, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27089 merged jbelmonte, 2021-07-12 00:22
PR 28730 merged jbelmonte, 2021-10-05 05:59
PR 28731 merged jbelmonte, 2021-10-05 06:10
Messages (8)
msg397230 - (view) Author: John Belmonte (John Belmonte) Date: 2021-07-09 23:59
Over at the Trio project, we have evidence that `AsyncExitStack.enter_async_context(foo())` is not actually equivalent to `async with foo()` regarding raised exception context.

The symptom is a very long, unhelpful tracebacks because the __context__ of raised exceptions is not set to the expected object.

https://github.com/python-trio/trio/issues/2001

I can't speak to this solution myself, but njsmith suggests this amendment to contextlib:

    saved_context = exc_details[1].__context__
        try:
            raise exc_details[1]
        finally:
            exc_details[1].__context__ = saved_context
msg397271 - (view) Author: John Belmonte (John Belmonte) Date: 2021-07-12 00:56
demonstrating the difference for async case:

    import contextlib
    import trio
    
    async def background():
        assert False
    
    async def main1():
        async with trio.open_nursery() as nursery:
            nursery.start_soon(background)
            await trio.sleep_forever()
    
    async def main2():
        async with contextlib.AsyncExitStack() as stack:
            nursery = await stack.enter_async_context(trio.open_nursery())
            nursery.start_soon(background)
            await trio.sleep_forever()
    
    try:
        trio.run(main1)
    except BaseException as e:
        print('main1, context:', e.__context__)
    
    try:
        trio.run(main2)
    except BaseException as e:
        print('main2, context:', e.__context__)


----
main1, context: None
main2, context: Cancelled
msg397284 - (view) Author: John Belmonte (John Belmonte) Date: 2021-07-12 06:31
To clarify the problem case, I believe the discrepancy is seen when raising exceptions as follows:

exc = foo()
try:
    raise exc
finally:
    exc.__context__ = None

(From my understanding, Trio has valid use cases for doing this since it wants to control complex exception chaining, and this is beyond the scope of __suppress_context__.)

Neither ExitStack nor AsyncExcitStack are preserving the None context in the case above.

  === `with` statement ===
  Traceback (most recent call last):
    File "exit_stack_test.py", line 251, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException

  === enter_context() ===
  Traceback (most recent call last):
    File "exit_stack_test.py", line 240, in my_cm
      yield
    File "exit_stack_test.py", line 259, in <module>
      assert False
  AssertionError

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "exit_stack_test.py", line 259, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 524, in __exit__
      raise exc_details[1]
    File "/.../python3.7/contextlib.py", line 509, in __exit__
      if cb(*exc_details):
    File "/.../python3.7/contextlib.py", line 377, in _exit_wrapper
      return cm_exit(cm, exc_type, exc, tb)
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException
msg397285 - (view) Author: John Belmonte (John Belmonte) Date: 2021-07-12 06:33
[reposting the example, with source]

example:
  class MyException(Exception):
      pass

  @contextmanager
  def my_cm():
      try:
          yield
      except BaseException:
          exc = MyException()
          try:
              raise exc
          finally:
              exc.__context__ = None

  print('\n=== `with` statement ===')
  try:
      with my_cm():
          assert False
  except BaseException as e:
      traceback.print_exc()

  print('\n=== enter_context() ===')
  try:
      with ExitStack() as stack:
          stack.enter_context(my_cm())
          assert False
  except BaseException as e:
      traceback.print_exc()


output:
  === `with` statement ===
  Traceback (most recent call last):
    File "exit_stack_test.py", line 251, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException

  === enter_context() ===
  Traceback (most recent call last):
    File "exit_stack_test.py", line 240, in my_cm
      yield
    File "exit_stack_test.py", line 259, in <module>
      assert False
  AssertionError

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "exit_stack_test.py", line 259, in <module>
      assert False
    File "/.../python3.7/contextlib.py", line 524, in __exit__
      raise exc_details[1]
    File "/.../python3.7/contextlib.py", line 509, in __exit__
      if cb(*exc_details):
    File "/.../python3.7/contextlib.py", line 377, in _exit_wrapper
      return cm_exit(cm, exc_type, exc, tb)
    File "/.../python3.7/contextlib.py", line 130, in __exit__
      self.gen.throw(type, value, traceback)
    File "exit_stack_test.py", line 244, in my_cm
      raise exc
  MyException
msg397289 - (view) Author: John Belmonte (John Belmonte) Date: 2021-07-12 07:48
cc: ncoghlan for help with ExitStack exception context
msg403123 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2021-10-04 06:50
New changeset e6d1aa1ac65b6908fdea2c70ec3aa8c4f1dffcb5 by John Belmonte in branch 'main':
bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089)
https://github.com/python/cpython/commit/e6d1aa1ac65b6908fdea2c70ec3aa8c4f1dffcb5
msg403197 - (view) Author: miss-islington (miss-islington) Date: 2021-10-05 06:21
New changeset 872b1e537e96d0dc4ff37c738031940b5d271366 by John Belmonte in branch '3.10':
[3.10] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28730)
https://github.com/python/cpython/commit/872b1e537e96d0dc4ff37c738031940b5d271366
msg403198 - (view) Author: miss-islington (miss-islington) Date: 2021-10-05 06:37
New changeset 7c2a040a10654d67ff543a55858ba2d7a9f7eea8 by John Belmonte in branch '3.9':
[3.9] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28731)
https://github.com/python/cpython/commit/7c2a040a10654d67ff543a55858ba2d7a9f7eea8
History
Date User Action Args
2022-04-11 14:59:47adminsetgithub: 88760
2022-01-14 14:05:30iritkatrielsetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: - Python 3.7, Python 3.8
2021-10-05 06:37:31miss-islingtonsetmessages: + msg403198
2021-10-05 06:21:47miss-islingtonsetnosy: + miss-islington
messages: + msg403197
2021-10-05 06:10:32jbelmontesetpull_requests: + pull_request27079
2021-10-05 05:59:53jbelmontesetpull_requests: + pull_request27078
2021-10-04 06:50:02njssetmessages: + msg403123
2021-10-01 12:55:11jbelmontesettype: behavior
versions: + Python 3.7, Python 3.8, Python 3.9, Python 3.10, Python 3.11
2021-07-12 07:48:36John Belmontesetnosy: + ncoghlan
messages: + msg397289
2021-07-12 06:33:45John Belmontesetmessages: + msg397285
2021-07-12 06:31:57John Belmontesetmessages: + msg397284
2021-07-12 00:56:20John Belmontesetmessages: + msg397271
2021-07-12 00:22:47jbelmontesetkeywords: + patch
nosy: + jbelmonte

pull_requests: + pull_request25637
stage: patch review
2021-07-10 22:31:24David Hoyessetnosy: + David Hoyes
2021-07-09 23:59:53John Belmontecreate