This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: traceback.clear_frames manages to deadlock a background task
Type: behavior Stage:
Components: asyncio Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, tvoinarovskyi, yselivanov
Priority: normal Keywords:

Created on 2019-01-16 10:34 by tvoinarovskyi, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
test_async_deadlock.py tvoinarovskyi, 2019-01-16 10:34 Script to reproduce issue
Messages (4)
msg333759 - (view) Author: Taras Voinarovskyi (tvoinarovskyi) * Date: 2019-01-16 10:34
My use case:
I have a background task, say called "coordination". In that task, I want to catch any errors and push those to the user waiting in the main task and only continue running the background coroutine after the user manually resolves the exception. 

Issue:
When testing the behaviour with ``unittest.Case`` and using ``assertRaises`` to catch the exception, the background coroutine manages to just freeze. I have narrowed it down to ``traceback.clear_frames`` in ``assertRaises`` that causes a GeneratorExit in the background coroutine.

I believe this issue is a duplicate to https://bugs.python.org/issue29211, but wanted to provide another actual use case where it can pop up. Also even if the generator raises a GeneratorExit, why did the background thread freeze is still a mystery to me.

Script to reproduce in my case is attached.
msg333770 - (view) Author: Taras Voinarovskyi (tvoinarovskyi) * Date: 2019-01-16 14:25
So yes, the `clear_frames` function will force a running generator to close. See https://github.com/python/cpython/blob/3.7/Objects/frameobject.c#L566, it explicitly does a Finalize. Would that be the desired behaviour for assertRaises is not clear. I find it strange that catching an exception is closing my running coroutine.

The reproduce example can be lowered to something like::

    import asyncio


    async def background(error_future):
        try:
            raise ValueError
        except Exception as exc:
            error_future.set_exception(exc)

        await asyncio.sleep(1)


    async def main():
        loop = asyncio.get_event_loop()
        error_future = loop.create_future()
        task = asyncio.create_task(background(error_future))

        await asyncio.wait([error_future])
        exc = error_future.exception()
        import traceback
        traceback.clear_frames(exc.__traceback__)

        # Will block forever, as task will never be waken up
        await task


    if __name__ == "__main__":
        asyncio.run(main())
msg333775 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2019-01-16 17:02
Very interesting. Thanks for reporting this issue. I'll definitely take a look at this before 3.8 is released.  For 3.7 you'll likely have to find a workaround.
msg333808 - (view) Author: Taras Voinarovskyi (tvoinarovskyi) * Date: 2019-01-17 00:50
For now, it seems like ``copy.copy(exception)`` before raising seems to prevent this behaviour. So for all exceptions originating from background tasks I raise a copy to the user, rather than the original exception. It prints correct stack, so no real impact on the library.
History
Date User Action Args
2022-04-11 14:59:10adminsetgithub: 79932
2019-01-17 00:50:59tvoinarovskyisetmessages: + msg333808
2019-01-16 17:02:24yselivanovsetmessages: + msg333775
2019-01-16 14:25:48tvoinarovskyisetmessages: + msg333770
2019-01-16 10:34:07tvoinarovskyicreate