Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asyncio: KeyboardInterrupt inside a coroutine causes AttributeError #66618

Closed
oconnor663 mannequin opened this issue Sep 17, 2014 · 9 comments
Closed

asyncio: KeyboardInterrupt inside a coroutine causes AttributeError #66618

oconnor663 mannequin opened this issue Sep 17, 2014 · 9 comments
Labels
topic-asyncio type-bug An unexpected behavior, bug, or error

Comments

@oconnor663
Copy link
Mannequin

oconnor663 mannequin commented Sep 17, 2014

BPO 22428
Nosy @gvanrossum, @vstinner, @1st1, @oconnor663

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-06.00:36:24.182>
created_at = <Date 2014-09-17.07:36:10.335>
labels = ['type-bug', 'expert-asyncio']
title = 'asyncio: KeyboardInterrupt inside a coroutine causes AttributeError'
updated_at = <Date 2015-01-06.00:36:24.180>
user = 'https://github.com/oconnor663'

bugs.python.org fields:

activity = <Date 2015-01-06.00:36:24.180>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-01-06.00:36:24.182>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2014-09-17.07:36:10.335>
creator = 'oconnor663'
dependencies = []
files = []
hgrepos = []
issue_num = 22428
keywords = []
message_count = 6.0
messages = ['226985', '226995', '227458', '228989', '228995', '233495']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'vstinner', 'yselivanov', 'oconnor663']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22428'
versions = ['Python 3.4']

@oconnor663
Copy link
Mannequin Author

oconnor663 mannequin commented Sep 17, 2014

The following test script prints a KeyboardInterrupt traceback (expected), but also an AttributeError traceback (unexpected):

import asyncio
@asyncio.coroutine
def main():
    raise KeyboardInterrupt
asyncio.get_event_loop().run_until_complete(main())
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.4/asyncio/base_events.py", line 203, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.4/asyncio/base_events.py", line 184, in run_forever
    self._run_once()
  File "/usr/lib/python3.4/asyncio/base_events.py", line 817, in _run_once
    handle._run()
  File "/usr/lib/python3.4/asyncio/events.py", line 39, in _run
    self._callback(*self._args)
  File "/usr/lib/python3.4/asyncio/tasks.py", line 321, in _step
    result = next(coro)
  File "/usr/lib/python3.4/asyncio/tasks.py", line 103, in coro
    res = func(*args, **kw)
  File "test.py", line 6, in main
    raise KeyboardInterrupt
KeyboardInterrupt
--- Logging error ---
Traceback (most recent call last):
--- Logging error ---
Traceback (most recent call last):
Exception ignored in: <bound method Task.__del__ of Task(<coro>)<exception=KeyboardInterrupt()>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/asyncio/futures.py", line 184, in __del__
  File "/usr/lib/python3.4/asyncio/base_events.py", line 725, in call_exception_handler
  File "/usr/lib/python3.4/logging/__init__.py", line 1296, in error
  File "/usr/lib/python3.4/logging/__init__.py", line 1402, in _log
  File "/usr/lib/python3.4/logging/__init__.py", line 1412, in handle
  File "/usr/lib/python3.4/logging/__init__.py", line 1482, in callHandlers
  File "/usr/lib/python3.4/logging/__init__.py", line 846, in handle
  File "/usr/lib/python3.4/logging/__init__.py", line 977, in emit
  File "/usr/lib/python3.4/logging/__init__.py", line 899, in handleError
  File "/usr/lib/python3.4/traceback.py", line 169, in print_exception
  File "/usr/lib/python3.4/traceback.py", line 153, in _format_exception_iter
  File "/usr/lib/python3.4/traceback.py", line 18, in _format_list_iter
  File "/usr/lib/python3.4/traceback.py", line 65, in _extract_tb_or_stack_iter
  File "/usr/lib/python3.4/linecache.py", line 15, in getline
  File "/usr/lib/python3.4/linecache.py", line 41, in getlines
  File "/usr/lib/python3.4/linecache.py", line 126, in updatecache
  File "/usr/lib/python3.4/tokenize.py", line 437, in open
AttributeError: 'module' object has no attribute 'open'

The issue is that Task._step() calls self.set_exception() for both Exceptions and BaseExceptions, but it reraises BaseExceptions. That means that the BaseEventLoop never gets to call future.result() to clear the exception when it's a BaseException. Future.__del__ eventually tries to log this uncleared exception, but something about the circumstances of program exit (caused by the same exception) has already cleaned up the builtins needed for logging.

Is that __del__ violating some best practices for finalizers by calling such a complicated function? Either way, I'm not sure this case should be considered an "unretrieved exception" at all. It's propagating outside the event loop after all. Should Task._step() be setting it in the first place, if it's going to reraise it? Should it be set without _log_traceback=True somehow?

@oconnor663 oconnor663 mannequin added topic-asyncio type-bug An unexpected behavior, bug, or error labels Sep 17, 2014
@vstinner vstinner changed the title KeyboardInterrupt inside a coroutine causes AttributeError asyncio: KeyboardInterrupt inside a coroutine causes AttributeError Sep 17, 2014
@vstinner
Copy link
Member

I see different issues in your example:

  • If the coroutine raises an exception which doesn't inherit from Exception (but inherits from BaseException), run_until_complete() doesn't consume the exception from the temporary task object

=> run_until_complete(coroutine) sets the _log_destroy_pending attribute of the temporary Task to False, but this attribute controls the "Task was destroyed but it is pending!" warning. The "%s exception was never retrieved" warning is controlled by the Future._log_traceback attribute (in Python 3.4+). This attribute is set to True in Future.set_exception().*

  • If a Task is deleted late during Python shutdown, the logging module fails to log the traceback because the builtin function has been deleted.

=> IMO it's an issue in the traceback module. It may catch the AttributeError on the call to linecache.getline(). It's not convinient to get a new exception (traceback) when trying to display a traceback... Maybe the traceback can check if Python is exiting before calling the linecache module?

  • If you call again loop.run_forever(): it exits immediatly because a call to loop.stop() was scheduled by future.set_exception()

=> I created the issue bpo-22429

I don't think that it's a bug that Task._step() calls set_exception() for BaseException. Otherwise, how do you know that a task failed?

@vstinner
Copy link
Member

The issue bpo-22480 has been marked as a duplicate of this issue.

@vstinner
Copy link
Member

=> IMO it's an issue in the traceback module. It may catch the AttributeError on the call to linecache.getline().

I created the issue bpo-22599 for this bug.

@vstinner
Copy link
Member

If the coroutine raises an exception which doesn't inherit from Exception (but inherits from BaseException), run_until_complete() doesn't consume the exception from the temporary task object

I created the issue bpo-22601 for this bug.

@vstinner
Copy link
Member

vstinner commented Jan 6, 2015

A lot of fixes has been commited to fix this general issue with asyncio at exit.

run_until_complete() doesn't log an error anymore when a BaseException (like KeyboardInterrupted) is raised. The caller is able to decide how to handle it.

The traceback module has been enhanced to try to fix the "AttributeError: 'module' object has no attribute 'open'" error (or at least reduce the risk of such error). A better solution is being developed.

The initial issue was fixed, I close the issue. Thanks for the report Jack. Sorry, I forgot to update this issue since it was splitted in many smaller and more specific issues.

@vstinner vstinner closed this as completed Jan 6, 2015
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
@Buco11-dev
Copy link

A lot of fixes has been commited to fix this general issue with asyncio at exit.

run_until_complete() doesn't log an error anymore when a BaseException (like KeyboardInterrupted) is raised. The caller is able to decide how to handle it.

The traceback module has been enhanced to try to fix the "AttributeError: 'module' object has no attribute 'open'" error (or at least reduce the risk of such error). A better solution is being developed.

The initial issue was fixed, I close the issue. Thanks for the report Jack. Sorry, I forgot to update this issue since it was splitted in many smaller and more specific issues.

Can you please point me to the fix for this issue? I'm still getting the same Task exception was never retrieved error with BaseException raised when KeyboardInterrupt is called.

@vstinner
Copy link
Member

A closed issue is the wrong place to ask questions.

@gvanrossum
Copy link
Member

@Buco11-dev If you want to, you can open a new issue. What you are seeing is unlikely caused by the same thing described in this issue. Please provide as much info as you can in your issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic-asyncio type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

3 participants