classification
Title: Protect tasks weak set manipulation in asyncio.all_tasks()
Type: Stage: resolved
Components: asyncio Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, miss-islington, ned.deily, yselivanov
Priority: Keywords: patch

Created on 2018-10-13 08:11 by asvetlov, last changed 2018-10-20 17:07 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 9837 merged asvetlov, 2018-10-13 08:22
PR 9849 merged miss-islington, 2018-10-13 18:12
Messages (13)
msg327632 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 08:11
Traceback (most recent call last):
  File "C:\Users\max\AppData\Local\Programs\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "D:\Proj\antwork\code\antwork_backend\server\data_handlers\order\playback_hd.py", line 147, in run
    send_to_ws(channel=self.room_name, data=json.loads(msg))
  File "D:\Proj\antwork\code\antwork_backend\server\data_exchange.py", line 26, in send_to_ws
    asyncio.run(channel_layer.group_send(channel, send_data))
  File "C:\Users\max\AppData\Local\Programs\Python\Python37\lib\asyncio\runners.py", line 46, in run
    _cancel_all_tasks(loop)
  File "C:\Users\max\AppData\Local\Programs\Python\Python37\lib\asyncio\runners.py", line 54, in _cancel_all_tasks
    to_cancel = tasks.all_tasks(loop)
  File "C:\Users\max\AppData\Local\Programs\Python\Python37\lib\asyncio\tasks.py", line 38, in all_tasks
    return {t for t in _all_tasks
  File "C:\Users\max\AppData\Local\Programs\Python\Python37\lib\asyncio\tasks.py", line 38, in <setcomp>
    return {t for t in _all_tasks
  File "C:\Users\max\AppData\Local\Programs\Python\Python37\lib\_weakrefset.py", line 61, in __iter__
    for itemref in self.data:
RuntimeError: Set changed size during iteration


https://github.com/python/cpython/commit/416c1ebd9896b394790dcb4f9f035b1a44ebe9ff#commitcomment-30887909
msg327633 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 08:15
The issue is hard to reproduce, the problem happens on an implicit garbage collector run during iteration over `_all_tasks` weakset.
msg327635 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 08:29
Ned, would be nice to have this trivial bugfix in 3.7.1.

The problem happens if wakes up on weakset iteration.
The chance is very low, it leads to a flaky scary bug.
msg327636 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-10-13 08:35
3.7.1rc2 is about to be released so to add this would require retagging and remanufacturing release bits.  If the chances of it happening are "very low", I would prefer to not further delay rc2.  If you and Yury agree that the risk is low and the benefit is high, we could consider risking cherry-picking it to 3.7.1 final.  Sound OK?
msg327642 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 09:03
Let's wait for Yuri opinion.
msg327651 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-10-13 13:15
>> we could consider risking cherry-picking it to 3.7.1 final.  Sound OK?
> Let's wait for Yuri opinion.

I agree it's a pretty serious bug; basically a time bomb that can crash a perfectly fine asyncio application.  The PR itself seems to be safe to cherry-pick to 3.7.1 even after rc2.  I suggest to merge it asap though to have some time for buildbots to test it.
msg327660 - (view) Author: miss-islington (miss-islington) Date: 2018-10-13 18:12
New changeset 97cf0828727ac2a269c89c5aa09570a69a22c83c by Miss Islington (bot) (Andrew Svetlov) in branch 'master':
bpo-34970: Protect tasks weak set manipulation in asyncio.all_tasks() (GH-9837)
https://github.com/python/cpython/commit/97cf0828727ac2a269c89c5aa09570a69a22c83c
msg327662 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 19:26
New changeset 5dbb1b7df1d9ecaa6b5344028cd1777502cf5c73 by Andrew Svetlov (Miss Islington (bot)) in branch '3.7':
bpo-34970: Protect tasks weak set manipulation in asyncio.all_tasks() (GH-9837) (GH-9849)
https://github.com/python/cpython/commit/5dbb1b7df1d9ecaa6b5344028cd1777502cf5c73
msg327663 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 19:28
Ned, the fix has landed on both master and 3.7

Your turn, please.
msg327666 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-10-13 20:03
Thanks, Andrew and Yury!  Just leave the issue open as a "release blocker" for now and I will take care of it for 3.7.1 final.
msg327667 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-10-13 20:35
Cool!
Thanks, Ned
msg328167 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-10-20 16:57
New changeset 60c663c0f76c790afbed4d673c3ce264dd226b8c by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-34970: Protect tasks weak set manipulation in asyncio.all_tasks() (GH-9837) (GH-9849)
https://github.com/python/cpython/commit/60c663c0f76c790afbed4d673c3ce264dd226b8c
msg328168 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-10-20 17:07
Released in 3.7.1
History
Date User Action Args
2018-10-20 17:07:37ned.deilysetstatus: open -> closed
priority: release blocker ->
messages: + msg328168

resolution: fixed
stage: patch review -> resolved
2018-10-20 16:57:26ned.deilysetmessages: + msg328167
2018-10-13 20:35:54asvetlovsetmessages: + msg327667
2018-10-13 20:03:17ned.deilysetmessages: + msg327666
2018-10-13 19:28:59asvetlovsetmessages: + msg327663
2018-10-13 19:26:52asvetlovsetmessages: + msg327662
2018-10-13 18:12:59miss-islingtonsetstage: patch review
pull_requests: + pull_request9219
2018-10-13 18:12:43miss-islingtonsetnosy: + miss-islington
messages: + msg327660
2018-10-13 13:15:23yselivanovsetmessages: + msg327651
2018-10-13 09:03:22asvetlovsetmessages: + msg327642
2018-10-13 08:35:21ned.deilysetmessages: + msg327636
2018-10-13 08:29:11asvetlovsetmessages: + msg327635
stage: patch review -> (no value)
2018-10-13 08:22:49asvetlovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request9211
2018-10-13 08:15:59asvetlovsetmessages: + msg327633
2018-10-13 08:13:48asvetlovsettitle: Protect all_tasks manipulation in asyncio.all_tasks() -> Protect tasks weak set manipulation in asyncio.all_tasks()
2018-10-13 08:11:44asvetlovcreate