Issue 45278: RuntimeError on race on weakset concurrent iteration

➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

This issue has been migrated to GitHub: https://github.com/python/cpython/issues/89441

classification

Title:	RuntimeError on race on weakset concurrent iteration
Type:	behavior	Stage:
Components:	Library (Lib)	Versions:	Python 3.11, Python 3.10, Python 3.9

process

Status:	open	Resolution:
Dependencies:		Superseder:
Assigned To:		Nosy List:	bjs, fdrake, graingert, pitrou, serhiy.storchaka
Priority:	normal	Keywords:

Created on 2021-09-24 11:03 by bjs, last changed 2022-04-11 14:59 by admin.

Files
File name	Uploaded	Description	Edit
weakset_concurrent_iter_runtimeerror.py	bjs, 2021-09-24 11:03

Messages (1)
msg402553 - (view)	Author: Ben (bjs) *	Date: 2021-09-24 11:03
This is a very subtle race WeakSet uses _weakrefset.py's _IterationGuard structure to protect against the case where the elements the WeakSet refers to get cleaned up while a thread is iterating over the WeakSet. It defers the actual removal of any elements which get gc'd during iteration until the end of the iteration. the WeakSet keeps track of all the iterators, and waits until there are no more threads iterating over the set before it does the removal: https://github.com/python/cpython/blob/main/Lib/_weakrefset.py#L30 However there is a race, if another thread begins iterating after the `if s:` check but before the _commit_removals call has ended, that iteration can get a RuntimeError. attached is an example script that can generate such RuntimeError's, although the race window here is very small and so to observe yourself you may have to tweak the magic constants around. As far as I'm aware nobody has reported seeing this bug happen in production, but some libraries (e.g. asyncio) do currently rely on concurrently iterating a weakset, so it's not implausible.

msg402553 - (view)

Author: Ben (bjs) *

Date: 2021-09-24 11:03

This is a very subtle race

WeakSet uses _weakrefset.py's _IterationGuard structure to protect against the case where the elements the WeakSet refers to get cleaned up while a thread is iterating over the WeakSet.

It defers the actual removal of any elements which get gc'd during iteration until the end of the iteration.

the WeakSet keeps track of all the iterators, and waits until there are no more threads iterating over the set before it does the removal: https://github.com/python/cpython/blob/main/Lib/_weakrefset.py#L30

However there is a race, if another thread begins iterating after the `if s:` check but before the _commit_removals call has ended, that iteration can get a RuntimeError.

attached is an example script that can generate such RuntimeError's,
although the race window here is very small and so to observe yourself you may have to tweak the magic constants around.

As far as I'm aware nobody has reported seeing this bug happen in production,  but some libraries (e.g. asyncio) do currently rely on concurrently iterating a weakset, so it's not implausible.

History
Date	User	Action	Args
2022-04-11 14:59:50	admin	set	github: 89441
2021-09-24 11:06:46	serhiy.storchaka	set	nosy: + fdrake, pitrou, serhiy.storchaka components: + Library (Lib) versions: + Python 3.9, Python 3.10, Python 3.11
2021-09-24 11:06:08	graingert	set	nosy: + graingert
2021-09-24 11:03:08	bjs	create