This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author colesbury
Recipients colesbury
Date 2021-12-30.17:12:53
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1640884373.19.0.652074258131.issue46205@roundup.psfhosted.org>
In-reply-to
Content
The runtest_mp.py has a race condition between checking for worker.is_alive() and processing the queue that can lead to indefinite hangs.

The hang happens when the all the results from the self.output queue are processed but at least one of the workers hasn't finished exiting.

https://github.com/python/cpython/blob/8d7644fa64213207b8dc6f555cb8a02bfabeced2/Lib/test/libregrtest/runtest_mp.py#L394-L418

The main thread tries to get a result from the output queue, but the queue is empty and remains empty. Although the queue.get() operation eventually times out (after 30 seconds), the main thread does not re-check if all the workers have exited (!), but instead retries the queue.get() in the "while True" loop.

https://github.com/python/cpython/blob/8d7644fa64213207b8dc6f555cb8a02bfabeced2/Lib/test/libregrtest/runtest_mp.py#L415-L418

To reproduce, apply the below patch which introduces a small delay to more reliably trigger the hang.

curl "https://gist.githubusercontent.com/colesbury/fe3769f43dfb724c86ecbb182b1f6749/raw/e29a4eaeebb8d5252cdd66f3f8a70f7bc5fa14e7/runtest_mp.diff" | patch -p1
./python -m test test_regrtest -m test_module_from_test_autotest -v
History
Date User Action Args
2021-12-30 17:12:53colesburysetrecipients: + colesbury
2021-12-30 17:12:53colesburysetmessageid: <1640884373.19.0.652074258131.issue46205@roundup.psfhosted.org>
2021-12-30 17:12:53colesburylinkissue46205 messages
2021-12-30 17:12:53colesburycreate