classification
Title: Race condition in test_queue can lead to test failures
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: colesbury, lukasz.langa, miss-islington, pitrou
Priority: normal Keywords: patch

Created on 2021-11-17 21:57 by colesbury, last changed 2021-11-18 16:21 by lukasz.langa. This issue is now closed.

Files
File name Uploaded Description Edit
issue45835_repro.patch colesbury, 2021-11-17 21:59
Pull Requests
URL Status Linked Edit
PR 29601 merged colesbury, 2021-11-17 22:29
PR 29612 merged miss-islington, 2021-11-18 08:51
PR 29613 merged miss-islington, 2021-11-18 08:51
Messages (5)
msg406498 - (view) Author: Sam Gross (colesbury) * (Python triager) Date: 2021-11-17 21:57
The test_queue suite has a race condition that can lead to test failures in test_many_threads, test_many_threads_nonblock, and test_many_threads_timeout. Consumers are signaled to exit by a sentinel value (None). The sentinel values are at the end of the input list, but that doesn't mean they are necessarily enqueued at the end of the inter-thread queue when there are multiple "feeder" threads.

In particular, a feeder thread may be delayed in enqueueing a non-sentinel value. The other feeder threads may finish popping and enqueueing the remaining values including all the sentinels, leading to the delayed non-sentinel value arriving AFTER all the sentinels. The "consumer" threads exit before processing all the values leading to the assertion error in run_threads() in test_queue.py:

  self.assertTrue(q.empty())

I will attach a patch that adds a delay in feed() to make the race condition occur more frequently so that the issue is easier to reproduce.
msg406521 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-11-18 08:51
New changeset df3e53d86b2ad67da9ac2b5a3f56257d1f394982 by Sam Gross in branch 'main':
bpo-45835: Fix race condition in test_queue (#29601)
https://github.com/python/cpython/commit/df3e53d86b2ad67da9ac2b5a3f56257d1f394982
msg406545 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-18 16:05
New changeset 5cf05c71d13a4b4a94cc77da4214ee2f0b9c7de7 by Miss Islington (bot) in branch '3.10':
bpo-45835: Fix race condition in test_queue (GH-29601) (GH-29612)
https://github.com/python/cpython/commit/5cf05c71d13a4b4a94cc77da4214ee2f0b9c7de7
msg406550 - (view) Author: miss-islington (miss-islington) Date: 2021-11-18 16:16
New changeset 9450c751cc2053b1c2e03ec92ed822a41223b142 by Miss Islington (bot) in branch '3.9':
bpo-45835: Fix race condition in test_queue (GH-29601)
https://github.com/python/cpython/commit/9450c751cc2053b1c2e03ec92ed822a41223b142
msg406553 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-11-18 16:21
Thanks, Sam! ✨ 🍰 ✨
History
Date User Action Args
2021-11-18 16:21:05lukasz.langasetstatus: open -> closed
resolution: fixed
messages: + msg406553

stage: patch review -> resolved
2021-11-18 16:16:20miss-islingtonsetmessages: + msg406550
2021-11-18 16:05:49lukasz.langasetnosy: + lukasz.langa
messages: + msg406545
2021-11-18 08:51:46miss-islingtonsetpull_requests: + pull_request27849
2021-11-18 08:51:41miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request27848
2021-11-18 08:51:34pitrousetnosy: + pitrou
messages: + msg406521
2021-11-17 22:29:03colesburysetstage: patch review
pull_requests: + pull_request27844
2021-11-17 21:59:50colesburysetfiles: + issue45835_repro.patch
keywords: + patch
2021-11-17 21:57:20colesburycreate