This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: ThreadPoolExecutor unbalanced semaphore count
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.11
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, bennieswart, pitrou, tinchester, yselivanov
Priority: normal Keywords: patch

Created on 2021-05-20 11:51 by bennieswart, last changed 2022-04-11 14:59 by admin.

File name Uploaded Description Edit bennieswart, 2021-05-20 11:51
Pull Requests
URL Status Linked Edit
PR 26265 open bennieswart, 2021-05-20 12:04
Messages (3)
msg394017 - (view) Author: Bennie Swart (bennieswart) * Date: 2021-05-20 11:51
The concurrent.futures.ThreadPoolExecutor class, which is the default asyncio executor, introduced the _idle_semaphore field in version 3.8 in order to track idle threads so they can be reused before increasing the pool size unnecessarily.
This semaphore counter becomes unbalanced when the thread pool is over-saturated, as can be seen in the file provided. This is due to workers always incrementing the count after finishing a job, whereas the executor only decrements the count if it is already greater than 0.
This seems to be a logic bug unrelated to the running environment and introduced since python 3.8.
msg400768 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2021-08-31 18:38
To be clear, this probably doesn't have any actual consequence, since the right number of threads is launched anyway (at least in the example). But it's probably worth making the implementation less quirky (also, the semaphore's internal counter *might* overflow at some point?).
msg400967 - (view) Author: Tin Tvrtković (tinchester) * Date: 2021-09-02 23:11
I was trying to instrument one of our executors to expose some metrics, so I tried using the value of the semaphore as the number of idle threads and I noticed it was way too large. If this was fixed getting these metrics would be easier.
Date User Action Args
2022-04-11 14:59:45adminsetgithub: 88354
2021-09-02 23:11:57tinchestersetnosy: + tinchester
messages: + msg400967
2021-08-31 18:38:49pitrousetnosy: + pitrou
messages: + msg400768
components: + Library (Lib), - asyncio
2021-08-31 18:10:27pitrousetversions: + Python 3.11, - Python 3.8
2021-05-20 12:04:14bennieswartsetkeywords: + patch
stage: patch review
pull_requests: + pull_request24869
2021-05-20 11:51:36bennieswartcreate