This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ThreadPoolExecutor is busy-waiting when idle.
Type: resource usage Stage:
Components: Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Avraham Mahfuda, kmaork, mark.dickinson
Priority: normal Keywords:

Created on 2020-02-05 07:38 by Avraham Mahfuda, last changed 2022-04-11 14:59 by admin.

Messages (6)
msg361410 - (view) Author: Avraham Mahfuda (Avraham Mahfuda) Date: 2020-02-05 07:38
In concurrent.futures.thread line 78 is busy-waiting if the queue is empty, which may cause the CPU to spin to 100% when idle.
msg361411 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2020-02-05 08:00
On the HEAD of the current 3.7 branch, the file you refer to has the following code starting at line 37:

        while True:
            work_item = work_queue.get(block=True)
            if work_item is not None:
                work_item.run()
                ...

Code link: https://github.com/python/cpython/blob/6ba8dc6aae6fa0a7e29ba4ac18227beb38872392/Lib/concurrent/futures/thread.py#L77-L80

Is that the code you're referring to?

If that's not the code you mean, please could you say exactly which version of Python you're looking at? (Code gets added and removed over time, so a line number isn't all that helpful without additional information).
msg361412 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2020-02-05 08:04
It would also be helpful to know which platform you're observing the 100% CPU usage on, so that we can try to reproduce.
msg361413 - (view) Author: Avraham Mahfuda (Avraham Mahfuda) Date: 2020-02-05 08:14
1. I can confirm that the posted code above is the correct one.
2. The 100% CPU was observed in a docker instance using python:3.7.0-alpine3.8 image.
3. The docker image was running on a 2 cores VM.
msg361415 - (view) Author: Avraham Mahfuda (Avraham Mahfuda) Date: 2020-02-05 08:23
If I understand the following code correctly, there seems to be a busy loop waiting for the SimpleQueue.put to signal that an item was entered to queue.
https://github.com/python/cpython/blob/3.7/Modules/_queuemodule.c#L217
msg362350 - (view) Author: Maor Kleinberger (kmaork) * Date: 2020-02-20 21:00
> If I understand the following code correctly, there seems to be a busy loop waiting for the SimpleQueue.put to signal that an item was entered to queue.

This is not a busywait. Actually, the loop you specified will never repeat more than twice. At the first iteration, the queue's lock is acquired. At the second iteration, the loop waits for the lock to be released (by trying to acquire the lock again, which is not busywait). The lock is only released when something is put into the queue, so at that point the loop exits and the function returns.

Avraham, could you provide a minimal python script that will reproduce the CPU usage spike?
History
Date User Action Args
2022-04-11 14:59:26adminsetgithub: 83738
2020-02-20 21:00:21kmaorksetnosy: + kmaork
messages: + msg362350
2020-02-05 08:23:03Avraham Mahfudasetmessages: + msg361415
2020-02-05 08:14:14Avraham Mahfudasetmessages: + msg361413
2020-02-05 08:04:17mark.dickinsonsetmessages: + msg361412
2020-02-05 08:00:49mark.dickinsonsetnosy: + mark.dickinson
messages: + msg361411
2020-02-05 07:38:18Avraham Mahfudacreate