I initially reported this on launchpad at https://bugs.launchpad.net/bugs/1782647.
I'm running a test for a project that hangs and requires a Control-C to cancel it. The results look like this: https://paste.ubuntu.com/p/SwXsCcghjt/
In narrowing down on this some more, the use of queue.SimpleQueue() seems to make the difference that causes the hang. If I switch that back to queue.Queue() (which was the case for py3.6) the hang goes away:
- self._work_queue = queue.Queue()
+ self._work_queue = queue.SimpleQueue()
To reproduce with OpenStack designate:
git clone https://github.com/openstack/designate.
Then, update tox.ini with the following:
--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
[tox]
minversion = 2.0
-envlist = py35,py27,flake8
+envlist = py35,py37,py27,flake8
skipsdist = True
[testenv]
@@ -39,6 +39,12 @@ commands =
{[testenv]commands}
stestr run '{posargs}'
+[testenv:py37]
+basepython = python3.7
+commands =
+ {[testenv]commands}
+ stestr run 'designate\.tests\.test_workers\.test_processing\.TestProcessingExecutor\.(test_execute_multiple_tasks)'
+
[testenv:docs]
basepython = python3
deps =
And run: tox -e py37
|
I looked into the code of test_execute_multiple_tasks and designate/worker/processing.py from which the executor was initialized. I have converted the test case code removing the project related code as a standalone script. I hope the attached code is the minimal code required. The change from Queue to SimpleQueue was made with commit : ab74504346a6e2569b3255b7b621c589716888c4 and relevant issue is https://bugs.python.org/issue32576 .
I ran the test code by checking out each commit obtained by `git log --oneline --format="%h" Lib/concurrent/futures/thread.py` . The tests finish successfully without any hanging as below
Testing on commit : c4b695f
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012586116790771484fs
Testing on commit : ab74504
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013058185577392578fs
Testing on commit : 63ff413
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012793540954589844fs
Testing on commit : bc61315
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013325214385986328fs
Testing on commit : a3d91b4
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012831687927246094fs
Testing on commit : 15f44ab
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013515949249267578fs
Testing on commit : 50abe87
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0020570755004882812fs
Testing on commit : cfd4661
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012753009796142578fs
Testing on commit : 20efceb
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012905597686767578fs
Testing on commit : 6b97374
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.001275777816772461fs
Testing on commit : 04842a8
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0014278888702392578fs
Testing on commit : 27be5da
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013408660888671875fs
Testing on commit : c13d454
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0014138221740722656fs
Testing on commit : 81c4d36
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013227462768554688fs
Can you please check if I am correct on the above approach?
Thanks
|
Thanks, can confirm it on Python 3.7 . Unfortunately I couldn't test it with master since I have some problems setting up virtualenv and pip with the compiled binary. Running it on Python 3.7 hangs
(bpo34173-env) ➜ cpython git:(master) ✗ python
Python 3.7.0 (default, Jun 28 2018, 02:32:19)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
(bpo34173-env) ➜ cpython git:(master) ✗ python bpo34173-recreate.py
^CException in worker
Traceback (most recent call last):
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
work_item = work_queue.get(block=True)
KeyboardInterrupt
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 14.368564367294312fs
Thanks
|