classification
Title: [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread.py
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: corey.bryant, xtreak
Priority: normal Keywords:

Created on 2018-07-20 19:31 by corey.bryant, last changed 2019-01-07 14:08 by xtreak. This issue is now closed.

Files
File name Uploaded Description Edit
bpo34173.py xtreak, 2018-07-21 17:30
bpo34173-recreate.py corey.bryant, 2018-07-24 17:27
Messages (9)
msg322032 - (view) Author: Corey Bryant (corey.bryant) Date: 2018-07-20 19:31
I initially reported this on launchpad at https://bugs.launchpad.net/bugs/1782647.

I'm running a test for a project that hangs and requires a Control-C to cancel it. The results look like this: https://paste.ubuntu.com/p/SwXsCcghjt/

In narrowing down on this some more, the use of queue.SimpleQueue() seems to make the difference that causes the hang. If I switch that back to queue.Queue() (which was the case for py3.6) the hang goes away:

- self._work_queue = queue.Queue()
+ self._work_queue = queue.SimpleQueue()

To reproduce with OpenStack designate:

git clone https://github.com/openstack/designate.

Then, update tox.ini with the following:

--- a/tox.ini
+++ b/tox.ini
@@ -1,6 +1,6 @@
 [tox]
 minversion = 2.0
-envlist = py35,py27,flake8
+envlist = py35,py37,py27,flake8
 skipsdist = True

 [testenv]
@@ -39,6 +39,12 @@ commands =
   {[testenv]commands}
   stestr run '{posargs}'

+[testenv:py37]
+basepython = python3.7
+commands =
+ {[testenv]commands}
+ stestr run 'designate\.tests\.test_workers\.test_processing\.TestProcessingExecutor\.(test_execute_multiple_tasks)'
+
 [testenv:docs]
 basepython = python3
 deps =


And run: tox -e py37
msg322110 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-07-21 17:30
I looked into the code of test_execute_multiple_tasks and designate/worker/processing.py from which the executor was initialized. I have converted the test case code removing the project related code as a standalone script. I hope the attached code is the minimal code required. The change from Queue to SimpleQueue was made with commit : ab74504346a6e2569b3255b7b621c589716888c4 and relevant issue is https://bugs.python.org/issue32576 .

I ran the test code by checking out each commit obtained by `git log --oneline --format="%h" Lib/concurrent/futures/thread.py` . The tests finish successfully without any hanging as below

Testing on commit :  c4b695f
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012586116790771484fs
Testing on commit :  ab74504
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013058185577392578fs
Testing on commit :  63ff413
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012793540954589844fs
Testing on commit :  bc61315
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013325214385986328fs
Testing on commit :  a3d91b4
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012831687927246094fs
Testing on commit :  15f44ab
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013515949249267578fs
Testing on commit :  50abe87
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0020570755004882812fs
Testing on commit :  cfd4661
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012753009796142578fs
Testing on commit :  20efceb
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0012905597686767578fs
Testing on commit :  6b97374
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.001275777816772461fs
Testing on commit :  04842a8
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0014278888702392578fs
Testing on commit :  27be5da
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013408660888671875fs
Testing on commit :  c13d454
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0014138221740722656fs
Testing on commit :  81c4d36
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 0.0013227462768554688fs


Can you please check if I am correct on the above approach?

Thanks
msg322300 - (view) Author: Corey Bryant (corey.bryant) Date: 2018-07-24 13:24
Karthikeyan, thanks for taking a look. I'm also unable to recreate with your test. I'm not sure what the difference is. I'll report back if I can figure it out.
msg322317 - (view) Author: Corey Bryant (corey.bryant) Date: 2018-07-24 17:27
I've narrowed this down a bit more. It appears to be caused by eventlet patching of standard library thread modules. See new attached patch bp034173-recreate.py. I'll get a bug opened against eventlet.
msg322318 - (view) Author: Corey Bryant (corey.bryant) Date: 2018-07-24 17:40
eventlet issue opened at: https://github.com/eventlet/eventlet/issues/508
msg322319 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-07-24 17:51
Thanks, can confirm it on Python 3.7 . Unfortunately I couldn't test it with master since I have some problems setting up virtualenv and pip with the compiled binary. Running it on Python 3.7 hangs

(bpo34173-env) ➜  cpython git:(master) ✗ python
Python 3.7.0 (default, Jun 28 2018, 02:32:19)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

(bpo34173-env) ➜  cpython git:(master) ✗ python bpo34173-recreate.py
^CException in worker
Traceback (most recent call last):
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
    work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
    work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
    work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
    work_item = work_queue.get(block=True)
KeyboardInterrupt
^CException in worker
Traceback (most recent call last):
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 78, in _worker
    work_item = work_queue.get(block=True)
KeyboardInterrupt
Finished Tasks ['UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask', 'UnnamedTask']s in 14.368564367294312fs



Thanks
msg322320 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-07-24 17:54
Sorry for the noise about the title change. It seems you have edited it while I was typing my comment. There was a form error while I tried submit but I ignored it to try again and it had set my title. I have reverted back to your edit.

Thanks
msg333149 - (view) Author: Corey Bryant (corey.bryant) Date: 2019-01-07 13:46
I think we can close this issue. It was narrowed down to eventlet. Please
see: https://github.com/eventlet/eventlet/issues/508

On Tue, Jul 24, 2018 at 1:54 PM Karthikeyan Singaravelan <
report@bugs.python.org> wrote:

>
> Karthikeyan Singaravelan <tir.karthi@gmail.com> added the comment:
>
> Sorry for the noise about the title change. It seems you have edited it
> while I was typing my comment. There was a form error while I tried submit
> but I ignored it to try again and it had set my title. I have reverted back
> to your edit.
>
> Thanks
>
> ----------
> title: [3.7] possible race condition in
> /usr/lib/python3.7/concurrent/futures/thread.py -> [3.7] deadlock in
> /usr/lib/python3.7/concurrent/futures/thread.py
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue34173>
> _______________________________________
>
msg333152 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2019-01-07 14:08
Thanks for the info. I am closing it as third party.
History
Date User Action Args
2019-01-07 14:08:41xtreaksetstatus: open -> closed
resolution: third party
messages: + msg333152

stage: resolved
2019-01-07 13:46:33corey.bryantsetmessages: + msg333149
2018-07-24 17:54:48xtreaksetmessages: + msg322320
title: [3.7] possible race condition in /usr/lib/python3.7/concurrent/futures/thread.py -> [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread.py
2018-07-24 17:51:33xtreaksetmessages: + msg322319
title: [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread.py -> [3.7] possible race condition in /usr/lib/python3.7/concurrent/futures/thread.py
2018-07-24 17:50:51corey.bryantsettitle: [3.7] possible race condition in /usr/lib/python3.7/concurrent/futures/thread.py -> [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread.py
2018-07-24 17:40:53corey.bryantsetmessages: + msg322318
2018-07-24 17:27:01corey.bryantsetfiles: + bpo34173-recreate.py

messages: + msg322317
2018-07-24 13:24:51corey.bryantsetmessages: + msg322300
2018-07-21 17:30:25xtreaksetfiles: + bpo34173.py

messages: + msg322110
2018-07-21 17:03:16xtreaksetnosy: + xtreak
2018-07-20 19:31:47corey.bryantcreate