Author dzhu
Recipients dzhu
Date 2018-11-17.01:06:37
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1542416799.14.0.788709270274.issue35267@psf.upfronthosting.co.za>
In-reply-to
Content
The attached snippet causes a deadlock just about every time it's run (tested with 3.6.7/Ubuntu, 3.7.1/Arch, 3.6.7/OSX, and 3.7.1/OSX -- deadlock seems to be less frequent on the last, but still common). The issue appears to be something like the following sequence of events:

1. The main thread calls pool.__exit__, eventually entering Pool._terminate_pool.
2. result_handler's state is set to TERMINATE, causing it to stop reading from outqueue.
3. The main thread, in _terminate_pool, joins on worker_handler, which is (usually) in the middle of sleeping for 0.1 seconds, opening a window for the next two steps to occur.
4. The worker process finishes its task and acquires the shared outqueue._wlock.
5. The worker attempts to put the result into outqueue, but its pickled form is too big to fit into the buffer of os.pipe, and it blocks here with the lock held.
6. worker_handler wakes up and exits, freeing _terminate_pool to continue.
7. _terminate_pool terminates the worker.
8. task_handler tries to put None into outqueue, but blocks, since the lock was acquired by the terminated worker.
9. _terminate_pool joins on task_handler, and everything is deadlocked.
History
Date User Action Args
2018-11-17 01:06:39dzhusetrecipients: + dzhu
2018-11-17 01:06:39dzhusetmessageid: <1542416799.14.0.788709270274.issue35267@psf.upfronthosting.co.za>
2018-11-17 01:06:38dzhulinkissue35267 messages
2018-11-17 01:06:37dzhucreate