This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Can't gracefully ctrl+C multiprocessing pool on Python3 & Windows
Type: behavior Stage: test needed
Components: Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: davin, eryksun, paul.moore, pitrou, rebeccafair, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-10-10 08:35 by rebeccafair, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
bug.py rebeccafair, 2019-10-10 08:35
Messages (5)
msg354344 - (view) Author: Rebecca Fair (rebeccafair) Date: 2019-10-10 08:35
I want to be able to Ctrl+C to exit a multiprocessing Pool.map gracefully, and have made a solution based on this Stack Overflow answer: https://stackoverflow.com/questions/11312525/catch-ctrlc-sigint-and-exit-multiprocesses-gracefully-in-python

However, this solution works on Linux, and on Windows 10 with Python 2.7, but not on Windows 10 with Python 3.7. The Ctrl+C is just ignored and I have to kill the processes manually. I've attached the minimum code required to reproduce the problem.

Is this a bug, expected, or is there a workaround? I believe it might be caused by the behaviour of threading.Condition.wait() being changed in commit 7c3e577 but I don't know enough about signalling to say any more than that
msg354346 - (view) Author: Rebecca Fair (rebeccafair) Date: 2019-10-10 08:39
Sorry, I didn't realise it wouldn't link to a commit automatically, the commit is here: https://github.com/python/cpython/commit/7c3e5773954009d65520eb063621cf7724da88e3
msg354393 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-10-10 16:10
I think you're right about that change touching the right code, but I'm pretty sure it wasn't working before either. (Python 2.7 had totally different code, IIRC.)

But the good news is that we should be able to use code similar to what's in Modules/timemodule.c#L1878 (the pysleep() function) to also wait on the event we use to emulate signals on Windows. But perhaps there's a reason Antoine (nosied) didn't do that originally?
msg354413 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-10-10 22:55
> Python 2.7 had totally different code, IIRC.

In 2.7 the wait() method of threading._Condition implements the timeout with a loop the calls time.sleep() with an increasing delay from 500 microseconds up to 50 ms, and no more than the remaining time.

> But the good news is that we should be able to use code similar to 
> what's in Modules/timemodule.c#L1878 (the pysleep() function) to also 
> wait on the event we use to emulate signals on Windows. 

If Python continues to use emulated condition variables, then the kernel wait in _PyCOND_WAIT_MS can easily be adapted to include the SIGINT event. However, isn't it planned at some point to remove the emulated condition-variable implementation in favor of native condition variables? If so and PyCOND_WAIT and PyCOND_TIMEDWAIT end up calling SleepConditionVariableSRW, there's no obvious way to interrupt this to support Ctrl+C and Ctrl+Break (given a custom SIGBREAK handler) in the main thread. 

In particular, in Windows 8+, SleepConditionVariableSRW begins by spinning in user mode for a fixed number of cycles. Then it calls NtWaitForAlertByThreadId to block until alerted by NtAlertThreadByThreadId, which gets called internally by Wake[All]ConditionVariable. It wouldn't be kosher to call NtAlertThreadByThreadId directly in order to alert the main thread because it's an undocumented system call, plus it's poking into the internal implementation details of SleepConditionVariableSRW and WakeConditionVariable, which could leave the variable in a bad state.
msg389273 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-22 03:12
In bpo-29971 it was suggested to split the PyThread lock API in Windows into an "interruptible lock" API that's based on emulated condition variables and a "fast lock" API that's based on native condition variables and SRW locks. Maybe the only API change that's needed in that regard is to add PyThread_allocate_lock_ex(int interruptible), and otherwise dispatch on the lock type that's set in the PNRMUTEX.
History
Date User Action Args
2022-04-11 14:59:21adminsetgithub: 82609
2021-03-22 03:12:37eryksunsetmessages: + msg389273
versions: + Python 3.10, - Python 3.7
2019-10-10 22:55:20eryksunsetnosy: + eryksun
messages: + msg354413
2019-10-10 16:10:15steve.dowersetversions: + Python 3.8, Python 3.9
nosy: + pitrou, davin

messages: + msg354393

type: behavior
stage: test needed
2019-10-10 08:39:50rebeccafairsetmessages: + msg354346
2019-10-10 08:35:51rebeccafaircreate