This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Windows: SystemError during os.kill(..., signal.CTRL_C_EVENT)
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: missing return in win32_kill?
View: 14484
Assigned To: Nosy List: William.Schwartz, elsamuko, eryksun, ncoghlan, paul.moore, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-01-18 22:12 by William.Schwartz, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (14)
msg385235 - (view) Author: William Schwartz (William.Schwartz) * Date: 2021-01-18 22:12
I don't have an automated test at this time, but here's how to get os.kill to raise SystemError. I did this on Windows 10 version 20H2 (build 19042.746) with Pythons 3.7.7, 3.8.5, and 3.9.1. os_kill_impl at Modules/posixmodule.c:7833 does not appear to have been modified recently (but I haven't checked its transitive callees), so I imagine you can get the same behavior from os.kill on Python 3.10.

1. Open two consoles, A and B. (I'm using tabs in Windows Terminal, if that matters.)
2. In console A, type but DO NOT EXECUTE the following command:

python -c"import os, signal; os.kill(, signal.CTRL_C_EVENT)"

Move your cursor back before the comma.

3. In console B, create a process that does nothing but print its process identifier and wait long enough for you to type it in console A:

python -c"import os, time; print(os.getpid()); time.sleep(60); print('exiting cleanly')"

Copy or remember the printed PID. Hurry to step 4 before your 60 seconds expires!

4. In console A, type the PID from console B and execute the command.
5. In console B, confirm that the Python exited without printing "exiting cleanly". Oddly, `echo %errorlevel%` will print `0` rather than `-1073741510` (which is `2**32 - 0xc000013a`, the CTRL_C_EVENT exit code), which is what it prints after `python -c"raise KeyboardInturrupt"`.
6. In console A, you should see the following traceback.

OSError: [WinError 87] The parameter is incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
SystemError: <built-in function kill> returned a result with an error set
msg385238 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-18 23:22
> 4. In console A, type the PID from console B and execute the command.

In Windows, os.kill() is a rather confused function. In particular, it confuses a process ID (pid) with a process group ID (pgid). If the signal is CTRL_C_EVENT or CTRL_BREAK_EVENT, it first tries WinAPI GenerateConsoleCtrlEvent(). This function takes a pgid for a process group in the console session. Passing it a pid of a process in the console session that's not a pgid has undefined behavior (usually it acts like passing 0 as the pgid, but it's complicated; I have an open issue on Microsoft's GitHub repo for this that's been languishing for a couple years). Passing it a pid of a process that's not in the console session will fail as an invalid parameter. 

In the latter case, os.kill() sets an error but still tries the regular pid (not pgid) path of OpenProcess() and TerminateProcess(). If the latter succeeds, it returns success. However, it doesn't clear the error that was set as a result of the GenerateConsoleCtrlEvent() call, which causes SystemError to be raised.

>  Oddly, `echo %errorlevel%` will print `0` rather than `-1073741510` 

There's nothing odd about that. The value of CTRL_C_EVENT is 0. GenerateConsoleCtrlEvent() does not work across different console sessions. So os.kill() has simply opened a handle to the process and called TerminateProcess(handle, 0).
msg385239 - (view) Author: William Schwartz (William.Schwartz) * Date: 2021-01-18 23:41
> In Windows, os.kill() is a rather confused function.

I know how it feels.

To be honest, I don't have an opinion about whether the steps I laid out ought to work. I just reported it because the SystemError indicates that a C-API function was returning non-NULL even after PyErr_Occurred() returns true. I think that's what you're referring to here...

> it doesn't clear the error that was set as a result of the GenerateConsoleCtrlEvent() call, which causes SystemError to be raised.

...but I lost you on where that's happening and why. Frankly, Windows IPC is not in my wheelhouse.

>>  Oddly, `echo %errorlevel%` will print `0` rather than `-1073741510` 
>
> There's nothing odd about that.

Here's why I thought it was odd. The following session is from the Windows Command shell inside *Command Prompt* (not Windows Terminal):

 C:\Users\wksch>python --version
 Python 3.9.1
 C:\Users\wksch>python -c"import os, signal, time; os.kill(os.getpid(), signal.CTRL_C_EVENT); time.sleep(1)"
 Traceback (most recent call last):
   File "<string>", line 1, in <module>
 KeyboardInterrupt
 ^C
 C:\Users\wksch>echo %errorlevel%
 -1073741510

In the Windows Command shell inside *Windows Terminal* (not Command Prompt):

 C:\Users\wksch>python -c"import os, signal, time; os.kill(os.getpid(), signal.CTRL_C_EVENT); time.sleep(1)"

 C:\Users\wksch>echo %errorlevel%
 0

 C:\Users\wksch>python -c"raise KeyboardInterrupt"
 Traceback (most recent call last):
   File "<string>", line 1, in <module>
 KeyboardInterrupt
 ^C
 C:\Users\wksch>echo %errorlevel%
 -1073741510
msg385248 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-19 09:05
> I just reported it because the SystemError indicates that a C-API function
> was returning non-NULL even after PyErr_Occurred() returns true

Fixing the SystemError should be simple. Just clear an existing error if TerminateProcess() succeeds. 

> Windows Command shell inside *Command Prompt* (not Windows Terminal):

The cmd.exe shell (aka command prompt or command interpreter) is a console client application like python.exe, which attaches to a console session that's hosted by conhost.exe or openconsole.exe. The host also implements the console UI, unless it's executed in pseudoconsole mode (e.g. a tab in Windows Terminal). 

> C:\Users\wksch>python -c"import os, signal, time; os.kill(os.getpid(),
> signal.CTRL_C_EVENT); time.sleep(1)"

The above kill() call is implemented by calling GenerateConsoleCtrlEvent() to send the cancel event to clients of the console session that are members of the process group, os.getpid(). But there is no such group since Python isn't executed as a new process group. GenerateConsoleCtrlEvent() should fail with an invalid parameter error, and kill() should fall back on TerminateProcess(). But GenerateConsoleCtrlEvent() has a bug in cases like this that causes it to behave as if it were passed group ID 0 (i.e. all processes in the console session). If not for the bug in the console, this example would also raise SystemError.

> In the Windows Command shell inside *Windows Terminal*

The misbehavior is different in a pseudoconsole session, which is probably due to an unrelated bug that's affecting the expected bug. Other than Windows Terminal, another simple way to get a pseudoconsole session is to run cmd.exe from WSL. (The UI will be in the same console as WSL, but cmd.exe will actually be attached to a pseudoconsole session that's hosted by a headless instance of conhost.exe.) In pseudoconsole mode, the console will broadcast the control event only after a key such as enter or escape is pressed, which is obviously a bug in the console's input event loop. It's still a case of GenerateConsoleCtrlEvent() nominally succeeding with buggy behavior where it should fail.
msg385277 - (view) Author: William Schwartz (William.Schwartz) * Date: 2021-01-19 16:53
> Fixing the SystemError should be simple. Just clear an existing error if TerminateProcess() succeeds.

Should there be a `return NULL;` between these two lines? https://github.com/python/cpython/blob/e485be5b6bd5fde97d78f09e2e4cca7f363763c3/Modules/posixmodule.c#L7854-L7855

I'm not the best person to work on a patch for this.
msg385291 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-19 20:38
> Should there be a `return NULL;` between these two lines?

In 3.x, os.kill() has always fallen back on TerminateProcess() when GenerateConsoleCtrlEvent() fails, and I assumed that was intentional. But I misunderstood that it's not actually chaining the exception via PyErr_Fetch and _PyErr_ChainExceptions, so it still fails with a SystemError when both GenerateConsoleCtrlEvent() and TerminateProcess() fail -- e.g. os.kill(4, 1). 

I checked the history for bpo-1220212. It appears that win32_kill() was committed to the 2.x branch first with a missing return statement, which was fixed a day later:

https://github.com/python/cpython/commit/ae509520de5a0321f58c79afffad10ae59dae8b9

A few days later it was ported to 3.x without that fix:

https://github.com/python/cpython/commit/eb24d7498f3e34586fee24209f5630a58bb1a04b

If Steve Dower or Victor Stinner is okay with changing the behavior to agree with 2.x -- after more than a decade -- then I say just add the missing return statement. In that case, if os.kill() is called with a sig value of CTRL_C_EVENT (0) or CTRL_BREAK_EVENT (1), then it will only try GenerateConsoleCtrlEvent() and no longer fall back on TerminateProcess(). 

That said, it's common to call TerminateProcess() with an exit status of 1. Maybe the enum values of signal.CTRL_C_EVENT and signal.CTRL_BREAK_EVENT can be checked by identity instead of checking the integer value. That way os.kill(pid, 1) can still be used to forcefully terminate an arbitrary process with an exit status of 1.

---

To reiterate previous comments, the sig values of signal.CTRL_C_EVENT and signal.CTRL_BREAK_EVENT are not supported for an individual, arbitrary process. To use them, the pid value must be a process group ID, such as the pid of a process created with the flag CREATE_NEW_PROCESS_GROUP, and the process must be in the same console session as the current process. Anything else either fails (assuming no fallback on TerminateProcess) or invokes buggy behavior in the console, which can even leave the console in a broken state.

For a new process group, the cancel event is initially ignored, but the break event is always handled. To enable the cancel event, the process must call SetConsoleCtrlHandler(NULL, FALSE), such as via ctypes with kernel32.SetConsoleCtrlHandler(None, False). I think the signal module should provide a function to enable/disable Ctrl+C handling without ctypes, and implicitly enable it when setting a new SIGINT handler.

---
Hypothetical la la land...

I wish os.kill() in Windows had been implemented to conform with Unix kill(), and that CTRL_C_EVENT and CTRL_BREAK_EVENT were not added since they are not signals. Route negative pid values to the process-group branch that calls GenerateConsoleCtrlEvent(), and special case pid -1 to send the event to all accessible processes in the console session. Support SIGINT and SIGBREAK sig values in this case, respectively mapped to the console's cancel and break events. Don't support pid 0 in Windows, since there's no documented function to get the process group ID of the current process. Route positive pid values to the TerminateProcess() branch. Special case the sig value 0 to test for existence and access via OpenProcess(). Also, the latter should only request PROCESS_TERMINATE access, not PROCESS_ALL_ACCESS. It's common to have terminate access but not all access, e.g. to an elevated process in the current desktop session.
msg385296 - (view) Author: William Schwartz (William.Schwartz) * Date: 2021-01-19 21:10
>For a new process group, the cancel event is initially ignored, but the break event is always handled. To enable the cancel event, the process must call SetConsoleCtrlHandler(NULL, FALSE), such as via ctypes with kernel32.SetConsoleCtrlHandler(None, False). I think the signal module should provide a function to enable/disable Ctrl+C handling without ctypes, and implicitly enable it when setting a new SIGINT handler.

That's what Lib/test/win_console_handler.py:39 does. What I don't understand is why that's necessary. Isn't that what PyConfig.install_signal_handlers is supposed to do?

Which brings me to how I ended up here in the first place: I wanted to write a test that PyConfig.install_signal_handlers is set in an embedded instance of Python I'm working with. In outline, the following test works on both Windows and macOS *except on Windows running under Tox*.

@unittest.removeHandler
def test_signal_handlers_installed(self):
    SIG = signal.SIGINT
    if sys.platform == 'win32':
        SIG = signal.CTRL_C_EVENT
    with self.assertRaises(KeyboardInterrupt):
        os.kill(os.getpid(), SIG)
        if sys.platform == 'win32':
            time.sleep(.1)  # Give handler's thread time to join

Using SetConsoleCtrlHandler if I detect that I'm running on Windows under Tox would, if I understand correctly, hide whether PyConfig.install_signal_handlers was set before the Python I'm running in started, right? (I know this isn't the right venue for discussing my embedding/testing problem. But maybe the use case informs the pending discussion of what to do about os.kill's semantics.)
msg385305 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-19 22:50
> That's what Lib/test/win_console_handler.py:39 does. 

No, that script is currently broken.

SetConsoleCtrlHandler(None, False) clears the Ctrl+C ignore flag in the PEB ProcessParameters, which is normally inherited by child processes. But using the CREATE_NEW_PROCESS_GROUP creation flag always causes Ctrl+C to be initially ignored in the process. The intent is to make it a 'background' process by default, to support CMD's `start /b` option.

Win32KillTests in Lib/test/test_os.py incorrectly assumes it only has to enable Ctrl+C in the parent process. win_console_handler.py incorrectly assumes it does not have to call SetConsoleCtrlHandler(None, False). This is why test_CTRL_C_EVENT() is currently being skipped. To fix it, win_console_handler.py needs to call SetConsoleCtrlHandler(None, False).

> Isn't that what PyConfig.install_signal_handlers is supposed to do?

The C runtime's signal() function does not clear the Ctrl+C ignore flag when a SIGINT handler is set. I think Python should. It's expected behavior for POSIX compatibility.

>    with self.assertRaises(KeyboardInterrupt):
>        os.kill(os.getpid(), SIG)

Unless you know the current process was created in a new process group, it is not correct to call os.kill(os.getpid(), signal.CTRL_C_EVENT). You'd be better off using os.kill(0, signal.CTRL_C_EVENT) and ensuring that Ctrl+C is temporarily ignored in the parent process, if that's in your control. Also, Python 3.8 added raise_signal(), which calls C raise(), so a more reliable way to test just the signal handler is to call signal.raise_signal(signal.SIGINT). This also eliminates needing to synchronize with the console control thread.
msg385551 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-23 17:32
If I understand correctly, this is open for someone to evaluate Eryk's suggestion in msg385291 that a 'missing' return be added to win32_kill.
msg385552 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-23 17:44
Thanks, Terry. I should have followed up with a short message asking for a core developer to sign off on that suggestion, in which case this is an "easy (C)" issue.
msg387732 - (view) Author: El Samuko (elsamuko) Date: 2021-02-26 17:36
FWIW, I could send CTRL-C to ffmpeg (else it won't write a valid mp4 header) under Windows by using the WINPID and not the PID returned from ps .

```
WINPID=$(ps aux | grep ffmpeg | awk '{print $4}')
python -c "import os, signal; os.kill($WINPID, signal.CTRL_C_EVENT)"
```

I put an example here:
https://gist.github.com/elsamuko/9c3fe69f00a0f847251ffa3ef1d080a2
msg387734 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-26 18:04
El Samuko, is that a Windows (Win32), Cygwin, MSYS2, WSL, or some other version of bash? The same for Python, and which version of Python is it. With these details, I can tell you what `os.kill($WINPID, signal.CTRL_C_EVENT)"` is probably doing.
msg387738 - (view) Author: El Samuko (elsamuko) Date: 2021-02-26 19:30
@eryksun

I can run the script successfully under cygwin and mingw(git bash).
The python version is 3.9.2
msg387739 - (view) Author: El Samuko (elsamuko) Date: 2021-02-26 19:32
I'm getting [WinError 87], too, when using the PID.
History
Date User Action Args
2022-04-11 14:59:40adminsetgithub: 87128
2021-12-11 21:00:33eryksunsetstatus: open -> closed
superseder: missing return in win32_kill?
resolution: duplicate
stage: resolved
2021-03-08 20:14:38vstinnersetnosy: - vstinner
2021-03-01 13:47:24eryksunlinkissue33245 superseder
2021-02-26 19:32:10elsamukosetmessages: + msg387739
2021-02-26 19:30:31elsamukosetmessages: + msg387738
2021-02-26 18:04:55eryksunsetmessages: + msg387734
2021-02-26 17:36:04elsamukosetnosy: + elsamuko
messages: + msg387732
2021-01-23 17:44:54eryksunsetmessages: + msg385552
2021-01-23 17:32:22terry.reedysetnosy: + terry.reedy

messages: + msg385551
versions: - Python 3.7
2021-01-19 22:50:17eryksunsetmessages: + msg385305
2021-01-19 21:10:22William.Schwartzsetmessages: + msg385296
2021-01-19 20:38:35eryksunsetmessages: + msg385291
components: + Library (Lib), - Extension Modules
2021-01-19 16:53:41William.Schwartzsetmessages: + msg385277
2021-01-19 09:05:56eryksunsetmessages: + msg385248
2021-01-18 23:58:56vstinnersetnosy: + vstinner
2021-01-18 23:41:19William.Schwartzsetmessages: + msg385239
2021-01-18 23:22:48eryksunsetnosy: + eryksun
messages: + msg385238
2021-01-18 22:33:41petr.viktorinsetnosy: - petr.viktorin
2021-01-18 22:12:34William.Schwartzcreate