This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: pthread_sigmask needs SIG_BLOCK behaviour explaination
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.11, Python 3.10
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, rpurdie
Priority: normal Keywords:

Created on 2022-03-28 09:48 by rpurdie, last changed 2022-04-11 14:59 by admin.

Messages (4)
msg416154 - (view) Author: Richard Purdie (rpurdie) Date: 2022-03-28 09:48
I've been struggling to get signal.pthread_sigmask to do what I expected it to do from the documentation. Having looked at the core python code handling signals I now (think?!) I understand what is happening. It might be possible for python to improve the behaviour, or it might just be something to document, I'm not sure but I though I'd mention it.

I'd added pthread_sigmask(SIG_BLOCK, (SIGTERM,)) and pthread_sigmask(SIG_UNBLOCK, (SIGTERM,)) calls around a critical section I wanted to protect from the SIGTERM signal. I was still seeing SIGTERM inside that section. Using SIGMASK to restore the mask instead of SIG_UNBLOCK behaves the same.

What I hadn't realised is that firstly python defers signals to a convenient point and secondly that signals are processed in the main thread regardless of the thread they arrived in. This means that I can see SIGTERM arrive in my critical section as one of my other threads created in the background by the core python libs helpfully handles it.  This makes SIG_BLOCK rather ineffective in any threaded code.

To work around it, I can add my own handlers and have them track whether a signal arrived, then handle any signals after my critical section by re-raising them. It is possible python itself could defer processing signals masked with SIG_BLOCK until they're unblocked. Alternatively, a note in the documentation warning of the pitfalls here might be helpful to save someone else from wondering what is going on!
msg416757 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-04-05 07:24
The irony... Documenting the caveat at least seems useful. Your workaround sounds reasonable.

I don't love the idea of implementing our own mask blocked/unblocked state check, though it probably wouldn't be very complicated. Might be interesting.

Another trick that'd "work" so long as people don't have multiple SIG_BLOCK calls (not something to depend on) is to force a check of our interpreter signal flags right before signal.pthread_sigmask returns to the Python caller.
msg416794 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2022-04-05 18:18
The "trick" wouldn't be too useful though as this API can't block and the signal flag needs to be processed on the main thread. So I guess documentation it is.

The way I think of this is that the signal.pthread_sigmask API is pretty low level. After that API is called, no more signals will _reach the process_, but the interpreter may process some that happened beforehand. So installing a handler is indeed appropriate.

Also, it is entirely possible for C extension modules or code embedding Python to call pthread_sigmask in its own threads outside of the Python runtimes knowledge. So we us tracking what signals are blocked on our own may not be accurate.  We could instead change the eval loop signal processing code to call `pthread_sigmask(SIG_UNBLOCK, NULL /* set */, &oldset);` to retrieve the processes current mask to decide if the flagged signals should be processed by Python or not.

BUT... I can imagine complex race cases where that'd surprise people who are chaining multiple signal handlers such as one from outside of Python that saved the Python handler and calls it afterwards.  Their C/process-level handler would be called, would chain to Python's "record that signal X happened and set the bit" handler, but Python wouldn't then be guaranteed to call the Python handler code if the sigmask changed before the eval loop did its pending signal check.

So I'm still inclined to keep this simple and stick with just documenting best practices for now.  An implementation of masking from the python eval handler may need to be something conditionally controllable for differing application situations if added.
msg416808 - (view) Author: Richard Purdie (rpurdie) Date: 2022-04-05 19:28
I think the python code implementing pthread_sigmask already does trigger interrupts if any have been queued before the function returns from blocking or unblocking.

The key subtlety which I initially missed is that if you have another thread in your python script, any interrupt it receives can be raised in the main thread whilst you're in the SIGBLOCK section. This obviously isn't what you expect at all as those interrupts are supposed to be blocked! It isn't really practical to try and SIGBLOCK on all your individual threads.

What I'd wondered is what you mention, specifically checking if a signal is masked in the python signal raising code with something like the "pthread_sigmask(SIG_UNBLOCK, NULL /* set */, &oldset)" before it raises it and if there is blocked, just leave it queued. The current code would trigger the interrupts when it was unmasked. This would effectively only apply on the main thread where all the signals/interrupts are raised. 

This would certainly give the behaviour that would be expected from the calls and save everyone implementing the workarounds as I have. Due to the threads issue, I'm not sure SIGBLOCK is actually useful in the real world with the current implementation unfortunately.

Equally, if that isn't an acceptable fix, documenting it would definitely be good too.
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91295
2022-04-05 19:28:09rpurdiesetmessages: + msg416808
2022-04-05 18:18:30gregory.p.smithsetmessages: + msg416794
2022-04-05 07:24:56gregory.p.smithsetversions: + Python 3.11
nosy: + gregory.p.smith

messages: + msg416757

type: behavior
stage: needs patch
2022-03-28 09:48:02rpurdiecreate