> To be clear, by "cancel" you are not talking about Future.cancel().  Rather, your handler causes all running tasks to finish (by sending a special message on the socket corresponding to each running task).  Is that right?

Correct. My tasks here are calls to functions from the `select` module (one select call per executor task), and cancelling them means writing a byte to a pipe set up for this purpose. 

The select calls could be given a timeout so there is never an infinite task, but that's not ideal - a timeout that's too low has a performance cost as calls timeout and restart even when the system is "at rest", and a too-long timeout is still going to be perceived as a hanging application. 

> * it does not make sure the task associated with the socket finishes (no way of knowing?)
> * so if a task hangs while trying to stop then the running thread in the ThreadPoolExecutor would block shutdown forever
> * similarly, if a task is stuck handling a request then it will never receive the special message on the socket, either blocking the send() in your handler or causing ThreadPoolExecutor shutdown/atexit to wait forever

Correct. If the task were buggy it could still cause a deadlock. In my case the task is simple enough (a single selector call) that this is not a risk. 

> * it vaguely implies a 1-to-1 relationship between sockets and *running* tasks
> * likewise that pending (queued) tasks do not have an associated socket (until started)

Each task is associated with a selector object (managing a set of sockets), not a single socket. There is only ever one task at a time; a task is enqueued only after the previous one finishes. (This thread pool is not used for any other purpose)

> * so once your handler finishes, any tasks pending in the ThreadPoolExecutor queue will eventually get started but never get stopped by your handler; thus you're back to the deadlock situation

In my case this one-at-a-time rule means that the queue is always empty. But yes, in a more general solution you'd need some sort of interlock between cancelling existing tasks and starting new ones. 

> Alternately, perhaps ThreadPoolExecutor isn't the right fit here, as implied by the route you ended up going. 

Yes, this is my conclusion as well. I filed this issue because I was frustrated that Python 3.9 broke previously-working code, but I'm willing to chalk this up to Hyrum's law and I'm not sure that this is something that ThreadPoolExecutor should be modified to support.