> I'm running some long-running (possibly infinite) tasks in the thread pool,
> and I cancel them in an `atexit` callback

To be clear, by "cancel" you are not talking about Future.cancel().  Rather, your handler causes all running tasks to finish (by sending a special message on the socket corresponding to each running task).  Is that right?

Some other things I inferred from your atexit handler:

* it does not make sure the task associated with the socket finishes (no way of knowing?)
* so if a task hangs while trying to stop then the running thread in the ThreadPoolExecutor would block shutdown forever
* similarly, if a task is stuck handling a request then it will never receive the special message on the socket, either blocking the send() in your handler or causing ThreadPoolExecutor shutdown/atexit to wait forever
* it vaguely implies a 1-to-1 relationship between sockets and *running* tasks
* likewise that pending (queued) tasks do not have an associated socket (until started)
* so once your handler finishes, any tasks pending in the ThreadPoolExecutor queue will eventually get started but never get stopped by your handler; thus you're back to the deadlock situation

Does all that sound right?  Most of that is relevant to some possible solutions I have in mind.