This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Processes in Python 3.9 exiting with code 1 when It's created inside a ThreadPoolExecutor
Type: behavior Stage:
Components: Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Genarito, davin, graingert, jack__d, pitrou, sicard_elda
Priority: normal Keywords:

Created on 2021-04-26 20:59 by Genarito, last changed 2022-04-11 14:59 by admin.

Messages (8)
msg391985 - (view) Author: Genaro Camele (Genarito) Date: 2021-04-26 20:59
I've a piece of code that submits a task to a [ThreadPoolExecutor][1] which starts a [Process][2]. I've realised that in Python 3.8 that Process finished with exit code `0`. But I've updated Python to the 3.9 version and this started to finishing with exit code `1`! Even when the Process executes an empty task.

Here's a minimal example:

```python
from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor


def some_task():
    pass


def execute_error():
    p = Process(target=some_task)
    p.start()
    p.join()
    print(p.exitcode)  # This is always 1 on a ThreadPoolExecutor!!!


executor = ThreadPoolExecutor(max_workers=4)
executor.submit(execute_error)
# execute_error()  # IMPORTANT: this works correctly (exit 0)
```

My versions:

```
Ubuntu 21.04
Python 3.9.4
```

**Note** that if `__execute_error` is called outside the ThreadPoolExecutor it works correctly.
Running on Python 3.8.6 exitcode = 0 too.


  [1]: https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor
  [2]: https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.Process
msg396522 - (view) Author: Alexandre Sicard (sicard_elda) Date: 2021-06-25 10:16
Thank you very much for this report, Genaro. I encountered the same bug with a Process running in the context of a Django view. Downgrading to Python 3.8 also fixed the issue for me.

Versions:
python:3.9-alpine Docker image, running Python 3.9.5 and Alpine 3.13. Can also reproduce on the standard (Debian Buster based) python:3.9 image, and on Arch Linux (bare metal), all also running Python 3.9.5.
msg398157 - (view) Author: Jack DeVries (jack__d) * Date: 2021-07-24 14:11
I am working on a fix for this bug. I'm a beginner cpython contributor, so if anyone recognizes this as a fool's errand, please let me know!
msg398159 - (view) Author: Jack DeVries (jack__d) * Date: 2021-07-24 14:30
Ah never mind. @Genarito, the ThreadPoolExecutor is supposed to be used as a context manager. In your current code, the script ends and Python starts tearing itself down while `execute_error` is still running in a subprocess.

If you simply use the ThreadPoolExecutor to a context manager, the error goes away::

```python
from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor


def some_task():
    pass


def execute_error():
    p = Process(target=some_task)
    p.start()
    p.join()
    print(p.exitcode)  # This is always 1 on a ThreadPoolExecutor!!!


# THIS IS THE IMPORTANT CHANGE
with ThreadPoolExecutor(max_workers=4) as executor:
    executor.submit(execute_error)
```
msg398175 - (view) Author: Genaro Camele (Genarito) Date: 2021-07-24 21:55
Hi @jack__d, thanks for your answer and time. Unfortunately, It's still a regression, as in Python < 3.9 my example works as expected
msg398202 - (view) Author: Jack DeVries (jack__d) * Date: 2021-07-26 00:24
I've identified the first bad commit with git-bisect:

commit b61b818d916942aad1f8f3e33181801c4a1ed14b
Author: Kyle Stanley <aeros167@gmail.com>
Date:   Fri Mar 27 15:31:22 2020 -0400

    bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)

    Remove daemon threads from :mod:`concurrent.futures` by adding
    an internal `threading._register_atexit()`, which calls registered functions
    prior to joining all non-daemon threads. This allows for compatibility
    with subinterpreters, which don't support daemon threads.
msg398203 - (view) Author: Jack DeVries (jack__d) * Date: 2021-07-26 00:25
The first bad commit was a fix for bpo-39812.
msg416966 - (view) Author: Thomas Grainger (graingert) * Date: 2022-04-08 09:35
the problem is multiprocessing/process is calling threading._shutdown which tries to join its own thread, because concurrent.futures.thread._threads_queues contains the main thread in the subprocess


  File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/multiprocessing/process.py", line 333, in _bootstrap
    threading._shutdown()
  File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1530, in _shutdown
    atexit_call()
  File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit
    t.join()
  File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1086, in join
    raise RuntimeError("cannot join current thread")
History
Date User Action Args
2022-04-11 14:59:44adminsetgithub: 88110
2022-04-08 09:35:24graingertsetnosy: + graingert
messages: + msg416966
2021-07-26 00:25:22jack__dsetmessages: + msg398203
2021-07-26 00:24:24jack__dsetmessages: + msg398202
2021-07-24 21:55:55Genaritosetmessages: + msg398175
2021-07-24 14:30:25jack__dsetmessages: + msg398159
2021-07-24 14:11:03jack__dsetnosy: + jack__d
messages: + msg398157
2021-06-25 10:16:24sicard_eldasetnosy: + sicard_elda
messages: + msg396522
2021-04-30 22:15:35terry.reedysetnosy: + pitrou, davin
2021-04-26 20:59:45Genaritocreate