This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Inconsistent ProcessPoolExecutor behaviour on macOS between 3.7 and 3.8/9
Type: behavior Stage: resolved
Components: macOS Versions: Python 3.8
process
Status: open Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: bquinlan, mustafaquraish, ned.deily, pitrou, ronaldoussoren
Priority: normal Keywords:

Created on 2020-11-07 00:19 by mustafaquraish, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
test.py mustafaquraish, 2020-11-07 00:19 Sample code with a bug
Messages (2)
msg380483 - (view) Author: Mustafa Quraish (mustafaquraish) * Date: 2020-11-07 00:19
The code attached produces weird behaviour on macOS (Catalina at least) on Python3.8 / Python3.9. Issue has been reproduced on at least one other mac belonging to a friend. Does not occur on Linux as far as I have tested. I have simplified the code as much as I can, so the example attached does not actually need to use parallelism.

Code description: Read text from a file, change directory using `os.chdir`, spawn 5 processes and pass the `print` function as the handler along with the text.

Expectation: File contents should be printed 5 times

Result: Code tries to load (again) the file in each thread, but fails since the directory was changed.


Precondition: Have `file.txt` in the same directory containing some text

Output when run on Python3.7, I get the following (expected output):

```
Reading file...
Text loaded was: 'file contents'
file contents
file contents
file contents
file contents
file contents
```

When running on 3.8 / 3.9 (on a mac) I get the following:

```
Reading file...
Text loaded was: 'file contents'
Error: /Users/mustafa/dev/ptest/test_dir/file.txt does not exist.
Error: /Users/mustafa/dev/ptest/test_dir/file.txt does not exist.
Error: /Users/mustafa/dev/ptest/test_dir/file.txt does not exist.
```

It seems to me that even opened `file.txt` is loaded in the main thread and text is read from it, this value isn't passed into the spawned processes and they are calling `load()` again (which is where the error message is produced).
msg380495 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-07 09:28
In 3.8 the spawn method for multiprocessing changed from "fork" to "spawn" (see https://docs.python.org/3/whatsnew/3.8.html#multiprocessing).

A side effect of this is that the module gets executed again in the child processes (the same as on Windows).

The reason for this change is that multiple higher level APIs, some of which used by the Python implementation, are are not safe to use with the "fork" spawning strategy (as in: causing crashing when using the fork strategy).
History
Date User Action Args
2022-04-11 14:59:37adminsetstatus: pending -> open
github: 86447
2020-11-07 09:28:38ronaldoussorensetstatus: open -> pending
resolution: wont fix
messages: + msg380495

stage: resolved
2020-11-07 00:19:51mustafaquraishcreate