Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

maggyero · 2022-02-11T14:57:10Z

BPO	46720
Nosy	@maggyero
PRs	bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #31279

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-11.14:57:09.587>
labels = ['type-feature', 'library', '3.9', '3.10', '3.11']
title = 'Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems'
updated_at = <Date 2022-02-11.22:25:00.201>
user = 'https://github.com/maggyero'

bugs.python.org fields:

activity = <Date 2022-02-11.22:25:00.201>
actor = 'maggyero'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-11.14:57:09.587>
creator = 'maggyero'
dependencies = []
files = []
hgrepos = []
issue_num = 46720
keywords = ['patch']
message_count = 1.0
messages = ['413073']
nosy_count = 1.0
nosy_names = ['maggyero']
pr_nums = ['31279']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46720'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

The text was updated successfully, but these errors were encountered:

maggyero · 2022-02-11T14:57:09Z

Any path-like object can be passed to multiprocessing.set_executable, i.e. objects with str, bytes, or os.PathLike type.

For instance these work (tested on MacOS with all start methods: ‘spawn’, ‘fork’, and ‘forkserver’):

multiprocessing.set_executable(sys.executable) (str);
multiprocessing.set_executable(sys.executable.encode()) (bytes);
multiprocessing.set_executable(pathlib.Path(sys.executable)) (os.PathLike).

This is because the ‘fork’ start method does not exec any program in the subprocess, the ‘spawn’ start method converts its path argument to bytes with os.fsencode before passing to _posixsubprocess.fork_exec, and the ‘forkserver’ start method spawns a server process (like with the ‘spawn’ start method) which then forks itself at each request (like the ‘fork’ start method):

        return _posixsubprocess.fork_exec(
            args, [os.fsencode(path)], True, passfds, None, None,
            -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
            False, False, None, None, None, -1, None)

Linux (and other Unix-like systems) uses the same code than MacOS for the three start methods so it should work for it too.

However I have not tested this on Windows which uses the function _winapi.CreateProcess for the ‘spawn’ start method (the only start method available on this OS) but I noticed that no conversion to str (not to bytes this time, since the function expects str) of the path argument with os.fsdecode (not os.fsencode this time) is performed before passing it to the function:

                hp, ht, pid, tid = _winapi.CreateProcess(
                    python_exe, cmd,
                    None, None, False, 0, env, None, None)

So on Windows only str path can be passed to multiprocessing.set_executable. This PR fixes this to be on a par with Unix-like systems which accept any path-like objects.

brettcannon · 2022-04-22T22:47:33Z

Closed via #31279 .

Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing.

…cutable` is `None` (#106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need.

…ys.executable` is `None` (pythonGH-106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need. (cherry picked from commit c60df36) Co-authored-by: Gregory P. Smith <greg@krypto.org>

…sys.executable` is `None` (GH-106464) (#106494) gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (GH-106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need. (cherry picked from commit c60df36) Co-authored-by: Gregory P. Smith <greg@krypto.org>

…sys.executable` is `None` (GH-106464) (#106495) gh-90876: Restore the ability to import multiprocessing when `sys.executable` is `None` (GH-106464) Prevent `multiprocessing.spawn` from failing to *import* in environments where `sys.executable` is `None`. This regressed in 3.11 with the addition of support for path-like objects in multiprocessing. Adds a test decorator to have tests only run when part of test_multiprocessing_spawn to `_test_multiprocessing.py` so we can start to avoid re-running the same not-global-state specific test in all 3 modes when there is no need. (cherry picked from commit c60df36) Co-authored-by: Gregory P. Smith <greg@krypto.org>

maggyero mannequin added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Feb 11, 2022

maggyero mannequin changed the title ~~Add support of path-like objects to multiprocessing.set_executable for Windows to match Unix-like systems~~ Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems Feb 11, 2022

ezio-melotti transferred this issue from another repository Apr 10, 2022

brettcannon closed this as completed Apr 22, 2022

bedevere-bot mentioned this issue Jul 5, 2023

gh-90876: Restore the ability to import multiprocessing when sys.executable is None #106464

Merged

This was referenced Jul 6, 2023

[3.12] gh-90876: Restore the ability to import multiprocessing when sys.executable is None (GH-106464) #106494

Merged

[3.11] gh-90876: Restore the ability to import multiprocessing when sys.executable is None (GH-106464) #106495

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

maggyero mannequin commented Feb 11, 2022 •

edited by bedevere-bot

maggyero mannequin commented Feb 11, 2022

brettcannon commented Apr 22, 2022

Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

Add support for path-like objects to multiprocessing.set_executable for Windows to be on a par with Unix-like systems #90876

Comments

maggyero mannequin commented Feb 11, 2022 • edited by bedevere-bot

Linked PRs

maggyero mannequin commented Feb 11, 2022

brettcannon commented Apr 22, 2022

maggyero mannequin commented Feb 11, 2022 •

edited by bedevere-bot