classification
Title: [Windows] multiprocessing: DupHandle.detach() race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE)
Type: compile error Stage: needs patch
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, jesvi22j, leonardo2, pablogsal, pemryan, pitrou, s0undt3ch
Priority: normal Keywords:

Created on 2019-09-24 10:36 by vstinner, last changed 2021-04-10 01:07 by leonardo2.

Messages (8)
msg353064 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-24 10:36
On Windows, the multiprocessing DupHandle.detach() method has race condition on DuplicateHandle(DUPLICATE_CLOSE_SOURCE).

Error on duplicate():

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Access is denied

Example: bpo-34714


Error on detach():

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\spawn.py", line 117, in spawn_main
    exitcode = _main(fd)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\spawn.py", line 127, in _main
    self = reduction.pickle.load(from_parent)
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\connection.py", line 951, in rebuild_pipe_connection
    handle = dh.detach()
  File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\lib\multiprocessing\reduction.py", line 133, in detach
    self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

Example: bpo-34513
msg362460 - (view) Author: Pedro Algarvio (s0undt3ch) * Date: 2020-02-22 11:20
Any possible workaround for this issue?
I seem to be consistingly hitting this issue.
msg362466 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-02-22 13:44
> I seem to be consistingly hitting this issue.

It will help with test development if you can provide a minimal example that reliably reproduces the problem. 

In msg353064 I see DuplicateHandle calls failing with ERROR_ACCESS_DENIED (5). Assuming the process handles have the required PROCESS_DUP_HANDLE access, it's most likely the case that the underlying NtDuplicateObject system call is failing with STATUS_PROCESS_IS_TERMINATING (0xC000010A). For example:

    import ctypes
    ntdll = ctypes.WinDLL('ntdll')
    from subprocess import Popen, PIPE
    from _winapi import GetCurrentProcess, TerminateProcess
    from _winapi import DuplicateHandle, DUPLICATE_SAME_ACCESS

    p = Popen('cmd.exe', stdin=PIPE, stdout=PIPE, stderr=PIPE)
    TerminateProcess(p._handle, 0)

Try to duplicate the process handle into the terminated process:

    >>> source = GetCurrentProcess()
    >>> target = handle = p._handle
    >>> try:
    ...     DuplicateHandle(source, handle, target,
    ...         0, False, DUPLICATE_SAME_ACCESS)
    ... except:
    ...     status = ntdll.RtlGetLastNtStatus() 
    ...     print(f'NTSTATUS: {status & (2**32-1):#010x}')
    ...     raise
    ...
    NTSTATUS: 0xc000010a
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    PermissionError: [WinError 5] Access is denied
msg362471 - (view) Author: Pedro Algarvio (s0undt3ch) * Date: 2020-02-22 18:40
What I'm able to copy from the console(though I doubt I'm getting all of
the traceback):

Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\spawn.py",
line 106, in spawn_main

    exitcode = _main(fd)

  File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\spawn.py",
line 116, in _main

    self = pickle.load(from_parent)

  File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\connection.py",
line 942, in rebuild_pipe_connection

    handle = dh.detach()

  File
"c:\hostedtoolcache\windows\python\3.5.4\x64\lib\multiprocessing\reduction.py",
line 128, in detach

    self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)

PermissionError: [WinError 5] Access is denied

Pedro Algarvio @ Phone

A sábado, 22/02/2020, 13:44, Eryk Sun <report@bugs.python.org> escreveu:

>
> Eryk Sun <eryksun@gmail.com> added the comment:
>
> > I seem to be consistingly hitting this issue.
>
> It will help with test development if you can provide a minimal example
> that reliably reproduces the problem.
>
> In msg353064 I see DuplicateHandle calls failing with ERROR_ACCESS_DENIED
> (5). Assuming the process handles have the required PROCESS_DUP_HANDLE
> access, it's most likely the case that the underlying NtDuplicateObject
> system call is failing with STATUS_PROCESS_IS_TERMINATING (0xC000010A). For
> example:
>
>     import ctypes
>     ntdll = ctypes.WinDLL('ntdll')
>     from subprocess import Popen, PIPE
>     from _winapi import GetCurrentProcess, TerminateProcess
>     from _winapi import DuplicateHandle, DUPLICATE_SAME_ACCESS
>
>     p = Popen('cmd.exe', stdin=PIPE, stdout=PIPE, stderr=PIPE)
>     TerminateProcess(p._handle, 0)
>
> Try to duplicate the process handle into the terminated process:
>
>     >>> source = GetCurrentProcess()
>     >>> target = handle = p._handle
>     >>> try:
>     ...     DuplicateHandle(source, handle, target,
>     ...         0, False, DUPLICATE_SAME_ACCESS)
>     ... except:
>     ...     status = ntdll.RtlGetLastNtStatus()
>     ...     print(f'NTSTATUS: {status & (2**32-1):#010x}')
>     ...     raise
>     ...
>     NTSTATUS: 0xc000010a
>     Traceback (most recent call last):
>       File "<stdin>", line 2, in <module>
>     PermissionError: [WinError 5] Access is denied
>
> ----------
> nosy: +eryksun
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue38263>
> _______________________________________
>
msg377296 - (view) Author: Yan Ren (pemryan) Date: 2020-09-22 03:24
append Nosy List.
msg389832 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-30 15:36
spawn_main() could handle a PermissionError by checking the exit code. If the parent has terminated already, then simply exit quietly. _winapi.PROCESS_QUERY_LIMITED_INFORMATION would need to be defined. For example:

    def spawn_main(pipe_handle, parent_pid=None, tracker_fd=None):
        """Run code specified by data received over a pipe."""
        assert is_forking(sys.argv), "Not forking"

        if sys.platform == 'win32':
            import msvcrt
            import _winapi

            if parent_pid is not None:
                source_process = _winapi.OpenProcess(
                    _winapi.SYNCHRONIZE |
                    _winapi.PROCESS_DUP_HANDLE |
                    _winapi.PROCESS_QUERY_LIMITED_INFORMATION,
                    False, parent_pid)
            else:
                source_process = None
            try:
                new_handle = reduction.duplicate(
                    pipe_handle, source_process=source_process)
                fd = msvcrt.open_osfhandle(new_handle, os.O_RDONLY)
                exitcode = _main(fd, source_process)
            except PermissionError:
                if (source_process is None or
                      _winapi.GetExitCodeProcess(source_process) ==
                      _winapi.STILL_ACTIVE):
                    raise
                exitcode = 1
        else:
            from . import resource_tracker
            resource_tracker._resource_tracker._fd = tracker_fd
            exitcode = _main(pipe_handle, os.dup(pipe_handle))

        sys.exit(exitcode)
msg390229 - (view) Author: Jesvi Jonathan (jesvi22j) Date: 2021-04-05 09:10
File "c:/Users/jesvi/Documents/GitHub/Jesvi-Bot-Telegram/scripts/main.py", line 144, in thread_test
    p.start()
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_thread.lock' object
Traceback (most recent call last):
  File "<string>", line 1, in <module>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 107, in spawn_main
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 79, in duplicate
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\jesvi\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid
msg390683 - (view) Author: Leonardo Rick (leonardo2) Date: 2021-04-10 01:07
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Program Files\Python39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "C:\Program Files\Python39\lib\multiprocessing\connection.py", line 967, in rebuild_pipe_connection
    handle = dh.detach()
  File "C:\Program Files\Python39\lib\multiprocessing\reduction.py", line 131, in detach
    return _winapi.DuplicateHandle(
PermissionError: [WinError 5] Acesso negado
History
Date User Action Args
2021-04-10 01:07:14leonardo2setnosy: + leonardo2

messages: + msg390683
versions: + Python 3.9, - Python 3.8
2021-04-05 09:10:36jesvi22jsetversions: - Python 3.9, Python 3.10
nosy: + jesvi22j

messages: + msg390229

type: behavior -> compile error
2021-03-30 16:04:59vstinnersetnosy: - vstinner
2021-03-30 15:36:07eryksunsettype: behavior
stage: needs patch
messages: + msg389832
versions: + Python 3.10, - Python 3.7
2020-09-22 03:24:51pemryansetnosy: + pemryan
messages: + msg377296
2020-02-22 18:40:50s0undt3chsetmessages: + msg362471
2020-02-22 13:44:17eryksunsetnosy: + eryksun
messages: + msg362466
2020-02-22 11:20:43s0undt3chsetnosy: + s0undt3ch
messages: + msg362460
2019-09-24 10:36:39vstinnercreate