This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Incorrect Argument Order for Calls to _winapi.DuplicateHandle() in multiprocessing.reduction.DupHandle
Type: behavior Stage: needs patch
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, m3rc1fulcameron, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-09-16 17:51 by m3rc1fulcameron, last changed 2022-04-11 14:59 by admin.

Messages (3)
msg352560 - (view) Author: Cameron Kennedy (m3rc1fulcameron) Date: 2019-09-16 17:51
The DuplicateHandle function is utilized by the DupHandle object to duplicate handles for the purpose of sending and receiving between processes on Windows systems. At least on Python 3.7.3, this function is invoked with an incorrect argument order. In multiprocessing.reduction, send_handle passes _winapi.DUPLICATE_SAME_ACCESS as the access argument to the DupHandle constructor, which in-turn passes it to the access argument for _winapi.DuplicateHandle(). Instead, per https://docs.microsoft.com/en-us/windows/win32/api/handleapi/nf-handleapi-duplicatehandle this constant should be passed into the options argument. This bug results in any handles communicated via this method to have meaningless permissions set, which makes them unusable. 

I've monkeypatched the issue with the following code:

try:
    import _winapi
    log = logging.getLogger('')
    log.warning('Patching multiprocessing.reduction to deal with the _winapi.DuplicateHandle() PROCESS_DUP_HANDLE argument order bug.')
    class _PatchedDupHandle(object):
        '''Picklable wrapper for a handle.'''
        def __init__(self, handle, access, pid=None, options=0):
            if pid is None:
                # We just duplicate the handle in the current process and
                # let the receiving process steal the handle.
                pid = os.getpid()
            proc = _winapi.OpenProcess(_winapi.PROCESS_DUP_HANDLE, False, pid)
            try:
                self._handle = _winapi.DuplicateHandle(
                    _winapi.GetCurrentProcess(),
                    handle, proc, access, False, options)
            finally:
                _winapi.CloseHandle(proc)
            self._options = options
            self._access = access
            self._pid = pid

        def detach(self):
            '''Get the handle.  This should only be called once.'''
            # retrieve handle from process which currently owns it
            if self._pid == os.getpid():
                # The handle has already been duplicated for this process.
                return self._handle
            # We must steal the handle from the process whose pid is self._pid.
            proc = _winapi.OpenProcess(_winapi.PROCESS_DUP_HANDLE, False,
                                       self._pid)
            try:
                return _winapi.DuplicateHandle(
                    proc, self._handle, _winapi.GetCurrentProcess(),
                    self._access, False, self._options|_winapi.DUPLICATE_CLOSE_SOURCE)
            finally:
                _winapi.CloseHandle(proc)
    DupHandle = _PatchedDupHandle
    def _patched_send_handle(conn, handle, destination_pid):
        '''Send a handle over a local connection.'''
        dh = DupHandle(handle, 0, destination_pid, _winapi.DUPLICATE_SAME_ACCESS)
        conn.send(dh)
    send_handle=_patched_send_handle
except ImportError:
    pass

The above seems to fix the problem on my machine by adding an additional options property to the DupHandle object and an options argument to send_handle function.
msg352610 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-09-17 06:35
As far as I can tell, reduction.send_handle isn't used internally in the Windows implementation, and it's also not a documented API function. However, it is tested on Windows in test_fd_transfer in Lib/test/_test_multiprocessing.py. As it turns out, the bug that Cameron's proposed solution fixes slips under the radar. By coincidence, DUPLICATE_SAME_ACCESS (2) has the same value as the file access right FILE_WRITE_DATA (2), and test_fd_transfer only checks whether the child can write to a file handle.

I propose adding test_fd_transfer_windows to _TestConnection in Lib/test/_test_multiprocessing.py, which will test whether the parent and child are granted the same access to a kernel file object after the handle is sent to the child.

    @classmethod
    def _check_handle_access(cls, conn):
        handle = reduction.recv_handle(conn)
        conn.send(get_handle_info(handle).GrantedAccess)

    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
    @unittest.skipIf(sys.platform != "win32", "Windows-only test")
    def test_fd_transfer_windows(self):
        if self.TYPE != 'processes':
            self.skipTest("only makes sense with processes")
        conn, child_conn = self.Pipe(duplex=True)
        p = self.Process(target=self._check_handle_access, args=(child_conn,))
        p.daemon = True
        p.start()
        try:
            with open(test.support.TESTFN, "wb") as f:
                self.addCleanup(test.support.unlink, test.support.TESTFN)
                handle = msvcrt.get_osfhandle(f.fileno())
                parent_access = get_handle_info(handle).GrantedAccess
                reduction.send_handle(conn, handle, p.pid)
                child_access = conn.recv()
                self.assertEqual(parent_access, child_access)
        finally:
            p.join()

get_handle_info() and the required ctypes support definitions [1] would be defined at module scope as follows:

    if WIN32:
        from ctypes import (WinDLL, WinError, Structure, POINTER, 
                            byref, sizeof, c_void_p, c_ulong)
        ntdll = WinDLL('ntdll')
        
        ntdll.NtQueryObject.argtypes = (
            c_void_p, # Handle
            c_ulong,  # ObjectInformationClass
            c_void_p, # ObjectInformation
            c_ulong,  # ObjectInformationLength
            POINTER(c_ulong)) # ReturnLength

        ObjectBasicInformation = 0

        class PUBLIC_OBJECT_BASIC_INFORMATION(Structure):
            (r"https://docs.microsoft.com/en-us/windows/win32/api"
             r"/winternl/nf-winternl-ntqueryobject")
            _fields_ = (('Attributes', c_ulong),
                        ('GrantedAccess', c_ulong),
                        ('HandleCount', c_ulong),
                        ('PointerCount', c_ulong),
                        ('Reserved', c_ulong * 10))


        def get_handle_info(handle):
            info = PUBLIC_OBJECT_BASIC_INFORMATION()
            status = ntdll.NtQueryObject(handle, ObjectBasicInformation,
                        byref(info), sizeof(info), None)
            if status < 0:
                error = ntdll.RtlNtStatusToDosError(status)
                raise WinError(error)
            return info

[1] https://docs.microsoft.com/en-us/windows/win32/api/winternl/nf-winternl-ntqueryobject
msg352615 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-09-17 07:10
Let's make test_fd_transfer_windows a bit less hangy by polling for up to 60 seconds instead of simply trying to recv() and by terminating before trying to join().

    @unittest.skipUnless(HAS_REDUCTION, "test needs multiprocessing.reduction")
    @unittest.skipIf(sys.platform != "win32", "Windows-only test")
    def test_fd_transfer_windows(self):
        if self.TYPE != 'processes':
            self.skipTest("only makes sense with processes")
        conn, child_conn = self.Pipe(duplex=True)
        p = self.Process(target=self._check_handle_access, args=(child_conn,))
        p.daemon = True
        p.start()
        try:
            with open(test.support.TESTFN, "wb") as f:
                self.addCleanup(test.support.unlink, test.support.TESTFN)
                handle = msvcrt.get_osfhandle(f.fileno())
                parent_access = get_handle_info(handle).GrantedAccess
                reduction.send_handle(conn, handle, p.pid)
                if not conn.poll(timeout=60):
                    raise AssertionError("could not receive from child process")
                child_access = conn.recv()
                self.assertEqual(parent_access, child_access)
        finally:
            p.terminate()
            p.join()
History
Date User Action Args
2022-04-11 14:59:20adminsetgithub: 82369
2021-03-22 00:46:22eryksunsetstage: needs patch
type: behavior
components: + Library (Lib)
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 3.7
2019-09-17 07:10:33eryksunsetmessages: + msg352615
2019-09-17 06:35:55eryksunsetmessages: + msg352610
2019-09-16 18:14:55ammar2setnosy: + eryksun
2019-09-16 17:51:37m3rc1fulcameroncreate