classification
Title: test_multiprocessing_spawn.WithProcessesTestPool.test_traceback() leaks 4 handles on Windows
Type: Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.8
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: davin, paul.moore, pitrou, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2018-06-26 11:55 by vstinner, last changed 2018-09-19 23:55 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7965 closed vstinner, 2018-06-27 12:22
PR 7966 closed vstinner, 2018-06-27 12:36
Messages (4)
msg320484 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-26 11:55
Using my PR 7827, the following command shows a leak of 4 Windows handles per run:

python -m test -R 3:3 test_multiprocessing_spawn -v -m test.test_multiprocessing_spawn.WithProcessesTestPool.test_traceback

See also bpo-33929: test_multiprocessing_spawn: WithProcessesTestProcess.test_many_processes() leaks 5 handles on Windows.
msg320525 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-26 23:53
* Pool._repopulate_pool() creates processes with args=(self._inqueue, ...)
* Pool._inqueue is a multiprocessing.queues.SimpleQueue
* Process.__init__() of multiprocessing.popen_spawn_win32 calls reduction.dump(process_obj, to_child) which indirectly contains the SimpleQueue
* A SimpleQueue object contains: self._reader, self._writer = connection.Pipe(duplex=False)
* dump() indirectly calls reduce_connection() (ex: for SimpleQueue._reader) of multiprocessing.connection
* reduce_connection() duplicates the pipe handle

It's unclear to me who is supposed to close the duplicated pipe handles? reduce_connection() creates a "ds = resource_sharer.DupSocket(s)" object, but this object doesn't seem to call CloseHandle()?
msg320528 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-27 00:21
> * dump() indirectly calls reduce_connection() (ex: for SimpleQueue._reader) of multiprocessing.connection
> * reduce_connection() duplicates the pipe handle

Sorry, it's reduce_pipe_connection(), not reduce_connection().

> It's unclear to me who is supposed to close the duplicated pipe handles? reduce_connection() creates a "ds = resource_sharer.DupSocket(s)" object, but this object doesn't seem to call CloseHandle()?

reduce_pipe_connection() creates a DupHandle object which duplicates the handle, and it's detach() method closes the handle.

The race condition is that sometimes the pool terminates a worker (worker() function of multiprocessing.pool) before the worker has time to close the duplicated handle.

The handle is closed by:
* rebuild_pipe_connection()
* dh.detach(), extract:

                return _winapi.DuplicateHandle(
                    proc, self._handle, _winapi.GetCurrentProcess(),
                    self._access, False, _winapi.DUPLICATE_CLOSE_SOURCE)
msg325833 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-19 23:55
I give up on that one :-( It doesn't seem easy to fix this corner case without changing the behaviour.
History
Date User Action Args
2018-09-19 23:55:24vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg325833

stage: patch review -> resolved
2018-06-27 12:36:43vstinnersetpull_requests: + pull_request7573
2018-06-27 12:22:39vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request7572
2018-06-27 00:21:58vstinnersetmessages: + msg320528
2018-06-26 23:53:31vstinnersetmessages: + msg320525
2018-06-26 11:55:02vstinnercreate