classification
Title: Multiprocessing spawn/forkserver fails to pass Queues
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Sean Murphy, davin, kumaar.nd, mouse07410, pitrou
Priority: normal Keywords:

Created on 2016-12-14 00:51 by Sean Murphy, last changed 2020-03-29 21:52 by mouse07410. This issue is now closed.

Messages (6)
msg283150 - (view) Author: Sean Murphy (Sean Murphy) Date: 2016-12-14 00:51
Python fails to pass a Queue when calling Process with multiprocessing.set_start_method set to "spawn" or "forkserver".

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main
    self = pickle.load(from_parent)
  File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory


Here is a minimized example:
```
#!/usr/bin/env python3

import multiprocessing

def check_child(q):
    print("Queue", q)


if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')
    # multiprocessing.set_start_method('fork')
    # multiprocessing.set_start_method('forkserver')

    q = multiprocessing.Queue(-1)
    print("q", q)

    proc = multiprocessing.Process(target=check_child, args=(q,))
    proc.start()
```

Also, this fails when the Queue is implicitly passed to the child.
```
class Blerg():
    def __init__(self):
        self.q = multiprocessing.Queue(-1)

    def print_queue(self):
        print("Queue", self.q)


if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')

    blerg = Blerg()

    blerg.print_queue()

    proc = multiprocessing.Process(target=blerg.print_queue)
    proc.start()
```

$ python3 --version
Python 3.5.2

Windows (which defaults to "spawn" style multiprocessing) does not seem to have this issue (at least in 2.7.12).
msg283305 - (view) Author: kumaar.nd (kumaar.nd) Date: 2016-12-15 13:58
hi,

few comments on this: [when tested with 'spawn']

1. the document(3x, 17.2.1.2) clearly mentions that 'spawn' doesnt inherit the file-handles (hence FileNotFoundError).
2. the document's examples has spawn() followed by join() where parent waits for the Child to finish and exits gracefully. The error "OSError: [Errno 9] Bad file descriptor" is seen for Child process.
3. Isnt exception-handling for null file descriptors sufficient (with corresponding Document updates)

am not a core developer yet and still trying to learn and contribute.

Thanks
msg298871 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-22 21:21
Ok, this is really an error in your script.  If you add `p.join()` to wait for the process to end, you'll see no error pops up.  What really happens is that the parent process ends before the child is fully started, and the resource underlying the queue is cleaned up at this point.
msg298872 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-22 21:25
More exactly, since the child process is not "daemonic", the parent process waits for it to end, but after collecting the queue's resources.
msg298873 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-22 21:26
So, while it might be possible to improve on this, it would risk fragilizing other use cases, and besides, this is not a problem that is likely to happen in real usage.  Therefore, I'd rather close this issue.
msg365280 - (view) Author: Mouse (mouse07410) Date: 2020-03-29 21:52
On MacOS Catalina 10.15.4, I still see this problem occasionally even with p.join() added. See https://bugs.python.org/msg365251 and subsequent messages.

Also, see https://bugs.python.org/issue40106.
History
Date User Action Args
2020-03-29 21:52:57mouse07410setnosy: + mouse07410
messages: + msg365280
2017-07-22 21:26:54pitrousetstatus: open -> closed
resolution: wont fix
messages: + msg298873

stage: resolved
2017-07-22 21:25:29pitrousetmessages: + msg298872
2017-07-22 21:21:47pitrousetnosy: + pitrou
messages: + msg298871
2016-12-15 13:58:20kumaar.ndsetnosy: + kumaar.nd
messages: + msg283305
2016-12-14 16:44:03davinsetnosy: + davin
type: crash -> behavior
2016-12-14 00:51:48Sean Murphycreate