classification
Title: Multiprocessing.Queue created by sub-process fails when used in sub-sub-process ("bad file descriptor" in q.get())
Type: crash Stage:
Components: Library (Lib) Versions: Python 3.0, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jnoller Nosy List: OG7, bbutler, jnoller
Priority: normal Keywords:

Created on 2009-02-05 01:52 by bbutler, last changed 2009-06-30 17:14 by jnoller. This issue is now closed.

Files
File name Uploaded Description Edit
queue_test.py bbutler, 2009-02-05 01:52 Test script
Messages (6)
msg81183 - (view) Author: Beau Butler (bbutler) Date: 2009-02-05 01:52
(Note: This issue only happens on linux - on Windows things work fine.)

Please see the attached script.

The script creates a process (TestProcess()). That process runs and
creates a sub-process (TestSubProcess()). When TestSubProcess tries to
access a Queue created by TestProcess, the .get() call fails with the
following stack trace:

Process Process-1:1:
Traceback (most recent call last):
  File "/opt/python2.6/lib/python2.6/multiprocessing/process.py", line
231, in _bootstrap
    self.run()
  File "/opt/python2.6/lib/python2.6/multiprocessing/process.py", line
88, in run
    self._target(*self._args, **self._kwargs)
  File "queue_test.py", line 7, in TestSubProcess
    item = q.get()                        # Fails with IOError if queue
created in TestProcess
  File "/opt/python2.6/lib/python2.6/multiprocessing/queues.py", line
91, in get
    res = self._recv()
IOError: [Errno 9] Bad file descriptor

If the same queue is created by main (top level), TestSubProcess
accesses it with no errors. This behavior can be stimulated by
uncommenting the q = multiprocessing.Queue() line in TestProcess.


Tested using:
Python 2.6.1 (r261:67517, Dec  4 2008, 16:51:00) [MSC v.1500 32 bit
(Intel)] on win32
(Installed using binary installer, windows XPSP2)

*No problems*

Python 2.6.1 (r261:67515, Jan  6 2009, 15:23:19) [GCC 4.2.4 (Ubuntu
4.2.4-1ubuntu3)] on linux2
(Compiled from source due to ubuntu 8.04 (Hardy) not having python2.6 yet)

*Fails on q.get() in TestSubProcess*
msg81184 - (view) Author: Beau Butler (bbutler) Date: 2009-02-05 02:30
Also reproduced in Python3.0 on OSX 10.5
msg84947 - (view) Author: Beau Butler (bbutler) Date: 2009-03-31 23:23
Quick note: Have tracked this down to what appears to be buggy behaviour
on the part of os.pipe() when called from within a Process.

First invocation of os.pipe() in a Process returns (0,x) - stdin(?)
resulting in the 'bad file descriptor' error.

Interactive prompt test follows:

>>> import multiprocessing as MP, os
>>> def r():  print os.pipe(), os.pipe(), os.pipe()
... 
>>> r()
(9, 10) (11, 12) (13, 14)
>>> MP.Process(target=r).start()
(0, 15) (16, 17) (18, 19)
>>> MP.Process(target=r).start()
(0, 15) (16, 17) (18, 19)
msg89196 - (view) Author: OG7 (OG7) Date: 2009-06-10 11:20
Issue 5313 seems to be the culprit.
msg89902 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-06-30 03:04
Patch attached to issue 5313, please review.
msg89944 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-06-30 17:14
Committed in r73708 on trunk
History
Date User Action Args
2009-06-30 17:14:17jnollersetstatus: open -> closed
resolution: fixed
messages: + msg89944
2009-06-30 03:04:30jnollersetmessages: + msg89902
2009-06-10 11:20:25OG7setnosy: + OG7
messages: + msg89196
2009-03-31 23:23:07bbutlersetmessages: + msg84947
2009-03-29 14:38:08jnollersetpriority: normal
2009-02-05 03:30:03benjamin.petersonsetassignee: jnoller
nosy: + jnoller
2009-02-05 02:30:43bbutlersetmessages: + msg81184
versions: + Python 3.0
2009-02-05 01:56:40bbutlersettitle: Multiprocessing.Queue created by sub-process fails when used in sub-sub-process -> Multiprocessing.Queue created by sub-process fails when used in sub-sub-process ("bad file descriptor" in q.get())
2009-02-05 01:54:51bbutlersettitle: Multiprocessing.Queue created by subprocess fails when used in sub-sub-process -> Multiprocessing.Queue created by sub-process fails when used in sub-sub-process
2009-02-05 01:52:52bbutlercreate