classification
Title: os.write and os.close on pipe from separate threads hangs on Mac OS X
Type: behavior Stage:
Components: IO, Macintosh Versions: Python 2.6, Python 2.5, 3rd party
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: hughsw, loewis, ronaldoussoren
Priority: normal Keywords:

Created on 2009-11-26 23:22 by hughsw, last changed 2009-12-01 15:47 by ronaldoussoren. This issue is now closed.

Files
File name Uploaded Description Edit
os_pipe_write_close_bug.py hughsw, 2009-11-26 23:22 script that documents and demostrates the hang
os_pipe_write_close_bug.c hughsw, 2009-12-01 15:31 C file that compiles to an executable that hangs
Messages (12)
msg95755 - (view) Author: Hugh Secker-Walker (hughsw) Date: 2009-11-26 23:22
The attached script demonstrates a problem on Mac OS X 10.5 whereby
Python can hang while simultaneously trying os.write with a large buffer
and os.close, from separate threads on the write and read fds
respectively of a pipe.
msg95759 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-11-27 07:41
Why do you think this is a bug in Python, and not an operating system
bug? I.e. if you would redo the script in C, would it still hang?
msg95763 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-11-27 11:27
I agree with martin, this is probably an OS bug. 

BTW. I can confirm that the issue occurs on OSX 10.6 as well (using the 
binary 2.6.4 installer on the python.org website).

I might get around to writing a C equivalent of the script in the future, 
but wouldn't mind if someone else did that work.  Such a script would 
pinpoint if the problem is in the OS or Python, and if it is in the OS it 
would allow filing a clear bug report at Apple.
msg95768 - (view) Author: Hugh Secker-Walker (hughsw) Date: 2009-11-27 13:58
Well, from *my* perspective it's a Python bug because I code to Python's
API and Python misbehaves! ;-)  

If I start on it soon enough I'll look into duplicating the problem in C.
msg95772 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-11-27 15:00
Hugh, for us as Python maintainers there is a distinction between issues 
with the Python implementation itself and those outside of Python. The 
former we can solve, the latter not. 

At best we can work around issues in the environment (I've recently 
committed a workaround for what basically amounts to a platform issue on 
OSX that caused crashes when importing an extension on a thread), but 
that's not always possible.
msg95843 - (view) Author: Hugh Secker-Walker (hughsw) Date: 2009-12-01 00:23
The attached C file exhibits the same symptoms, on Mac OS X 10.5 it
eventually hangs in an uninteruptible disk wait.  On Linux it doesn't
hang.  Further support for the idea that this is a Mac OS X problem.

How does one provide this reproducible case to Apple?
msg95844 - (view) Author: Hugh Secker-Walker (hughsw) Date: 2009-12-01 00:27
Fixed some typos in the C file
msg95848 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-12-01 07:16
Hugh: Do you have an ADC account (either free or payed)? If you do you can 
file bugs at http://bugreport.apple.com/. If you don't I can file the bug 
for you.
msg95855 - (view) Author: Hugh Secker-Walker (hughsw) Date: 2009-12-01 13:39
I have an ADC account that I only ever use to get XCode.  I'll file a
bug-report with Apple.
msg95861 - (view) Author: Hugh Secker-Walker (hughsw) Date: 2009-12-01 15:31
Updated the C program to deal with command-line args.  This is the
version submitted to Apple Bug Reporter, issue 7433004.
msg95864 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2009-12-01 15:43
Thanks, closing it here as third-party. Feel free to follow up here with
news on Apple's resolution of the issue.
msg95865 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2009-12-01 15:47
Hugh: never mind ....


There is a workaround for this issue: use socketpair(2) instead of 
pipe(2). I haven't thought enough about the consequences yet to have an 
firm opinion on implementing os.pipe using socketpair(2) on OSX. My gut 
feeling is that we shouldn't because pipes and sockets can have slightly 
different behavior, especially w.r.t. buffering.

As a workaround for this issue you can at least use socket.socketpair 
instead of os.pipe.
History
Date User Action Args
2009-12-01 15:47:47ronaldoussorensetmessages: + msg95865
2009-12-01 15:44:00loewissetstatus: open -> closed
resolution: wont fix
messages: + msg95864

versions: + 3rd party
2009-12-01 15:31:13hughswsetfiles: + os_pipe_write_close_bug.c

messages: + msg95861
2009-12-01 15:28:59hughswsetfiles: - os_pipe_write_close_bug.c
2009-12-01 13:39:57hughswsetmessages: + msg95855
2009-12-01 07:16:16ronaldoussorensetmessages: + msg95848
2009-12-01 00:27:29hughswsetfiles: + os_pipe_write_close_bug.c

messages: + msg95844
2009-12-01 00:26:43hughswsetfiles: - os_pipe_write_close_bug.c
2009-12-01 00:23:43hughswsetfiles: + os_pipe_write_close_bug.c

messages: + msg95843
2009-11-27 15:00:47ronaldoussorensetmessages: + msg95772
2009-11-27 13:58:40hughswsetmessages: + msg95768
2009-11-27 11:27:19ronaldoussorensetmessages: + msg95763
2009-11-27 07:41:25loewissetnosy: + loewis
messages: + msg95759
2009-11-27 03:20:28hughswsetassignee: ronaldoussoren

components: + Macintosh
nosy: + ronaldoussoren
2009-11-27 03:19:40hughswsetversions: + Python 2.5
2009-11-26 23:22:24hughswcreate