classification
Title: pty.read raises IOError when slave pty device is closed
Type: behavior Stage: patch review
Components: IO, Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, amaury.forgeotdarc, exarkun, gvanrossum, marciof, marduk, ocean-city, pitrou, zmedico
Priority: normal Keywords: patch

Created on 2009-02-27 08:03 by zmedico, last changed 2013-01-12 12:49 by marciof.

Files
File name Uploaded Description Edit
fromfile_pty_ioerror.py zmedico, 2009-02-27 08:03 test case demonstrating array.fromfile() failure with pty
simple_test.py ocean-city, 2009-02-27 09:38
simple_test_2.py ocean-city, 2009-02-27 10:04
io-openpty.patch pitrou, 2010-04-11 00:55 review
Messages (16)
msg82824 - (view) Author: Zac Medico (zmedico) Date: 2009-02-27 08:03
With python-3.0, array.fromfile() raises an IOError when reading from a
master pty device after the slave device has been closed. This causes
remaining data that had been written to the slave device to be lost. I
have observed this problem with python-3.0.1 on linux (I get the same
result with or without the patch from issue 5334). The traceback
produced by the attached test case looks like this:

    Traceback (most recent call last):
    File "./fromfile_pty_ioerror.py", line 26, in <module>
        buf.fromfile(master_file, bufsize)
    File "/usr/lib/python3.0/io.py", line 918, in read
        return self._read_unlocked(n)
    File "/usr/lib/python3.0/io.py", line 952, in _read_unlocked
        chunk = self.raw.read(wanted)
    IOError: [Errno 5] Input/output error

With python-2.x, the remaining data is appended to the array and an
EOFError is raised like one would expect.
msg82826 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-02-27 09:38
I think this is not array modules' bug. Attached test program outputs
different results on trunk/py3k.

debian:~/python-dev/trunk# ./python /mnt/windows/simple_test.py
os.pipe: success
pty.openpty: success

debian:~/python-dev/py3k# ./python /mnt/windows/simpled_test.py
b'os.pipe: success'
Traceback (most recent call last):
  File "/mnt/windows/simpled_test.py", line 17, in <module>
    gotdata = master_file.read(len(data) + 1)
  File "/root/python-dev/py3k/Lib/io.py", line 918, in read
    return self._read_unlocked(n)
  File "/root/python-dev/py3k/Lib/io.py", line 952, in _read_unlocked
    chunk = self.raw.read(wanted)
IOError: [Errno 5] Input/output error

And if you use io.open instead of os.fdopen, you can see same error
happens on trunk. So I think this is io module's bug. (py3k is using io
module deeply)
msg82827 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-02-27 10:04
This OSError(5) happens when we tries to read from pty after data runs out.
So simple_test_2.py fails with same error even if we don't use io module.

Modules/posixmodule.c (posix_read) simply calls read(2) once, but io module

    while avail < n:
        chunk = self.raw.read(wanted)
        if chunk in empty_values:
            nodata_val = chunk
            break
        avail += len(chunk)
        chunks.append(chunk)

chunk is shorter than wanted (data runs out), but not empty, so io
module's read tries to read again => error happens.

I said this is io module's bug, but now I'm not sure.
msg82828 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-27 10:28
Interesting. The reason the io module calls read() more than once is
that BufferedReader is a generic wrapper which can be used on different
kinds of file-like objects, including sockets.

I'm not sure how to satisfy that use-case without compromising normal
error-handling behaviour. Perhaps the FileIO object, when receiving an
errno=5 on read(), should check for S_IFIFO on the fstat() result and
then return an empty string instead?
msg82861 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-02-27 19:27
IIUC the problem is that a read() syscall on the pty after the other end
has been closed raises an error instead of reading 0 bytes?  Isn't that
a bug in the pty implementation? For lots of devices (e.g. sockets,
pipes) a short non-empty read just means that you have to call read()
again to get more data -- it does not mean EOF.
msg82871 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-27 20:56
Guido: I don't know if it can be considered as a "bug" rather than a
misguided "feature". However, at least 3 of us (the OP, Hirokazu and I)
reproduce it (as for me, it's on a quite recent x86-64 Mandriva Linux
setup), so I imagine it's not totally exotic behaviour.
msg82879 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-02-27 22:04
That may be how it works, but how do you expect to deal with it?
msg82880 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-27 22:12
Well, as I suggested, in FileIO.read(): when receiving errno=5 on a
read() call and if S_IFIFO() returns true, clear errno and return an
empty string.
The question is whether a genuine EIO error ("low level IO error") can
occur on a FIFO. Intuitively, I'd say "no" since a FIFO is only a
software communication channel, but who knows...
msg82882 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-02-27 22:27
> Well, as I suggested, in FileIO.read(): when receiving errno=5 on a
> read() call and if S_IFIFO() returns true, clear errno and return an
> empty string.
> The question is whether a genuine EIO error ("low level IO error") can
> occur on a FIFO. Intuitively, I'd say "no" since a FIFO is only a
> software communication channel, but who knows...

OK, that sounds reasonable. (I missed that in the discussion on the
bug, sorry. I tend not to download files unless I actually am on the
hook for code reviewing them, so any details that were only obvious
from the patch may have gone by me.)

Of course, you should check if those symbols even exist before referencing them.
msg87763 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-14 20:53
Will try to work out a patch before the RC.
msg87926 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-16 20:38
Uh, a file descriptor returned by openpty() doesn't satisfy S_ISFIFO().
It's just reported as a character device by fstat (st_mode is 0o20666).
Perhaps the best thing is to just let the error propagate, since after
all the user tries to read more bytes than are available.
msg102797 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-10 22:03
It turns out isatty() returns True for fds created by openpty(), which helps quite a bit: we can silence EIO for ttys while keeping it for non-ttys, where a low-level I/O (hardware) error should be raised properly.
Here is a patch for trunk.
msg102802 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 00:44
After triggering the buildbots, it seems that reading from the master fd
after the slave fd has been closed is rather OS-dependent. The FreeBSDs
return an empty string. I wonder whether this use case (reading from the
master after the slave is closed) should really be supported.
msg102803 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 00:55
Regardless of the decision, a new patch.
msg102804 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 01:14
Worse, the test timed out (probably froze) on the Solaris buildbot:
http://www.python.org/dev/buildbot/trunk/builders/sparc%20solaris10%20gcc%20trunk/builds/649/steps/test/logs/stdio
msg126683 - (view) Author: Zac Medico (zmedico) Date: 2011-01-21 02:28
This issue no longer appears to be a problem for my purposes, since it seems that array.fromfile() does not lose any data as long as the input file is opened in unbuffered mode (I use fdopen with 0 for the bufsize argument).
History
Date User Action Args
2013-01-12 12:49:35marciofsetnosy: + marciof
2011-01-21 02:28:30zmedicosetnosy: gvanrossum, exarkun, amaury.forgeotdarc, pitrou, ocean-city, Arfrever, marduk, zmedico
messages: + msg126683
2010-04-11 01:14:59pitrousetmessages: + msg102804
2010-04-11 00:55:42pitrousetfiles: - io-openpty.patch
2010-04-11 00:55:21pitrousetfiles: + io-openpty.patch

messages: + msg102803
2010-04-11 00:44:02pitrousetmessages: + msg102802
2010-04-10 22:17:47pitrousetfiles: + io-openpty.patch
2010-04-10 22:17:39pitrousetfiles: - io-openpty.patch
2010-04-10 22:03:51pitrousetfiles: + io-openpty.patch
versions: + Python 2.7, Python 3.2
messages: + msg102797

keywords: + patch
stage: needs patch -> patch review
2010-03-02 23:58:07marduksetnosy: + marduk
2009-09-25 04:52:23Arfreversetnosy: + Arfrever
2009-05-16 20:38:31pitrousetassignee: pitrou ->
messages: + msg87926
2009-05-14 20:53:08pitrousetpriority: normal

assignee: pitrou
components: + IO
versions: + Python 3.1, - Python 3.0
nosy: gvanrossum, exarkun, amaury.forgeotdarc, pitrou, ocean-city, zmedico
messages: + msg87763
stage: needs patch
2009-02-27 22:27:52gvanrossumsetmessages: + msg82882
2009-02-27 22:12:24pitrousetmessages: + msg82880
2009-02-27 22:04:28gvanrossumsetmessages: + msg82879
2009-02-27 20:56:41pitrousetmessages: + msg82871
2009-02-27 19:51:55exarkunsetnosy: + exarkun
2009-02-27 19:27:01gvanrossumsetmessages: + msg82861
2009-02-27 10:28:14pitrousetnosy: + gvanrossum
2009-02-27 10:28:01pitrousetnosy: + amaury.forgeotdarc
messages: + msg82828
2009-02-27 10:07:07ocean-citysetnosy: + pitrou
2009-02-27 10:04:49ocean-citysetfiles: + simple_test_2.py
messages: + msg82827
2009-02-27 09:38:40ocean-citysetfiles: + simple_test.py
nosy: + ocean-city
messages: + msg82826
title: array.fromfile() on master pty raises IOError when slave pty device is closed -> pty.read raises IOError when slave pty device is closed
2009-02-27 08:03:32zmedicocreate