classification
Title: pty.read raises IOError when slave pty device is closed
Type: behavior Stage: patch review
Components: IO, Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, amaury.forgeotdarc, exarkun, gvanrossum, marduk, martin.panter, ocean-city, pitrou, xdegaye, zmedico
Priority: normal Keywords: patch

Created on 2009-02-27 08:03 by zmedico, last changed 2015-10-10 15:38 by gvanrossum. This issue is now closed.

Files
File name Uploaded Description Edit
fromfile_pty_ioerror.py zmedico, 2009-02-27 08:03 test case demonstrating array.fromfile() failure with pty
simple_test.py ocean-city, 2009-02-27 09:38
simple_test_2.py ocean-city, 2009-02-27 10:04
io-openpty.patch pitrou, 2010-04-11 00:55 review
Messages (24)
msg82824 - (view) Author: Zac Medico (zmedico) Date: 2009-02-27 08:03
With python-3.0, array.fromfile() raises an IOError when reading from a
master pty device after the slave device has been closed. This causes
remaining data that had been written to the slave device to be lost. I
have observed this problem with python-3.0.1 on linux (I get the same
result with or without the patch from issue 5334). The traceback
produced by the attached test case looks like this:

    Traceback (most recent call last):
    File "./fromfile_pty_ioerror.py", line 26, in <module>
        buf.fromfile(master_file, bufsize)
    File "/usr/lib/python3.0/io.py", line 918, in read
        return self._read_unlocked(n)
    File "/usr/lib/python3.0/io.py", line 952, in _read_unlocked
        chunk = self.raw.read(wanted)
    IOError: [Errno 5] Input/output error

With python-2.x, the remaining data is appended to the array and an
EOFError is raised like one would expect.
msg82826 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-02-27 09:38
I think this is not array modules' bug. Attached test program outputs
different results on trunk/py3k.

debian:~/python-dev/trunk# ./python /mnt/windows/simple_test.py
os.pipe: success
pty.openpty: success

debian:~/python-dev/py3k# ./python /mnt/windows/simpled_test.py
b'os.pipe: success'
Traceback (most recent call last):
  File "/mnt/windows/simpled_test.py", line 17, in <module>
    gotdata = master_file.read(len(data) + 1)
  File "/root/python-dev/py3k/Lib/io.py", line 918, in read
    return self._read_unlocked(n)
  File "/root/python-dev/py3k/Lib/io.py", line 952, in _read_unlocked
    chunk = self.raw.read(wanted)
IOError: [Errno 5] Input/output error

And if you use io.open instead of os.fdopen, you can see same error
happens on trunk. So I think this is io module's bug. (py3k is using io
module deeply)
msg82827 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-02-27 10:04
This OSError(5) happens when we tries to read from pty after data runs out.
So simple_test_2.py fails with same error even if we don't use io module.

Modules/posixmodule.c (posix_read) simply calls read(2) once, but io module

    while avail < n:
        chunk = self.raw.read(wanted)
        if chunk in empty_values:
            nodata_val = chunk
            break
        avail += len(chunk)
        chunks.append(chunk)

chunk is shorter than wanted (data runs out), but not empty, so io
module's read tries to read again => error happens.

I said this is io module's bug, but now I'm not sure.
msg82828 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-27 10:28
Interesting. The reason the io module calls read() more than once is
that BufferedReader is a generic wrapper which can be used on different
kinds of file-like objects, including sockets.

I'm not sure how to satisfy that use-case without compromising normal
error-handling behaviour. Perhaps the FileIO object, when receiving an
errno=5 on read(), should check for S_IFIFO on the fstat() result and
then return an empty string instead?
msg82861 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-02-27 19:27
IIUC the problem is that a read() syscall on the pty after the other end
has been closed raises an error instead of reading 0 bytes?  Isn't that
a bug in the pty implementation? For lots of devices (e.g. sockets,
pipes) a short non-empty read just means that you have to call read()
again to get more data -- it does not mean EOF.
msg82871 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-27 20:56
Guido: I don't know if it can be considered as a "bug" rather than a
misguided "feature". However, at least 3 of us (the OP, Hirokazu and I)
reproduce it (as for me, it's on a quite recent x86-64 Mandriva Linux
setup), so I imagine it's not totally exotic behaviour.
msg82879 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-02-27 22:04
That may be how it works, but how do you expect to deal with it?
msg82880 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-27 22:12
Well, as I suggested, in FileIO.read(): when receiving errno=5 on a
read() call and if S_IFIFO() returns true, clear errno and return an
empty string.
The question is whether a genuine EIO error ("low level IO error") can
occur on a FIFO. Intuitively, I'd say "no" since a FIFO is only a
software communication channel, but who knows...
msg82882 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-02-27 22:27
> Well, as I suggested, in FileIO.read(): when receiving errno=5 on a
> read() call and if S_IFIFO() returns true, clear errno and return an
> empty string.
> The question is whether a genuine EIO error ("low level IO error") can
> occur on a FIFO. Intuitively, I'd say "no" since a FIFO is only a
> software communication channel, but who knows...

OK, that sounds reasonable. (I missed that in the discussion on the
bug, sorry. I tend not to download files unless I actually am on the
hook for code reviewing them, so any details that were only obvious
from the patch may have gone by me.)

Of course, you should check if those symbols even exist before referencing them.
msg87763 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-14 20:53
Will try to work out a patch before the RC.
msg87926 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-16 20:38
Uh, a file descriptor returned by openpty() doesn't satisfy S_ISFIFO().
It's just reported as a character device by fstat (st_mode is 0o20666).
Perhaps the best thing is to just let the error propagate, since after
all the user tries to read more bytes than are available.
msg102797 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-10 22:03
It turns out isatty() returns True for fds created by openpty(), which helps quite a bit: we can silence EIO for ttys while keeping it for non-ttys, where a low-level I/O (hardware) error should be raised properly.
Here is a patch for trunk.
msg102802 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 00:44
After triggering the buildbots, it seems that reading from the master fd
after the slave fd has been closed is rather OS-dependent. The FreeBSDs
return an empty string. I wonder whether this use case (reading from the
master after the slave is closed) should really be supported.
msg102803 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 00:55
Regardless of the decision, a new patch.
msg102804 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-04-11 01:14
Worse, the test timed out (probably froze) on the Solaris buildbot:
http://www.python.org/dev/buildbot/trunk/builders/sparc%20solaris10%20gcc%20trunk/builds/649/steps/test/logs/stdio
msg126683 - (view) Author: Zac Medico (zmedico) Date: 2011-01-21 02:28
This issue no longer appears to be a problem for my purposes, since it seems that array.fromfile() does not lose any data as long as the input file is opened in unbuffered mode (I use fdopen with 0 for the bufsize argument).
msg252464 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-10-07 11:33
I don’t think it is right to assume pseudo-terminals behave exactly like pipes. I suspect the behaviour when the slave is closed is undefined, or at best platform-specific, and there is no bug to fix in Python.

Also, it looks like array.fromfile() assumes any short read implies EOF, so the unbuffered workaround may not work in all cases. The documentation is vague on what it accepts, but I suspect it only works properly with a binary “buffered” file in general.
msg252478 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-10-07 16:30
Indeed, ptys and pipes behave very differently (that's why we have both -- they serve different purposes).

More relevant question: we should find a way to decide whether to either apply Antoine's patch or close this issue.

Finally, maybe array.fromfile() can be fixed to deal better with short reads (if that's a real issue)? (However *empty* reads should still instantly stop it. Otherwise you end up busy-waiting.)
msg252544 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2015-10-08 15:34
TLPI (The Linux Programming Interface book) says about the pty implementation on linux, at section 5 of chapter 64:

    If we close all file descriptors referring to the pseudoterminal slave, then:
    a) A read() from the master device fails with the error EIO. (On some other UNIX implementations, a read() returns end-of-file in this case.)

and also adds this (which is slightly off topic here):

    b)  A write() to the master device succeeds, unless the input queue of the slave device is full, in which case the write() blocks. If the slave device is subsequently reopened, these bytes can be read.

    UNIX implementations vary widely in their behavior for the last case. On some UNIX implementations, write() fails with the error EIO. On other implementations, write() succeeds, but the output bytes are discarded (i.e., they can’t be read if the slave is reopened). In general, these variations don’t present a problem. Normally, the process on the master side detects that the slave has been closed because a read() from the master returns end-of-file or fails. At this point, the process performs no further writes to the master.
msg252554 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-10-08 16:29
Honestly, Antoine's patch looks reasonable to me (except that the names of the test modules perpetuate the confusion that pipes and ptys are similar).

Can someone just port that to 3.6? (The change in semantics is big enough that I don't think we should shove it into a bugfix release.)

In other news, the pty.py module is pretty pathetic. Kill or improve? It feels like one of those "included" batteries that runs out of juice after 5 minutes.
msg252586 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-10-09 05:07
My biggest worry about the patch is that it looks like it will mask other EIO errors from normal terminals (e.g. perhaps reading from a slave or a real serial port can also produce an EIO error that does not mean EOF). Another option may be to add a specialized pseudo terminal master wrapper class that does the right thing on Linux. But I’m not a pseudo terminal expert and don’t really have a strong view about this.

I agree that the “pty” module is rather limited, although the documentation does say pty.openpty() is more portable than os.openpty(). Maybe if Python used posix_openpt(), portability would be less of a problem?
msg252587 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-10-09 05:18
So maybe we should just close this as won't fix. Whoever wants to use the
master end of a pty had better be prepared for that IOError.

On Thursday, October 8, 2015, Martin Panter <report@bugs.python.org> wrote:

>
> Martin Panter added the comment:
>
> My biggest worry about the patch is that it looks like it will mask other
> EIO errors from normal terminals (e.g. perhaps reading from a slave or a
> real serial port can also produce an EIO error that does not mean EOF).
> Another option may be to add a specialized pseudo terminal master wrapper
> class that does the right thing on Linux. But I’m not a pseudo terminal
> expert and don’t really have a strong view about this.
>
> I agree that the “pty” module is rather limited, although the
> documentation does say pty.openpty() is more portable than os.openpty().
> Maybe if Python used posix_openpt(), portability would be less of a problem?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org <javascript:;>>
> <http://bugs.python.org/issue5380>
> _______________________________________
>
msg252702 - (view) Author: Xavier de Gaye (xdegaye) * Date: 2015-10-10 10:33
The line discipline [1] of a terminal driver and a pty is controlled by terminal attributes set with tcsetattr() [2][3].

IMHO (as a developer running ptys over asyncio), using a pty implies writing code at the low level such as configuring the line discipline, or possibly dealing with the notions of controlling terminal, process group and so on. So it seems right that Python io would not support ptys as Python io sits at a higher level.

[1] http://en.wikipedia.org/wiki/Line_discipline
[2] http://www.gnu.org/software/libc/manual/html_node/Terminal-Modes.html
[3] http://docs.python.org/3/library/termios.html?highlight=termios#module-termios
msg252719 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-10-10 15:38
OK, I'm closing this as won't fix. If someone disagrees please explain.
History
Date User Action Args
2015-10-10 15:38:22gvanrossumsetstatus: open -> closed
resolution: wont fix
messages: + msg252719
2015-10-10 10:33:57xdegayesetmessages: + msg252702
2015-10-09 05:30:04marciosetnosy: - marcio
2015-10-09 05:18:33gvanrossumsetmessages: + msg252587
2015-10-09 05:07:33martin.pantersetmessages: + msg252586
2015-10-08 16:29:14gvanrossumsetmessages: + msg252554
2015-10-08 15:34:21xdegayesetnosy: + xdegaye
messages: + msg252544
2015-10-07 16:30:49gvanrossumsetmessages: + msg252478
2015-10-07 11:33:03martin.pantersetnosy: + martin.panter
messages: + msg252464
2013-01-12 12:49:35marciosetnosy: + marcio
2011-01-21 02:28:30zmedicosetnosy: gvanrossum, exarkun, amaury.forgeotdarc, pitrou, ocean-city, Arfrever, marduk, zmedico
messages: + msg126683
2010-04-11 01:14:59pitrousetmessages: + msg102804
2010-04-11 00:55:42pitrousetfiles: - io-openpty.patch
2010-04-11 00:55:21pitrousetfiles: + io-openpty.patch

messages: + msg102803
2010-04-11 00:44:02pitrousetmessages: + msg102802
2010-04-10 22:17:47pitrousetfiles: + io-openpty.patch
2010-04-10 22:17:39pitrousetfiles: - io-openpty.patch
2010-04-10 22:03:51pitrousetfiles: + io-openpty.patch
versions: + Python 2.7, Python 3.2
messages: + msg102797

keywords: + patch
stage: needs patch -> patch review
2010-03-02 23:58:07marduksetnosy: + marduk
2009-09-25 04:52:23Arfreversetnosy: + Arfrever
2009-05-16 20:38:31pitrousetassignee: pitrou ->
messages: + msg87926
2009-05-14 20:53:08pitrousetpriority: normal

assignee: pitrou
components: + IO
versions: + Python 3.1, - Python 3.0
nosy: gvanrossum, exarkun, amaury.forgeotdarc, pitrou, ocean-city, zmedico
messages: + msg87763
stage: needs patch
2009-02-27 22:27:52gvanrossumsetmessages: + msg82882
2009-02-27 22:12:24pitrousetmessages: + msg82880
2009-02-27 22:04:28gvanrossumsetmessages: + msg82879
2009-02-27 20:56:41pitrousetmessages: + msg82871
2009-02-27 19:51:55exarkunsetnosy: + exarkun
2009-02-27 19:27:01gvanrossumsetmessages: + msg82861
2009-02-27 10:28:14pitrousetnosy: + gvanrossum
2009-02-27 10:28:01pitrousetnosy: + amaury.forgeotdarc
messages: + msg82828
2009-02-27 10:07:07ocean-citysetnosy: + pitrou
2009-02-27 10:04:49ocean-citysetfiles: + simple_test_2.py
messages: + msg82827
2009-02-27 09:38:40ocean-citysetfiles: + simple_test.py
nosy: + ocean-city
messages: + msg82826
title: array.fromfile() on master pty raises IOError when slave pty device is closed -> pty.read raises IOError when slave pty device is closed
2009-02-27 08:03:32zmedicocreate