With python-3.0, array.fromfile() raises an IOError when reading from a
master pty device after the slave device has been closed. This causes
remaining data that had been written to the slave device to be lost. I
have observed this problem with python-3.0.1 on linux (I get the same
result with or without the patch from issue 5334). The traceback
produced by the attached test case looks like this:

    Traceback (most recent call last):
    File "./fromfile_pty_ioerror.py", line 26, in <module>
        buf.fromfile(master_file, bufsize)
    File "/usr/lib/python3.0/io.py", line 918, in read
        return self._read_unlocked(n)
    File "/usr/lib/python3.0/io.py", line 952, in _read_unlocked
        chunk = self.raw.read(wanted)
    IOError: [Errno 5] Input/output error

With python-2.x, the remaining data is appended to the array and an
EOFError is raised like one would expect.

I think this is not array modules' bug. Attached test program outputs
different results on trunk/py3k.

debian:~/python-dev/trunk# ./python /mnt/windows/simple_test.py
os.pipe: success
pty.openpty: success

debian:~/python-dev/py3k# ./python /mnt/windows/simpled_test.py
b'os.pipe: success'
Traceback (most recent call last):
  File "/mnt/windows/simpled_test.py", line 17, in <module>
    gotdata = master_file.read(len(data) + 1)
  File "/root/python-dev/py3k/Lib/io.py", line 918, in read
    return self._read_unlocked(n)
  File "/root/python-dev/py3k/Lib/io.py", line 952, in _read_unlocked
    chunk = self.raw.read(wanted)
IOError: [Errno 5] Input/output error

And if you use io.open instead of os.fdopen, you can see same error
happens on trunk. So I think this is io module's bug. (py3k is using io
module deeply)

This OSError(5) happens when we tries to read from pty after data runs out.
So simple_test_2.py fails with same error even if we don't use io module.

Modules/posixmodule.c (posix_read) simply calls read(2) once, but io module

    while avail < n:
        chunk = self.raw.read(wanted)
        if chunk in empty_values:
            nodata_val = chunk
            break
        avail += len(chunk)
        chunks.append(chunk)

chunk is shorter than wanted (data runs out), but not empty, so io
module's read tries to read again => error happens.

I said this is io module's bug, but now I'm not sure.

Interesting. The reason the io module calls read() more than once is
that BufferedReader is a generic wrapper which can be used on different
kinds of file-like objects, including sockets.

I'm not sure how to satisfy that use-case without compromising normal
error-handling behaviour. Perhaps the FileIO object, when receiving an
errno=5 on read(), should check for S_IFIFO on the fstat() result and
then return an empty string instead?

IIUC the problem is that a read() syscall on the pty after the other end
has been closed raises an error instead of reading 0 bytes?  Isn't that
a bug in the pty implementation? For lots of devices (e.g. sockets,
pipes) a short non-empty read just means that you have to call read()
again to get more data -- it does not mean EOF.

Guido: I don't know if it can be considered as a "bug" rather than a
misguided "feature". However, at least 3 of us (the OP, Hirokazu and I)
reproduce it (as for me, it's on a quite recent x86-64 Mandriva Linux
setup), so I imagine it's not totally exotic behaviour.

That may be how it works, but how do you expect to deal with it?

Well, as I suggested, in FileIO.read(): when receiving errno=5 on a
read() call and if S_IFIFO() returns true, clear errno and return an
empty string.
The question is whether a genuine EIO error ("low level IO error") can
occur on a FIFO. Intuitively, I'd say "no" since a FIFO is only a
software communication channel, but who knows...

> Well, as I suggested, in FileIO.read(): when receiving errno=5 on a
> read() call and if S_IFIFO() returns true, clear errno and return an
> empty string.
> The question is whether a genuine EIO error ("low level IO error") can
> occur on a FIFO. Intuitively, I'd say "no" since a FIFO is only a
> software communication channel, but who knows...

OK, that sounds reasonable. (I missed that in the discussion on the
bug, sorry. I tend not to download files unless I actually am on the
hook for code reviewing them, so any details that were only obvious
from the patch may have gone by me.)

Of course, you should check if those symbols even exist before referencing them.

Will try to work out a patch before the RC.

Uh, a file descriptor returned by openpty() doesn't satisfy S_ISFIFO().
It's just reported as a character device by fstat (st_mode is 0o20666).
Perhaps the best thing is to just let the error propagate, since after
all the user tries to read more bytes than are available.

It turns out isatty() returns True for fds created by openpty(), which helps quite a bit: we can silence EIO for ttys while keeping it for non-ttys, where a low-level I/O (hardware) error should be raised properly.
Here is a patch for trunk.

After triggering the buildbots, it seems that reading from the master fd
after the slave fd has been closed is rather OS-dependent. The FreeBSDs
return an empty string. I wonder whether this use case (reading from the
master after the slave is closed) should really be supported.

Regardless of the decision, a new patch.

Worse, the test timed out (probably froze) on the Solaris buildbot:
http://www.python.org/dev/buildbot/trunk/builders/sparc%20solaris10%20gcc%20trunk/builds/649/steps/test/logs/stdio

This issue no longer appears to be a problem for my purposes, since it seems that array.fromfile() does not lose any data as long as the input file is opened in unbuffered mode (I use fdopen with 0 for the bufsize argument).

I don’t think it is right to assume pseudo-terminals behave exactly like pipes. I suspect the behaviour when the slave is closed is undefined, or at best platform-specific, and there is no bug to fix in Python.

Also, it looks like array.fromfile() assumes any short read implies EOF, so the unbuffered workaround may not work in all cases. The documentation is vague on what it accepts, but I suspect it only works properly with a binary “buffered” file in general.

Indeed, ptys and pipes behave very differently (that's why we have both -- they serve different purposes).

More relevant question: we should find a way to decide whether to either apply Antoine's patch or close this issue.

Finally, maybe array.fromfile() can be fixed to deal better with short reads (if that's a real issue)? (However *empty* reads should still instantly stop it. Otherwise you end up busy-waiting.)

TLPI (The Linux Programming Interface book) says about the pty implementation on linux, at section 5 of chapter 64:

    If we close all file descriptors referring to the pseudoterminal slave, then:
    a) A read() from the master device fails with the error EIO. (On some other UNIX implementations, a read() returns end-of-file in this case.)

and also adds this (which is slightly off topic here):

    b)  A write() to the master device succeeds, unless the input queue of the slave device is full, in which case the write() blocks. If the slave device is subsequently reopened, these bytes can be read.

    UNIX implementations vary widely in their behavior for the last case. On some UNIX implementations, write() fails with the error EIO. On other implementations, write() succeeds, but the output bytes are discarded (i.e., they can’t be read if the slave is reopened). In general, these variations don’t present a problem. Normally, the process on the master side detects that the slave has been closed because a read() from the master returns end-of-file or fails. At this point, the process performs no further writes to the master.

Honestly, Antoine's patch looks reasonable to me (except that the names of the test modules perpetuate the confusion that pipes and ptys are similar).

Can someone just port that to 3.6? (The change in semantics is big enough that I don't think we should shove it into a bugfix release.)

In other news, the pty.py module is pretty pathetic. Kill or improve? It feels like one of those "included" batteries that runs out of juice after 5 minutes.

My biggest worry about the patch is that it looks like it will mask other EIO errors from normal terminals (e.g. perhaps reading from a slave or a real serial port can also produce an EIO error that does not mean EOF). Another option may be to add a specialized pseudo terminal master wrapper class that does the right thing on Linux. But I’m not a pseudo terminal expert and don’t really have a strong view about this.

I agree that the “pty” module is rather limited, although the documentation does say pty.openpty() is more portable than os.openpty(). Maybe if Python used posix_openpt(), portability would be less of a problem?

So maybe we should just close this as won't fix. Whoever wants to use the
master end of a pty had better be prepared for that IOError.

On Thursday, October 8, 2015, Martin Panter <report@bugs.python.org> wrote:

>
> Martin Panter added the comment:
>
> My biggest worry about the patch is that it looks like it will mask other
> EIO errors from normal terminals (e.g. perhaps reading from a slave or a
> real serial port can also produce an EIO error that does not mean EOF).
> Another option may be to add a specialized pseudo terminal master wrapper
> class that does the right thing on Linux. But I’m not a pseudo terminal
> expert and don’t really have a strong view about this.
>
> I agree that the “pty” module is rather limited, although the
> documentation does say pty.openpty() is more portable than os.openpty().
> Maybe if Python used posix_openpt(), portability would be less of a problem?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org <javascript:;>>
> <http://bugs.python.org/issue5380>
> _______________________________________
>

The line discipline [1] of a terminal driver and a pty is controlled by terminal attributes set with tcsetattr() [2][3].

IMHO (as a developer running ptys over asyncio), using a pty implies writing code at the low level such as configuring the line discipline, or possibly dealing with the notions of controlling terminal, process group and so on. So it seems right that Python io would not support ptys as Python io sits at a higher level.

[1] http://en.wikipedia.org/wiki/Line_discipline
[2] http://www.gnu.org/software/libc/manual/html_node/Terminal-Modes.html
[3] http://docs.python.org/3/library/termios.html?highlight=termios#module-termios

OK, I'm closing this as won't fix. If someone disagrees please explain.