[forwarded from http://bugs.debian.org/315888]

for line in sys.stdin: doesn't notice EOF the first time when reading from tty.

The test program:

    import sys
    for line in sys.stdin:
            print line,
    print "eof"

A sample session:

    liw@esme$ python foo.py
    foo         <--- I pressed Enter and then Ctrl-D
    foo         <--- then this appeared, but not more
    eof         <--- this only came when I pressed Ctrl-D a second time
    liw@esme$

Seems to me that there is some buffering issue where Python needs to
read end-of-file twice to notice it on all levels. Once should be 
enough.

Same thing occurs on Windows. Even worse, if the line does not end with CR, Ctrl-Z (EOF in Windows, equivalent to Ctrl-D) has to be pressed 3 times:

D:\Temp>python foo.py
foo  <--- I pressed Enter
^Z   <--- I pressed Ctrl-Z and then Enter again
foo  <--- this appeared
^Z   <--- I pressed Ctrl-Z and then Enter again
<EOF>

D:\Temp>python foo.py
foo^Z   <--- I pressed Ctrl-Z and then Enter
^Z      <--- cursor stays here; I pressed Ctrl-Z and then Enter again
^Z      <--- cursor stays here; I pressed Ctrl-Z and then Enter again
foo <EOF>

I am not entirely sure that this is a bug.

$ cat testfile
line1
line2

$ python foo.py < testfile

This command behaves as expected. Only when the input is from tty, the above described behaviour happens. That could be because of the terminal settings where characters may be buffered until a newline is entered.

I am not entirely sure that this is a bug.

$ cat testfile
line1
line2

$ python foo.py < testfile

This command behaves as expected. Only when the input is from tty, the above described behaviour happens. That could be because of the terminal settings where characters may be buffered until a newline is entered.

Sorry for my duplicate comment. It was a mistake. On closer examination, the OP's description does seem to indicate some issue. Please look at (attached) stdin_noiter.py which uses readline() directly and it does not have the problem described here. It properly detects EOF on first CTRL-D. This points to some problem with the iterator function fileobject.c:file_iternext(). I think that the first CTRL-D might be getting lost somewhere in the read ahead code (which only comes into picture with iterator).

Ok. This may sound stupid but I couldn't find a way to attach a file to this bug report. So I am copying the code here:

************
import sys

line = sys.stdin.readline()
while (line):
    print  line,
    line = sys.stdin.readline()

print "eof"
*************

I tested two kinds of inputs with iter and noiter verisons. I posted "noter" code and OP's code is the iter version.

1) For input without newline at all (line1<CTRL-D><CTRL-D><CTRL-D>) behaves same with both versions.
2) The noiter version prints "eof" with "line1\n<CTRL-D>" while the iter version requires an additional CTRL-D. This is because iter version uses read ahead which is implemented using fread() . A simple C program using fread() behaves exactly same way. 

I tested on Linux but am sure windows behaviour (as posted by  gagenellina) will have same reasons. Since the issue is with platform's stdio library, I don't think python should fix anything here. However, it may be worthwhile to mention something about this in documentation. I will open a bug for this purpose. 

BTW, I opened bug 1643712 for doc change.

I think this should be considered a bug.  These
two command lines (on unix) should behave the same:

cat | ./foo.py
./foo.py

But they do not.  The first (using cat) behaves typically,
needing only one control-D.  The second needs the two
control-D's as noted.

This is fixed in py3k but still exists in 2.7.

Benjamin, is it too late too have this fixed in 2.7?

I guess http://bugs.python.org/issue1195 might be related

This most certainly is a bug under Unix and an annoying one.  "Since the
issue is with platform's stdio library" is wrong;  stdio is being used
incorrectly.  It would be nice to see it fixed in the 2.x line.

I've two test programs.

    $ head -42 stdin2.6 stdin3.1
    ==> stdin2.6 <==
    #! /usr/bin/python2.6

    import sys

    for l in sys.stdin:
        print repr(l)
    print 'end'

    ==> stdin3.1 <==
    #! /usr/bin/python3.1

    import sys

    for l in sys.stdin:
        print(repr(l))
    print('end')

    $

For both of them I will type "1 Enter 2 Enter 3 Enter Ctrl-D" without
the spaces, Ctrl-D being my tty's EOF, stty -a.

    $ ./stdin2.6
    1
    2
    3
    '1\n'
    '2\n'
    '3\n'

On the EOF the first iteration of sys.stdin returns and then so do the
others with the buffered lines.  The loop doesn't terminate, a second
Ctrl-D is required, giving.

    end
    $

Next,

    $ ./stdin3.1
    1
    '1\n'
    2
    '2\n'
    3
    '3\n'
    end
    $

perfect output.  Only one Ctrl-D required and better still each line is
returned as it's entered.

ltrace shows python2.6 uses fread(3).  I'm assuming it treats only a
zero return as EOF whereas whenever the return value is less than the
number of requested elements, EOF could have been reached;  feof(3) must
be called afterwards to decide.  Really, ferror(3) should also be called
to see if, as well as returning some elements, an error was detected.

It's this lack of feof() that means the second fread() is required to
trigger the flawed `only 0 return is EOF' logic.

Here's some C that shows stdio works fine if feof() and ferror() are
combined with fread().

    #include <stdio.h>

    int main(void)
    {
        unsigned char s[8192], *p;
        size_t n;

        while ((n = fread(s, 1, 8192, stdin))) {
            printf("%zd", n);
            p = s;
            while (n--)
                printf(" %02hhx", *p++);
            putchar('\n');

            if (feof(stdin)) {
                puts("end");
                break;
            }
            if (ferror(stdin)) {
                fputs("error", stderr);
                return 1;
            }
        }

        return 0;
    }