14:23 ~ $ python3
Python 3.3a0 (default:4a5782a2b074, Mar 21 2011, 15:20:28)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more
information.
>>> ^Z
[1]+  Stopped                 python3
14:25 ~ $
14:25 ~ $ fg
python3

[56455 refs]
[36537 refs]


And


14:29 ~/src/cpython $ ./python.exe
Python 3.3a0 (default:267578b2422d, Mar 23 2011, 13:27:15)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more
information.
>>> ^Z
[3]+  Stopped                 ./python.exe
14:29 ~/src/cpython $ fg
./python.exe

[56559 refs]
[36610 refs]

I don't have this behaviour on Linux. Is it specific to Mac OS X?

On Wed, Mar 23, 2011 at 01:38:46PM +0000, STINNER Victor wrote:
> I don't have this behaviour on Linux. Is it specific to Mac OS X?

(Wish i could tell ;-)

On linux it looks the same for me, but when I press enter the prompt appears again:

$ ./python 
Python 3.3a0 (default:f8d6f6797909, Mar 20 2011, 05:55:16)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
[1]+  Stopped           ./python
wolf@hp:~/dev/py/py3k$ fg
./python

<hit enter here>
[57710 refs]
>>>

On Wed, Mar 23, 2011 at 01:44:06PM +0000, Ezio Melotti wrote:
> On linux it looks the same for me, but when I press enter the prompt appears again:

14:49 ~ $ jobs
14:49 ~ $ python3
Python 3.3a0 (default:4a5782a2b074, Mar 21 2011, 15:20:28) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more 
information.
>>> ^Z
[1]+  Stopped                 python3
14:49 ~ $ fg
python3

[56455 refs]
[36537 refs]
14:49 ~ $ jobs
14:49 ~ $

8~~~~)

What's the problem here ?
CTRL-Z causes the controlling terminal to send a SIGTSTP to the process, and the default handler stops the process, pretty much like a SIGSTOP.
If you don't want that to happen:
import signal
signal.signal(signal.SIGTSTP, signal.SIG_IGN)

On Wed, Mar 23, 2011 at 02:05:46PM +0000, Charles-Francois Natali wrote:
> What's the problem here ?
> CTRL-Z causes the controlling terminal to send a SIGTSTP to the process, and the default handler stops the process, pretty much like a SIGSTOP.
> If you don't want that to happen:
> import signal
> signal.signal(signal.SIGTSTP, signal.SIG_IGN)

(What's happening: it's so unresponsive when i try your code .. :)
Rather, i want an interactive python to integrate itself 
neatlessly into normal shell job control, say! 
Thus i always hope that a program takes care about SIGCONT!!

I'm stuck here, grep(1)ing everywhere and only find 
Modules/signalmodule.c for SIGCONT and SIGTSTP (it's *NOT* SunOS). 
Where is Python handling job control?

On Wed, Mar 23, 2011 at 02:05:46PM +0000, Charles-Francois Natali wrote:
> import signal
> signal.signal(signal.SIGTSTP, signal.SIG_IGN)

15:27 ~/tmp $ python3
Python 3.3a0 (default:4a5782a2b074, Mar 21 2011, 15:20:28) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more 
information.
>>> import signal
[56457 refs]
>>> signal.signal(signal.SIGCONT, lambda sig,frame: print('GOT SIG', sig))
0
[56488 refs]
>>> ^Z
[1]+  Stopped                 python3
15:29 ~/tmp $ fg
python3
GOT SIG 19

[56489 refs]
[36546 refs]
15:29 ~/tmp $

davide@macrisorto ~/cpython $ ./python.exe 
Python 3.3a0 (default:4a5782a2b074, Mar 23 2011, 15:26:35) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z
[1]+  Stopped                 ./python.exe
davide@macrisorto ~/cpython $ fg
./python.exe

davide@macrisorto ~/cpython $ 


----

System Python on OS X 10.6.6:


davide@macrisorto ~/py2.6 $ python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
[1]+  Stopped                 python
davide@macrisorto ~/py2.6 $ fg
python

>>> 
>>> ^D
davide@macrisorto ~/py2.6 $ 

(works as expected)


------

Python 2.6.6 from hg:

davide@macrisorto ~/py2.6 $ ./python.exe 
Python 2.6.6+ (unknown, Mar 23 2011, 15:19:22) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z
[1]+  Stopped                 ./python.exe
davide@macrisorto ~/py2.6 $ fg
./python.exe

[38594 refs]
[16875 refs]
davide@macrisorto ~/py2.6 $

(same behavior as first post)

I'm still not sure I understand the problem.
- when you hit CTRL-Z, the process is put in background, since it receives a SIGTSTP : normal
- when you put it in foreground with 'fg', it doesn't resume ? Did you try to hit <ENTER> to have sys.ps1 '>>> ' printed to stdout ? Or did the process exit ?

The process did exit on fg. Compare with the 2nd paste on my previous message (Python shipped with OS X).

The exit status code is always 0. 
It seems to me somewhere in a run() somebody sets some 'do exit' 
and thus causing a normal exit. 
But i really can't find something down in pythonrun.c at a short 
glance (and i just dived shallow into Python yet), and i'm in 
a hurry, too much to compile+fprintf() pythonrun.c right now. 
Maybe someone with glue on Python mainloop can give some hints, 
i'll try this evening to instrument the call graph a bit.

In that case, it's likely due to the way OS-X handles interrupted syscalls.
Under Linux, getchar and friends (actually read with default SA_RESTART) won't return EINTR on (SIGSTOP|SIGTSTP)/SIGCONT.
Under OS-X, it seems that e.g. getchar (read) does return EOF with errno set to EINTR, in which case the interactive interpreter will exit, if errno is not checked.
Out of curiosity, could you try the C snippet:

#include <stdio.h>

int main(int argc, char *argv[])
{
    int c;

    if ((c = getchar()) == EOF) {
        perror("getchar");
    }

    return 0;
}

And interrupt it with CTRL-Z ?

You are right. The previous runs were without readline. With readline it behaves as expected.

For the sake of completeness, here's the output of your snippet after Ctrl+Z, fg:
getchar: Interrupted system call

my_fgets in Parser/myreadline.c is broken:
There's a comment saying that a fgets is retried on EINTR, but the code doesn't retry. It used to in older cPython versions, but there was also a bug, so my guess is that this bug has been here for a long time.
Could you try with the attached patch ?
It's just a quick'n dirty patch, but it should fix it.

The patch works fine, thank you.

I was trying the same fix, but got stuck trying to understand what led to the decision in issue 960406. Still not sure.

[versions=Python 3.2,Python 3.1,Python 2.7;nosy:+akuchling]
Reply-To: 
In-Reply-To: <72975942018.issue11650@psf.upfronthosting.co.za">1300905163.47.0.72975942018.issue11650@psf.upfronthosting.co.za>

On Wed, Mar 23, 2011 at 06:32:43PM +0000, Charles-Francois Natali wrote:
> my_fgets Parser/myreadline.c patch should fix it.

Works perfect! 
Wish i would always find things worked out when i come home :). 
(And would have taken a long time for me to come to Parser/, 
i think, so, even more thanks for the patch, Charles-Francois!)

The Apple-shipped Python's (2.5, 2.6) work correct, but my 
self-compiled 2.7 and unpatched 3.3 do not. 
My self-compiled backward-test 3.2rc2 works, and you are right, 
Davide, that has readline.so, the others not.
(Building of the readline module requires either MacOS X magic or 
setting of the MACOSX_DEPLOYMENT_TARGET= configure variable, 
though;  Ronald and Ned have worked on that on PyCon (#11485), but 
this is not yet included and so...)

And regardless of all that my_fgets() reacts falsely (in respect 
to it's own preceeding comment), so this is a bug, no? 
I make Andrew M. Kuchling nosy, he committed 5fcb0e0be89e.

(Oh - when will i get this tracker right?
Is there somewhere *real* documentation which applies
to what actually happens??
Sorry once again, all of you!)

The faulty behavior was presumably introduced in r36346, when the continue statement was removed. I already linked the relevant discussion, but I'm not sure whether that was a desired change or just an oversight. I'm inclined to believe the latter, as it wouldn't have made much sense to leave the loop there otherwise. In this case a patch like the already posted one would do the fix.

Steffen, on a side note, I got readline working with brew.

Sorry to ask that, but would it be possible to write an automated test for this issue?

I couldn't get an automated test to fail on the bug with subprocess. Apparently an interpreter launched with Popen is not subject to the bug as it is when launched directly from the shell (i.e. fgets is not interrupted by a SIGSTOP sent by the parent process), so I may be getting something wrong here.
However I don't believe the important thing is to test this particular issue (which may be unimportant by itself if nobody noticed it for seven years, and not many build Python without readline) but rather make sure it doesn't break things elsewhere. As I understand it the code in r36346 didn't have proper tests, so we can only guess?

On Thu, Mar 23, 2011 at 21:50:42PM +0000, Davide Rizzo wrote:
> Steffen, on a side note, I got readline working with brew.

Say - readline not libedit which does not take care of .inputrc? 
Without permanently modifying GNU autoconf stuff? 
How do you do that?

To be clear, I have no idea why the patch for issue 960406 removed the continue from my_fgets.  It may have been simply a mistake.

The bugs seems not to be limited to the REPL.

# Python 2.6 with readline on Mac OS X
$ python -c "raw_input()"
^Z
[1]+  Stopped                 python -c "raw_input()"
$ fg
python -c "raw_input()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
EOFError


# Python 3.1 on Debian Linux
$ python3.1 -c "print(input())"
^Z
[1]+  Stopped                 python3.1 -c "print(input())"
$ fg
python3.1 -c "print(input())"
hello
hello


# Python 3.3(267578b2422d) without readline, OS X (I've put two extra printfs around fgets in myreadline.c)
$ ./python.exe -c "print(input())"
Entering fgets
^Z
[1]+  Stopped                 ./python.exe -c "print(input())"
$ fg
./python.exe -c "print(input())"
fgets returned 0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
EOFError
[36537 refs]


# same as before with Charles-Francois patch applied
$ ./python.exe -c "print(input())"
^Z
[1]+  Stopped                 ./python.exe -c "print(input())"
$ fg
./python.exe -c "print(input())"
hello
hello
[36538 refs]


Some remarks here:
1) the bug is in all tested versions running on OS X, and the patch does fix it.

2) I tested with and without GNU readline and input()'s behavior doesn't change when running with python -c (i.e. it uses myreadline.c instead of GNU readline). Though, if run interactively, input() uses readline.
Isn't input() supposed to use readline whenever reading from a tty, not only if called from the REPL? Or even if it were to fallback, shouldn't that be on TextIOWrapper(sys.stdin).readline()?

3) io module seems to be doing it right already. sys.stdin.read() raises an IOError (not an EOFError as input() incorrectly does, this is acceptable as a platform quirk) when interrupted, while sys.stdin.readline() handles the interruption without complaining at all.
Maybe myreadline.c is too old and wants to be replaced (or backed) by io in 3.3?

On Fri, Mar 25, 2011 at 12:25:21PM +0000, Davide Rizzo wrote:
> Some remarks here:
> 1) the patch does fix it.

Yes!

I can't comment the rest except for what
    .. function:: input([prompt])
states, which Davide surely have read himself.

Anyway it seems to be a good idea to replace parts of the code 
where such severe bugs can sleep for such a long time. 
This code will not be used automatically on, yes, *all* modern 
UNIX platforms (don't let me think on what "modern" means here). 

So if there are no dependencies i don't know of (but even without 
knowing i would think there should be none - this is readline), 
it seems that providing a simple Python-based fallback 
implementation, which will use functions which themselves will be 
used daily in real life, is a good idea.

\|/
   _ .
    |
    =

Yes!?

An effective way to automate user interaction tests is by using pseudo-terminals. The attached test uses Pexpect and bash as tools to test the issue and demonstrates the bug on affected builds/platforms. There are a number of (possibly related) other issues that may require similar testing.

Pexpect is a nice tool to automate testing over ptys, but it is Python 2. I had to run the test on Python 3.3 through the installed Python 2 interpter. My thought: the smallest useful subset of Pexpect can be ported to Py3 and maintained within Python. Or alternatively an equally useful minimal implementation of the same features. Note that most of Pexpect is already tested.

I've used bash to test this one issue because it was ready and I could rely on bash jobs handling. I don't know how good is to expect bash to be on the system or whether any sh-like shell is compatible.

There is a WinPexpect which supposedly does the same thing on Windows in a native way. I couldn't get the time to try it out, but it includes a modified version of Pexpect and claims Py3 compatibility.

Also, I don't really understand sdaoden ascii art messages.

Let me know if someone wants to work on this with me. :)

Ping!
Note that whatever reason caused jesstess, to name a few,
to drop that loop (and the continue), Charles-Francois posted
a correctly working patch!
I have no idea why such a severe bug could sleep in code which is
executed for each and every input(), but it needs to be fixed!

Do you want a robot test?
I've tested it dozens of times and it still works fine!!
It's idiot-proof!
(And i definitely have no time in my free-time to write an
automated test which fiddles around with timeout values
and such to simulate user input.
To me all of this is mystic.
There are almost 14000 *registered* Python users here!!)

New changeset 2222f343ac51 by Victor Stinner in branch '3.1':
Issue #11650: PyOS_StdioReadline() retries fgets() if it was interrupted
http://hg.python.org/cpython/rev/2222f343ac51

New changeset fc2f251e660a by Victor Stinner in branch '3.2':
(Merge 3.1) Issue #11650: PyOS_StdioReadline() retries fgets() if it was
http://hg.python.org/cpython/rev/fc2f251e660a

New changeset 64de1ded0744 by Victor Stinner in branch 'default':
(Merge 3.2) Issue #11650: PyOS_StdioReadline() retries fgets() if it was
http://hg.python.org/cpython/rev/64de1ded0744

New changeset 7febd5ef7619 by Victor Stinner in branch '2.7':
(Merge 3.1) Issue #11650: PyOS_StdioReadline() retries fgets() if it was
http://hg.python.org/cpython/rev/7febd5ef7619

Merci, STINNER Victor!

I emulated Mac OS X behaviour on Linux by hacking my_fgets(): do { p=NULL; errno = EINTR; }, only after the first call to fgets(). Without the patch, Python does exit immediatly. With the patch, Python doesn't exit.

I applied neologix's patch to 3.1, 3.2, 3.3 and 2.7: thank you Charles-Francois!

Can someone retest on Mac OS X?

I noticied a strange behaviour:
----------
$ python
>>> 1+^Z
[1]+  Stopped                 ./python
$ fg
./python
2<ENTER>
2
----------
"1+2" input becomes "2". But we can not do better because buf doesn't contain "1+\0" when fgets() is interrupted.

Note: If the readline module is available, it is used, and readline doesn't have this issue. And "1+^Z(...)2" gives 3 with readline.

Victor, I have neither OS X nor Linux available right now, but if I remember correctly the same happens on both systems when programs call input() (but not from the REPL). See also my previous message with "python -c" tests and my second remark.

What about the Pexpect based tests? Could it be interesting to Python to test the behavior on the terminal?

About the committed patch: is it the right thing to do to call PyOS_InputHook at every restart?

Thank you!

On Sat, Apr 09, 2011 at 02:18:01PM +0000, STINNER Victor wrote:
> I noticied a strange behaviour:

So forget all this girlie s...!
Here is a real man's patch!!
You'll notice mysterious function calls with a Py prefix -
they're necessary in this environment, but anyway sorry for that.
(L'ascenseur est là, à gauche.
And i'll hope to come to lesson 2 soon...)

P.S.: this is very simple minded in respect to multibyte 
characters; our terminal library does something like this
(and the complete picture is even more complicated):

jnext:  uni = s_textcodec_decode_single(_sterm_io_codec, cbuf, cbuf_len,
                                        &bytes_consumed);
        cbuf += bytes_consumed;
        cbuf_len -= bytes_consumed;
        if (s_PREDICT_FALSE(uni < 0)) {
                if (uni == -s_ERROR_INPROGRESS) {
                        seq_cpl = s_FAL0;
                        goto jleave;
                }
                /*if (uni == -s_ERROR_ILSEQ || uni == -s_ERROR_INVAL) {*/
                        cbuf_len = 0;
                        uni = sterm_KEY_UNKNOWN;
                        goto jemit;
                /*}*/
        }

On Sat, Apr 09, 2011 at 02:18:01PM +0000, STINNER Victor wrote:
> I noticied a strange behaviour:

Still fun, but this one could even make it except for termios
flags, multibyte and the real problem, signal handling.
Hm.

I've opened it and it's fixed, so i'll close it now.

If someone finds the single bug in 11650.termios-1.diff
in termios_resume() and also has an idea of how to call
termios_suspend() in case Python crashes or gives back
the terminal in any other way i would vote for reopening
this and using my version, because it is better.

Take care otherwise.
Thanks, haypo.