Currently, if one redirects stdout, readline hook is not used (see https://hg.python.org/cpython/file/default/Parser/myreadline.c#l208). I would assume that if I run Python as "py -i > output.txt", I can use GNU readline or other readline hook for input just like normal; only the output is redirected to the file.

I think Gnu Readline uses standard output for echoing input, prompts, controlling the cursor, etc. You can already see this by redirecting to a separate terminal. In Linux:

$ python > /dev/pts/2

So I think your assumption is not valid. Perhaps you can redirect standard error instead? (python -i . . . 2> output.txt, at least in Linux.) Or maybe use some terminal logging program, like the “script” Unix command. But I’m not sure there is a good use case for enabling Readline without a terminal to output to.

I really don't know how stdio in console in Linux works. In my package https://github.com/Drekin/win-unicode-console I set custom sys.std* streams and a custom readline hook to support Unicode in Python run in Windows console. If I run `py` with my fixes enabled, >>> "α" really yields "α", but when I run `py -i > output.txt`, >>> "α" yields "?" (written to the file of course) same as when my fixes are disabled. From this point of view it seemed wrong that redirection of output affects how a line is read.

Regarding the prompt, I think it should be always written to sys.stdout, so we may hypothetically add a wrapper around GNU readline writing the prompt to sys.stdout and calling the actual readline with empty prompt.

So you are telling me that the characters displayed in console when entering input are actually written to stdout in Linux (I think it's not the case in Windows)? Maybe it woudn't matter if they were written to console even when stdout is redirected.

I'm not saying it is extra useful to redirect stdout while providing interactive input, I was just wondering what should be the right behavior. And from my point of view motivated as explained above, readlinehook should be used even with redirected stdout, at least in Windows.

The behavior of programs definitely differs if stdout is redirected.  For example on linux a program might output color codes to stdout normally, but if it is redirected to a non-tty, no color codes would be output.

Windows console handling of unicode is an entirely separate problem for which there are open issues and some possible solutions being worked on.

R. David Murray: I understand that the behavior of programs differ if stdout is redirected. I just claim that at least in Windows, stdin and stdout are completely independent, so redirecting *stdout* shouldn't affect how data are read from *stdin*.

I know that Unicode handling in Windows is a completely different issue. I was just explaining how I ran into this issue. During my work on `win-unicode-console` I found out there is some (at least for me) suprising behavior regarding the stdio implementation in Python. So I raised issues (this one, issue17620, issue18597, issue18838).

Well, I think we're constrained by the behavior of readline here.

Behavior of which readline? GNU readline? Note that it is only one particular implmentation on one particular platform while Python should be as multiplatform as possible / viable.

And what behavior it is? More precisely, what incorrect behavior in Linux would it bring if the condition in 

    if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout)))
        rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt);
    else
        rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, prompt);

was replaced by `!isatty (fileno (sys_stdin))`?

It's the one we started with, and the one in most common use.

I don't even know if there would be a problem.  The thing to do would be to try it and find out (on both linux and windows).

In Linux, the original stdin, stdout, stderr file descriptors refer to the same console by default, and you use normal file reads and writes on them. In Linux when Python uses Gnu Readline, the displayed input characters are indeed written to the original stdout file descriptor. For comparison, when Readline is not used, the displayed characters go to the original stdin console, because they are handled at a lower OS level before they are even read by Python and Readline.

By experimentation, the Bash shell seems to work similarly, except it uses stderr rather than stdout. You can freely redirect stdout, but if you redirect stderr it affects the output seen on the terminal. It may be possible to write to stdin to avoid the input and output being split up, but I think writing to an input-only file descriptor is a bad idea.

If you dropped the isatty() check for stdout in Linux, you may end up with the output of Readline in the file (assuming Readline is okay with this). The file would include all the cursor positioning codes, backspaces, etc, that Readline generates. But I haven’t actually tried this.

> If you dropped the isatty() check for stdout in Linux, you may 
> end up with the output of Readline in the file (assuming
> Readline is okay with this). The file would include all the
> cursor positioning codes, backspaces, etc, that Readline
> generates. But I haven’t actually tried this.

Yes, that's what happens. 

It seems to me the isatty check could be moved to call_readline in Modules/readline.c. This way PyOS_ReadlineFunctionPointer decides whether to call PyOS_StdioReadline. (See issue 512981 for the patch that introduced the isatty check.)

A related issue in Windows is that when stdout isn't a console, the text encoding defaults to the locale's ANSI encoding (e.g. cp1252). UTF-8 can be forced by setting the environment variable PYTHONIOENCODING=utf-8.

Also of concern is the pyreadline module. Its [Console class] assumes that the process standard handles are console handles. This is generally valid given the isatty check. OTOH, when I removed this check and redirected stdout to a file, pyreadline crashed in an endless loop. 

pyreadline's hook could punt to PyOS_StdioReadline if stdin and stdout don't both refer to the console. The problem there is calling the CRT's _fileno (POSIX) function in Python. Maybe the os module could wrap fileno in Modules/posixmodule.c. In addition to accepting a FILE pointer, os.fileno could accept an enum of C_STDIN, C_STDOUT, and C_STDERR.

[1]: https://github.com/pyreadline/pyreadline/blob/0be3f019ecfdaf740555dc91a1156ddd5dd195f2/pyreadline/console/console.py#L176

I recommend this feature request be rejected.  The current behavior has been in place for a long time.  Changing it may cause more harm than good (especially if the behavior between operating systems becomes more divergent).

How about reconsidering in the case that the machinery around PyOS_Readline is rewritten as I suggest in #17620 ?