Title: document the special features (eg: fdclose=False) of the standard streams
Type: behavior Stage: needs patch
Components: Documentation, IO Versions: Python 3.10, Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Arfrever, akira, docs@python, eryksun, martin.panter, r.david.murray, rbcollins, snaphat, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2014-10-19 19:00 by snaphat, last changed 2021-02-23 11:40 by eryksun.

Messages (8)
msg229690 - (view) Author: Aaron Myles Landwehr (snaphat) Date: 2014-10-19 19:00
If I execute the following code, the file descriptor for CONOUT$ has a fileno != 1. With CONIN$ the fileno != 0. Similar code in another language such as perl produces the desired results. 

sys.stdout = open("CONOUT$", "w");

I believe it has to do with the fact that stdout is an object in python and in perl or c, you are operating directly on the stream ala freopen() or the equivalent.
msg229694 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-19 20:39
This is Windows specific, right?

From a quick google is not clear that CONOUT$ *should* point 1, since it supposed to get at the console regardless of whether there has been any redirection.  Certainly, relying on it doing so would seem to be problematic.  What does PERL return if stdout is redirected, and where does the output go in that case?

The io module does not use the c IO library, unlike the default open in python2 (the io module is the default in python3), so the behavior may also be different in the two cases.
msg229699 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2014-10-19 21:59
In Python 2, the closer for sys.stdout is _check_and_flush, which flushes the stream without closing the underlying FILE.

In Python 3, io.FileIO won't close the underlying file descriptor if closefd is False.

    >>> sys.stdout.buffer.raw.closefd

It isn't a writable attribute. It gets initialized to False by create_stdio (line 1034).

You'll can call os.close.

Python 2:

    >>> sys.stdout.close(); os.close(1); sys.stdout = open('CONOUT$', 'w', buffering=0)
    >>> sys.stdout.fileno()

Python 3:

    >>> sys.stdout.close(); os.close(1); sys.stdout = open('CONOUT$', 'w') 
    >>> sys.stdout.fileno()
msg229701 - (view) Author: Aaron Myles Landwehr (snaphat) Date: 2014-10-19 23:07
Yeah, it is windows specific. The problem is that if you open conout$ and the descriptor isn't 1, the buffer doesn't flush normally so the console won't display anything unless you manually flush it.

Now, why would you want to redirect STDOUT to the console in the first place like this? It's in the case where AllocConsole() has been called and since the console is new, STDOUT is directed elsewhere or nowhere. So you can't dup it or restore it or anything like that. In order, to get the normal behavior for the windows console the redirect is needed for STDOUT with fileno=1. But, as far as I can tell, it is impossible to actually do the redirect in python.

Anyway, let me discuss a bit the behavior between C, Python, and Perl w.r.t. to opening files. It'll help clarify where python is lacking in comparison to perl and c. You basically have two different 'flavors' of prototypes between the languages:
1) handle fd = open(filename, mode)
2) open(filename, mode, fd)

In C, the first prototype matches 'fopen' and is used to open a file. And the second prototype matches 'freopen' and is used to associate a different file with the an existing file descriptor. Moreover, you can't use stdout as an lvalue so you simply can't redefine it.

In code terms:
   stdout = fopen("CONOUT$", "w"); //Invalid lvalue
   FILE* file= fopen("CONOUT$", "w"); // Valid fileno=3 to N.
   freopen("CONOUT$", "w", stdout); // Valid fileno=1.
   freopen("CONOUT$", "w", file); //Valid IFF fd!=0, the value of fd is whatever it was previously.

In perl the open function is similar to the second prototype mentioned above, but the parameters are backwards. Moreover, It can either be used for new file descriptors or for reusing file descriptors unlike C's version of the second prototype.

In code terms:
   open (STDOUT, ">", "CONOUT$"); //Valid fileno=1. Reuses the desciptor number.
   open (FILE, ">", "CONOUT$"); //Valid fileno=3 to N. Reuses the fileno if a valid descriptor. Otherwise, creates a new one. 

In python, the function matches the first prototype with the exception that it can be used to redirect stdout unlike C's version.

In code terms:
   sys.stdout ="CONOUT$", "w"); // Valid fileno=3 to N.

So when you look at the behavior, you'll note that both C and Perl allow you to redirect using an existing descriptor; where as, Python seems to be unable to do this. Normally, it doesn't matter since you can just save stdout ahead of time, but it matters in the case where you don't start with a console (pythonw), or deallocated the console and allocated a new one later on.

As it stands, I don't believe this is fixable without an API extension/change of some sort that would allow passing filehandles... either that or... just making $CONOUT always allocate to 1 regardless. Though the latter would change the behavior for anyone who currently uses CONOUT$ in that it currently needs to be manually flushed to display to the console; where as, it wouldn't with the change.

Boy that is long winded, but hopefully it describes the issue adequately.
msg229703 - (view) Author: Aaron Myles Landwehr (snaphat) Date: 2014-10-19 23:12
Note, I just read eryksun's response. That does indeed fix the issues without API changes.
msg229707 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-10-20 01:37
OK, do we have a documentation issue here with respect to "actually" closing the standard streams, or do we just close this as not a bug?  Now that issue 17401 is closed, in python 3.5 you will at least be able to tell that the standard stream fdclose is set False by its repr.
msg229708 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-10-20 01:45
I think its worth a note that the stdio streams are constructed specially, even with the repr improvements.
msg233093 - (view) Author: Akira Li (akira) * Date: 2014-12-25 06:02
Two minor details:

1. It is possible that `fileno(stdout) != 1` even in C [1]. 
   I don't know what happens if the code from the answer is
   run on Windows.

   In principle, it may break eryksun's workaround. I don't 
   know how likely it is in practice.

2. you can redirect at the file descriptor level in Python [2] 
   using os.dup2(). I don't know whether the code in the 
   answer works on Windows.

Date User Action Args
2021-02-23 11:40:13eryksunsetcomponents: + IO
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.4, Python 3.5
2014-12-25 06:02:41akirasetnosy: + akira
messages: + msg233093
2014-12-21 01:50:28martin.pantersetnosy: + martin.panter
2014-10-20 14:26:21Arfreversetnosy: + Arfrever
2014-10-20 02:20:05r.david.murraysetnosy: + docs@python
title: Incorrect fileno for CONOUT$ / stdout -> document the special features (eg: fdclose=False) of the standard streams
assignee: docs@python
versions: + Python 3.4, Python 3.5
components: + Documentation, - IO
stage: needs patch
2014-10-20 01:45:46rbcollinssetnosy: + rbcollins
messages: + msg229708
2014-10-20 01:37:49r.david.murraysetmessages: + msg229707
2014-10-19 23:12:43snaphatsetmessages: + msg229703
2014-10-19 23:07:34snaphatsetmessages: + msg229701
2014-10-19 21:59:54eryksunsetnosy: + eryksun
messages: + msg229699
components: - Windows
2014-10-19 20:39:53r.david.murraysetnosy: + tim.golden, r.david.murray, zach.ware, steve.dower
messages: + msg229694
components: + Windows
2014-10-19 19:00:58snaphatcreate