In pythonrun.c, there is function initstdio() with the following test:

static int
initstdio(void)
{
...
    /* Set sys.stdin */
    fd = fileno(stdin);
    /* Under some conditions stdin, stdout and stderr may not be connected
     * and fileno() may point to an invalid file descriptor. For example
     * GUI apps don't have valid standard streams by default.
     */
    if (fd < 0) {
#ifdef MS_WINDOWS
        std = Py_None;
        Py_INCREF(std);
#else
        goto error;
#endif
    }
    else {
...
}

This function is fails for non-console applications (i.e. MFC) built using Visual C++ 11.0 (Visual Studio 2012), becasue **strangely**, fileno(stdin) == 0, so this test results in false and Python initialisation routines attempt to setup streams.

Apparently, fileno(stdin) return value has changed between Visual C++ 10.0 (VS 2010)
and 11.0. The VC++ 10.0 reports fileno(stdin) == -2.

And does it cause an issue later? How?

Yes, it does. In file Modulfileio.c, in function fileio_init, there is this code:

    if (fd >= 0) {
        if (check_fd(fd))
            goto error;
        self->fd = fd;
        self->closefd = closefd;
    }

The check_fd tests:

if (!_PyVerify_fd(fd) || (fstat(fd, &buf) < 0 && errno == EBADF)) {

The _PyVerify_fd(fd) == 1, but errno is "Bad file descriptor".

This eventually leads to Py_InitializeEx failure at:

    if (initstdio() < 0)
        Py_FatalError(
            "Py_Initialize: can't initialize sys standard streams");

> In file Modulfileio.c,

I messed the path and filename above I meant: In file Modules/_io/fileio.c,

Maybe check_fd(fd) could be used in initstdio as well.

Can you check whether it's the same for the other files?
What are the values for fileno(stdout) and fileno(stderr)?

Replacing if the current test in Python 3.2

if (fd < 0)
with
if (check_fd(fd) < 0)

Seems to be a working solution.

I just noticed, that in current Python/pythonrun.c in the repo, there the fd < 0 tests have been replaced with new function is_valid_fd(). But, its semantic is equivalent to fd < 0, so it does not check anything really. Perhaps is_valid_fd could be redefined as check_fd.

Here are Visual C++ 11.0 (1700) vs 10.0 differences of fileno return value for all the standard streams

    int fdi, fdo, fde;
    fdi = fileno(stdin);
    fdo = fileno(stdout);
    fde = fileno(stderr);
#if _MSC_VER < 1700
    assert(fdi == -2);
    assert(fdo == -2);
    assert(fde == -2);
#else
    assert(fdi == 0);
    assert(fdo == 1);
    assert(fde == 2);
#endif

By the way, I assume such sudden change in fileno(std*) behaviour between Visual C++ versions is suspicious, so I also submitted bug report to Visual Studio:
https://connect.microsoft.com/VisualStudio/feedback/details/785119/

I've just got an update on the bug report [1] I submitted to Microsoft.
The Visual C++ team confirmed "It does appear to be a regression from Visual Studio 2010."

So, it's not a bug in Python, but I think it may be important for Python to consider applying some reliable workaround (i.e. use of check_fd() function)

[1] http://connect.microsoft.com/VisualStudio/feedback/details/785119/

Tested on MSVCRT110.DLL (11.0.51106.1 released in 2012/11/05). 
Previous one 11.0.50727.1 released in 2012/07/26 does not have this problem.
This problem better be fixed in Python for MS does not have clear document.

Python 3.2 is in security fix mode. Bug fixes such as this issue don't count as security fix. Are you able to update to Python 3.3? Python 2.7 has a different implementation and uses FILE* instead of file descriptors.

Hi Christian,

The latest runtime Microsoft provided is buggy.
Tried fix the PyVerify_fd with more GetFileType verification. But a lot more problems came for isatty returns true in non-console program. The fix in Python side shall be large.
Details is reported to Microsoft, we can only hope they make it right.

This is still an issue in VS 2012 Version 11.0.60610.01 Update 3

After contacting Microsoft they answered that they cannot fix that within the VS2012 or VS2013 cycle. Very bad. Any ideas?

Here are some solutions which might help you until MS fixed the bug.

1) Even if the subsystem is Windows one solution is to call AllocConsole() before Py_Initialize() to create a console.

2) Second Solution: If you don't want to open a console and your application has its own output window you could redirect stdout/stderr/stdin with freopen to a temp file first. (e.g: freopen("file.txt","w",stdout);) Call freopen for all std handles before you call Py_Initialize(). You can keep the redirection to the file or after Py_Initialize() succeeded you change the redirection now in Python by redirect sys.stdout to the place you want.

Which 3.2 version does this break with? As far as I can tell, it should be fixed in 3.2.3: see changeset f15943505db0, issue #7111.

@Antoine (Re msg197480)

I'm not sure which minor version of Python 3.2 it was, but in my comment in msg187362, I confirmed that I have tested the later version with added is_valid_fd function. As far as I understand, the changes in http://hg.python.org/cpython/rev/f15943505db0/ introduce that is_valid_fd function, which does not seem to solve the problem.

Sorry, I used the latest Python3.4 branch, I haven't tested this with prior versions.

> I'm not sure which minor version of Python 3.2 it was, but in my
> comment in msg187362, I confirmed that I have tested the later
> version with added is_valid_fd function. As far as I understand, the
> changes in http://hg.python.org/cpython/rev/f15943505db0/ introduce
> that is_valid_fd function, which does not seem to solve the problem.

Could you try to investigate this a bit further? Does is_valid_fd()
return true, and why?

On 11 September 2013 12:34, Antoine Pitrou <report@bugs.python.org> wrote:
>> I'm not sure which minor version of Python 3.2 it was, but in my
>> comment in msg187362, I confirmed that I have tested the later
>> version with added is_valid_fd function. As far as I understand, the
>> changes in http://hg.python.org/cpython/rev/f15943505db0/ introduce
>> that is_valid_fd function, which does not seem to solve the problem.
>
> Could you try to investigate this a bit further? Does is_valid_fd()
> return true, and why?

I have already checked that (see msg187362)

Shortly, is_valid_fd always returns true because fd < 0 is always
false as my tests with Visual Studio 2012 (Visual C++ 11.0 (1700)
confirmed (see also bug report linked in msg188117)

    assert(fdi == 0);
    assert(fdo == 1);
    assert(fde == 2);

IOW, fd is always larger than 0.

> Shortly, is_valid_fd always returns true because fd < 0 is always
> false as my tests with Visual Studio 2012

Well, that's not all there is, is_valid_fd() does other checks before returning true.

On 12 September 2013 00:06, Antoine Pitrou <report@bugs.python.org> wrote:
>> Shortly, is_valid_fd always returns true because fd < 0 is always
>> false as my tests with Visual Studio 2012
>
> Well, that's not all there is, is_valid_fd() does other checks before returning true.

Given the function:

is_valid_fd(int fd)
{
    int dummy_fd;
    if (fd < 0 || !_PyVerify_fd(fd))
        return 0;
    dummy_fd = dup(fd);
    if (dummy_fd < 0)
        return 0;
    close(dummy_fd);
    return 1;
}

for fd values of 0, 1 or 2

1. fd < 0 is always false
2. _PyVerify_fd(fd) is always true. Given the current definition:
#define _PyVerify_fd(fd) (_get_osfhandle(fd) >= 0)
for those values of fd _get_osfhandle(fd) >= 0, always.
3. for those fd values, dup() never returns fd < 0

> 2. _PyVerify_fd(fd) is always true. Given the current definition:
> #define _PyVerify_fd(fd) (_get_osfhandle(fd) >= 0)
> for those values of fd _get_osfhandle(fd) >= 0, always.

Hum, are you sure this is the selected implementation?
- this code is only in 2.7
  http://hg.python.org/cpython/file/v2.7/Include/fileobject.h#l73
- it's protected by #if statements. In your case the first case should be selected, can you verify if it's the case, and if not, why?

I have just tested Windows GUI application built against Python 3.2.1 with is_valid_fd patch according to http://hg.python.org/cpython/rev/f15943505db0/

All built using VS2012.

Again, I can confirm that is_valid_fd does NOT solve the problem.
Here is extract of execution flow in initstdio function:

1. fd = 0, despite it is GUI app, see https://connect.microsoft.com/VisualStudio/feedback/details/785119/

fd = fileno(stdin);

2. is_valid_fd will return __true__, so it moves to calling create_stdio()

if (!is_valid_fd(fd)) {
   ...
}
else {
   std = create_stdio(iomod, fd, 0, "<stdin>", encoding, errors);
   if (std == NULL)
       goto error;
}

3. The create_stdio() call fails though, causing error followed by abort


Still, the only solution that solves this problem in Windows GUI applications buitl using VS2012 is to use check_fd() function to check fd, instead of is_valid_fd(). The check_fd() is more reliable as it will return -1 due to errno == EBADF.

My previous attempt to analyse _PyVerify_fd() is not relevant for the problem, let's forget it.

Hi all,

I ran into this problem myself today with 3.3.2 and I thought I'd provide some more detail on what's going on. As has been described, prior to VC11, fileno(stdin) returned -2 in applications with no console, so is_valid_fd() was properly rejecting it and the standard streams were being set to None. With VC11, it now returns 0, and the NT handle that underlies the FILE object associated with file descriptor 0 is -2. The problem is that -2 is a pseudo-handle representing the current thread. dup() is simply a wrapper around the Windows API function DuplicateHandle(), so when dup() is called on file descriptor 0, it returns success because DuplicateHandle() is successfully creating a new handle referring to the current thread. It's not until the call to fstat() in check_fd() much later that the CRT realizes it's not dealing with a file.

I see in Mercurial that 3.3.4 and 3.4 that the is_valid_fd() code hasn't been changed. This is definitely a Microsoft bug, but if (as it sounds) they're not going to be able to fix it anytime soon, I think it would be  ideal to have a change for this in Python. Replacing this with check_fd() would fix it, or if you don't want to rely on fstat(), some Windows-specific code using GetStdHandle():

#ifdef MS_WINDOWS
	if (!is_valid_fd(fd) || GetStdHandle(STD_INPUT_HANDLE) == NULL) {
#else
    if (!is_valid_fd(fd)) {
#endif

That would have to be repeated for stdout and stderr using GetStdHandle(STD_OUTPUT_HANDLE) and GetStdHandle(STD_ERROR_HANDLE), as those descriptors have the same problem.

FYI, I've got it confirmed fix for the bug in VS has been released [1]

"""
We have fixed this behavior for the next major release, Visual Studio "14." The fix is present in the Visual Studio "14" CTP that was released earlier this month.
"""

[1] https://connect.microsoft.com/VisualStudio/feedback/details/785119/fileno-stdin-0-for-non-console-application#tabs

It seems that bug remains *not* fixed.

Just like eaducac pointed out, there are cases when underlying file handle is -2, which is not a valid file handle, but *is* a valid handle (which is GetCurrentThread).

Thus, provided dup solution accepts it, while it shouldn't. However, suggested GetStdHandle check still doesn't work for me. While working on the related Spoon issue, I've worked out a working solution (see the patch attached).

Tomek,
Spoon.net dev

@steve.dower: Do you have info on this issue?

This is fixed in VS2015 last time I tried it. I've proposed workarounds for this before, but it's never affected any official build do there's never been any traction. The fstat patch is normally the check that I use.

I don't know what Tomasz means that it's not fixed. Fixes like this don't magically appear in earlier VS versions, and while VS2015 does have the fix, it's not quite ready for production work.

I don't mean fixing VS, but providing a workaround in Python code, when compiled with VS2012.

I know newer VS versions are fixed. Do you mean, VS2012 is not supported? If it's not - then LibreOffice team have a problem, because their official release is built with this suite.

If it is supported, and you don't want to alter behavior for other versions, then a patch may also contain an ifdef for MSVC 11.0.

It's not supported for building Python, is what I meant. I don't know what the status is for Microsoft support of VS2012, but I'd expect it's security fix only by now.

Being able to support building Python with alternative compilers is a task that we don't have the volunteers to support, especially on Windows. That said, I'm not against taking patches like this to improve it, but I'd expect they'll be treated as an enhancement and won't make it into any version before 3.4.

An enhancement can only go into 3.5, not 3.4.  This has to happen before beta 1 which is currently scheduled for May 24 2015.  I'm against the idea as it would create too many compatibility issues.

In such case, I agree it's an "enhancement", not a "bugfix". Could go into 3.5 branch.

How about changing the ifdef to the following, to avoid any issues on non-MSVS2012:

#if defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(_MSC_VER) && _MSC_VER == 1700

Hi all, a small update to confirm this issue with version 3.5.0a2 embedded in a native C++ GUI application, having everything built with VS2013 SP4.

Same execution flow as described by Mateusz in msg198689.

I've modified Python/pylifecycle.c file according to Tomasz's attached patch.
It just "works for me" and I've just slightly modified the #ifdef line:

#if defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(_MSC_VER) && _MSC_VER >= 1700 && _MSC_VER < 2000

It would definitely be nice to have this workaround applied to the 3.5 branch since MS' fix is lost in release cycle. Would it be too intrusive?

Oops, mistyping, #ifdef test would be:

#if defined(MS_WINDOWS) && defined(HAVE_FSTAT) && defined(_MSC_VER) && _MSC_VER >= 1700 && _MSC_VER < 1900

There's not much point putting this into 3.5 anymore, as we are going to break pre-VS2015 builds completely sooner or later (too many changes to the CRT that we can't just work around with #ifdefs - and too much opposition to having code purely to support unsupported compilers).

It's probably only worth putting this into 3.4 at this stage, which I'm okay with but I'm not going to push hard to make it happen. Also worth noting that the code here in 3.5 has diverged from 3.4, and so the attached patch doesn't apply.

Any of the other core devs strongly opposed to me updating 3.4 so people embedding that version can have the fix?

Well, just in case, I've attached the patch to apply against 3.5.0a2.

Re msg238016, I confirm python-3.5.0a2-fdvalidation.patch fixes the problem for Python 3.5.0a4 and VS2013.

The only issue I encountered was with HAVE_FSTAT which is missing from PC/pyconifg.h, so I edited the patch and removed the check if defined(HAVE_FSTAT). Does PC/pyconifg.h need update?

I also confirm vanilla Python 3.5.0a4 with VS2015RC does not require this patch, because the fileno regression/bug has been fixed in VS2015RC (details at https://connect.microsoft.com/VisualStudio/feedback/details/785119/).

I see issue to be fixed but patch wasn't applied yet. Is it still supposed to be included in 3.5 or there is something wrong?

Rereading the discussion, there seems to be agreement that this is an enhancement. It does not apply for Python 3.5 (which requires a compiler without the bug that causes this), and Python 3.4 is no longer eligible for enhancements. I'm closing this as out of date.

The patches will be here, so anyone who is rebuilding Python 3.4 or earlier for their own use with VS 2012 is welcome to apply the patch themselves.

Steve,

What's going to be the required msvc compiler for 3.5 on Windows?

The 3.5 build uses MSVC 14 (VS 2015):

https://docs.python.org/3.5/using/windows.html#compiling-python-on-windows
https://hg.python.org/cpython/file/3.5/PCbuild/readme.txt

In case someone else need it, here is a patch for python 3.5.0b1