This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python 2.7.11rc1 not building with Visual Studio 2015
Type: behavior Stage:
Components: Documentation Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, kovidgoyal, paul.moore, r.david.murray, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2015-11-28 20:10 by kovidgoyal, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (14)
msg255550 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-28 20:10
The Pcbuild/readme.txt file implies that it is possible to build python 2.7.11rc1 with Visual Studio 2015 (although it is not officially supported). However, there are at least a couple of problems, that I have encountered so far:

1) timemodule.c uses timezone, tzname and daylight which are no longer defined in visual studio, as a quick hackish workaround, one can do
#if defined _MSC_VER && MSC_VER >= 1900
#define timezone _timezone
#define tzname _tzname
#define daylight _daylight
#endif

2) More serious, the code in posixmodule.c to check if file descriptors are valid no longer links, since it relies on an internal structure from microsoft ddls, __pioinfo that no longer exists. See
https://bugs.python.org/issue23524 for discussion about this in the python 3.x branch

As a quick and dirty fix one could just replace _PyVerify_fd with a stub implementation that does nothing for _MSC_VER >= 1900

However, a proper fix should probably be made.
msg255551 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-11-28 21:17
The proper fix is to remove any hint in the readme file that this will work. Building 2.7 with VS 2015 will not work unless someone forks CPython and makes the updates themselves.
msg255559 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-29 02:29
OK, I had hoped to avoid having to maintain my own fork of python 2 for a while longer, but, I guess not. 

Could you at least tell me if there are any other issues I should be aware of, to avoid me having to search through the python 3 sourcecode/commit history. 

I will be happy to make my work public so others can benefit from it as well.
msg255562 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-11-29 06:25
Not off the top of my head, but you only really need to look at my commits, and looking for issues assigned to me is an easy way to find those.

Most of the issues are obvious once you start trying to build.
msg255563 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-29 06:49
I have it building with just two simple patches:

https://github.com/kovidgoyal/cpython/commit/fd1ceca4f21135f12ceb72f37d4ac5ea1576594d

https://github.com/kovidgoyal/cpython/commit/edb740218c04b38aa0f385188103100a972d608c

However, in developing the patches, I discovered what looks like a bug in the CRT close() function. If you double close a valid file descriptor it crashes, rather than calling the invalid parameter handler.

python -c "import os; os.close(2); os.close(2)"

crashes. This is true for python 2.7.10 built against VS 2008 as well. This contrasts with the behavior of double close() on other operating systems, where it sets errno to EBADF and does not crash.

I have not tested it with python 3.5, but I assume the bug is present there as well.
msg255573 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-29 13:26
I missed a few places in my initial patch, updated patch:
https://github.com/kovidgoyal/cpython/commit/a9ec814d466d3c0139d10b69666f88eed10e4940

Also fixed the code not clearing errno before calling CRT functions, while I was there. Regardless of whether you want to allow your fork to be compiled with VS 2015 or not, I suggest you consider merging this patch, anyway, since the errno clearing is the correct thing to do, regardless. You can always cherrypick the errno clearing bits if you like :)

Just FYI, the code in my fork of 2.7 passes all tests on 64bit builds with VS 2015, except for 5 small ones that I have yet to track down. (test_ctypes test_distutils test_gzip test_mailbox test_zipfile)

I dont anticipate any difficulty in fixing the remaining test failures. Famous last words ;)
msg255588 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-11-29 20:07
You are aware that you can't use existing pre-compiled extension modules with your 2015 build, right?

It would be great if you could open a separate issue for the double close problem.  This should be a doc issue for fixing the docs.
msg255594 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-29 20:30
Yes, I am aware. I embed python in my application, which includes large C++ libraries. Those libraries are going to start requiring to be compiled with a modern compiler soon, which means I need python to also be compiled with a modern compiler. I already manually compile all python extensions in my build system, so that is not a problem. And before someone suggests I upgrade to python 3, porting half a million lines of python is simply not worth it for me. 

I'll be happy to open a separate bug report, but first I want some advice. I have got all the other tests passing as well, except one single test. test_gzip.test_many_append. 

The reason that test fails is apparently because of a buffering bug in the stdio C functions in VS 2015. Combining lots of seeks relative to SEEK_CUR causes read() to return incorrect data. I can make the test pass by modify the gzip module to open files with bufferring=0, or by putting in a seek(0, 0) to cause the stdio layer to flush its read buffer at the appropriate point. However, this is not an actual fix, just an inefficient workaround.

My question is, how do I properly workaround this bug? And how come this bug is not triggered in Python 3.5.0? Am I diagnosing this correctly? Any other alternative explanations?
msg255607 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-30 04:32
To answer part of my question, the reason the fseek()+fread() bug does not affect python 3.5.0 appears to be because it implements its own buffering and does not use fseek()/fread() at all. 

Sigh, I really hope the answer does not end up being that I have to re-implement fseek()/ftell()/fread()/fwrite() using lseek()/read()/write() on windows. Or I could wait and hope Microsoft fixes the bug :)

As a first step, to confirm that the bug is in the CRT, I'll have the gzip module record all reads/seeks/tells and then see if I can reproduce the bug in a plain C program.
msg255611 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-30 06:10
Doesn't seem like a bug in the CRT, I cannot reproduce in a plain CRT program, so now I get to try to figure out what is broken in fileobject.c by VS 2015. That's a relief :)
msg255615 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-30 10:24
I take it back, my methodology in reproducing the function calls used by the gzip module was flawed. 

It does look like a bug in the CRT, but I have not been able to isolate a simple way of reproducing it. I have however, found a workaround for it, that has an acceptable performance impact. 

https://github.com/kovidgoyal/cpython/commit/72ae720ab057b1ac0402d67a7195d575d34afbbd

Now all tests pass (except for tcl/tk and distutils, neither of which I care about -- well I will probably need to fix up distutils at some point, but not now :). Running testsuite as

./PCbuild/amd64/python_d.exe Lib/test/regrtest.py -u network,cpu,subprocess,urlfetch

@steve: Thank you for all the work you did porting python 3.x to VS 2015, that certainly made by life a lot easier.

I would of course, be ecstatic if you were to consider merging my work into the python 2.7 branch, but if not, I understand -- no one likes to maintain a legacy codebase.

In any case, for interested third parties, my work is available here:

https://github.com/kovidgoyal/cpython (2.7 branch)

and instructions on building python on windows using a nice cygwin environment are here: 

https://github.com/kovidgoyal/calibre/blob/master/setup/installer/windows/notes2.rst
msg255620 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-11-30 14:00
Unfortunately it is unlikely we'll merge it, since because of the compiled extension issue we have a negative motivation for supporting anything other than the release compiler currently used.
msg255621 - (view) Author: Kovid Goyal (kovidgoyal) Date: 2015-11-30 15:10
No worries, as I said, I understand, I would probably do the same, were I in your shoes. I have found that being a maintainer of a complex software project tends to naturally increase conservatism :)
msg258427 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2016-01-16 22:01
Checked PCBuild/readme.txt and it's pretty clear that you can only use VS 2010+ with the SDK and the old compiler. So I'm closing this as wontfix as we will not be merging support to build Python 2 with VC10 or later.
History
Date User Action Args
2022-04-11 14:58:24adminsetgithub: 69945
2016-01-16 22:01:44steve.dowersetstatus: open -> closed
resolution: wont fix
messages: + msg258427

stage: needs patch ->
2015-11-30 15:10:49kovidgoyalsetmessages: + msg255621
2015-11-30 14:00:45r.david.murraysetmessages: + msg255620
2015-11-30 10:24:15kovidgoyalsetmessages: + msg255615
2015-11-30 06:10:42kovidgoyalsetmessages: + msg255611
2015-11-30 04:32:51kovidgoyalsetmessages: + msg255607
2015-11-29 20:30:32kovidgoyalsetmessages: + msg255594
2015-11-29 20:07:21r.david.murraysetassignee: docs@python
type: compile error -> behavior
components: + Documentation, - Interpreter Core

nosy: + r.david.murray, docs@python
messages: + msg255588
stage: needs patch
2015-11-29 13:26:31kovidgoyalsetmessages: + msg255573
2015-11-29 06:49:52kovidgoyalsetmessages: + msg255563
components: - Build, Windows
2015-11-29 06:25:42steve.dowersetmessages: + msg255562
2015-11-29 02:29:56kovidgoyalsetmessages: + msg255559
2015-11-28 21:17:34steve.dowersetmessages: + msg255551
2015-11-28 20:16:47SilentGhostsetnosy: + paul.moore, tim.golden, steve.dower, zach.ware
components: + Build, Windows
2015-11-28 20:10:41kovidgoyalcreate