classification
Title: distutils fails to locate vcvarsall with Visual C++ Compiler for Python
Type: enhancement Stage:
Components: Distutils Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Gregory.Szorc, Michael.Clerx, Nathan Jensen, dstufft, jason.coombs, merwok, naufraghi, piotr.dobrogost, steve.dower
Priority: normal Keywords: patch

Created on 2015-01-16 02:59 by Gregory.Szorc, last changed 2016-12-15 14:57 by BreamoreBoy. This issue is now closed.

Files
File name Uploaded Description Edit
vsforpython.diff naufraghi, 2016-12-15 13:42 review
Messages (21)
msg234110 - (view) Author: Gregory Szorc (Gregory.Szorc) Date: 2015-01-16 02:59
distutils as of Python 2.7.9 is unable to locate vcvarsall.bat if Visual C++ Compiler for Python is the only Visual Studio "distribution" installed.

STR:

1) Do a fresh install of Windows + all updates
2) Install Microsoft Visual C++ Compiler for Python from http://www.microsoft.com/en-us/download/details.aspx?id=44266
3) Enter a Visual C++ 2008 command prompt via start menu 
4) Attempt to run any |python.exe setup.py| that contains C extensions
5) "Unable to find vcvarsall.bat"

Examining the behavior of MSVC for Python's bat scripts and filesystem layout, it is different enough from "full" "distributions" that it confused distutils.

First, MSVC for Python doesn't appear to set any meaningful registry entries. So, the registry checking in msvc9compiler fails to find anything.

Second, the command environment for MSVC for Python doesn't export VS90COMNTOOLS, so msvc9compiler has no clue where to go looking for files.

Third, even if VS90COMNTOOLS is set, msvc9compiler isn't able to find vcvarsall.bat because it is installed in %installdir%/vcvarsall.bat and not %installdir%/VC/vcvarsall.bat, unlike every other Visual Studio AFAICT.

Another concern is that distutils.msvc9compiler.find_vcvarsall() first attempts to read from the registry, not the environment. If you are in a MSVC for Python command shell and you also have Visual Studio 2008 installed, distutils will use vcvarsall.bat from the full Visual Studio installation instead of the Python one. I think this is wrong.

The MSVC for Python command prompt does have an environment variable that can be used: VCINSTALLDIR. It is set to %installdir%\VC\, which is equivalent to ~/AppData/Local/Programs/Common/Microsoft/Visual C++ for Python/9.0/VC/ if you launch the installer with default options. distutils could be patched to find vcvarsall.bat in %VCINSTALLDIR%\..\vcvarsall.bat

Fortunately, a workaround is available:

1) Enter MSVC for Python command prompt
2) SET DISTUTILS_USE_SDK=1
3) SET MSSdk=1
4) python.exe setup.py ...
msg234111 - (view) Author: Gregory Szorc (Gregory.Szorc) Date: 2015-01-16 03:08
The first sentence in my original report is ambiguous. I tested with distutils on Python 2.7.9. But considering the code in question hasn't changed since 2011, I think it's safe to say this isn't a regression in CPython.
msg234129 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-16 13:42
Setuptools has the code to find the compiler package. We deliberately put it there instead of in distutils to make sure more people would get it.

I should probably port the extra check into 2.7.10, but the immediate fix is to import setuptools.
msg234137 - (view) Author: Gregory Szorc (Gregory.Szorc) Date: 2015-01-16 17:48
Thanks, Steve.

The package I was trying to build has its setup.py importing "setup" from distutils, not setuptools. I'll see about porting them to the future.

For posterity, https://bitbucket.org/pypa/setuptools/issue/258, which was first released in setuptools 6.0.
msg234140 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-01-16 18:20
FWIW, at some point I will need to do some serious work on this code for Python 3.5, so I'll certainly take your suggestions into account there. But I won't be doing the same for old versions of Python - at most 2.7 may get a check for the extra registry key, but 2.6 and 3.0-3.2 will still need to be using setuptools>=6.0 (which 2.7.10 will ship with).

Also, pip forces setuptools onto packages whether they like it or not, so pip installs should always build. Because it's a monkeypatch, "import setuptools" is sufficient to port old setup.py files.

All together, there doesn't seem to be an urgent need to get this into the core release. If someone comes up with a simple patch I'll happily review and apply it (and Jason will want a setuptools patch to skip the monkeypatch there), but I need to spend my own dev time on the next release.
msg237634 - (view) Author: Michael Clerx (Michael.Clerx) Date: 2015-03-09 10:52
Is there anyway to fix this without using setuptools? I'm using distutils to compile on the fly which has worked brilliantly for the last few years, not sure I want to destabilize everything by switching to setuptools.
msg237635 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-03-09 11:00
@Michael my very old work around here http://www.mail-archive.com/python-win32@python.org/msg06755.html may help.
msg237643 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-03-09 13:10
Setuptools just does a monkey patch of distutils, so importing it is sufficient. Or you can find the patching in the setuptools source and copy it into your setup.py file.
msg237647 - (view) Author: Michael Clerx (Michael.Clerx) Date: 2015-03-09 13:36
Thanks! However, importing setuptools causes a string of errors about PYTHONPATH and .pth files (even on a linux system)

In addition, a monkeypatched version of find_vcvarsall allowed things to run, but it seems finding plain C libraries with Visual C++ for python is very different from finding libraries in mingw

I think I'll just start telling my windows users to install MinGW instead
msg237655 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-03-09 15:01
@Michael in case you are not aware Python does not support MINGW.
msg237659 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-03-09 15:14
Python does not support being built with MinGW, but we have "support" to build extensions with MinGW. It isn't great support, and we're open to patches/contributions (none of our core Windows devs use MinGW AFAIK).

And yes, VC is different from MinGW because MinGW is trying to copy a different operating system. Where possible, Python generally tries to follow the conventions of the underlying operating system, which is why we build with VC and not gcc on Windows.
msg237660 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-03-09 15:15
You could also distribute wheels for your Windows users, which they will appreciate much more than being told to install MinGW. (If you do go the latter route, WinPython is a pretty good distribution that sets that stuff up fairly transparently.)
msg237661 - (view) Author: Michael Clerx (Michael.Clerx) Date: 2015-03-09 15:15
@Mark I don't quite understand what you're saying. Distutils supports it, provided you add a line to distutils.cfg. I've been using the PythonXY versions of Python with MinGW (everything before 2.7.9.0) to happily compile for nearly 4 years now.

Regarding your earlier suggestion about distutils and Visual C++ for Python, I hacked something together to do it dynamically in the script using distutils. Very ugly :-)

import platform
if platform.system() == 'Windows':
    vcpath = os.environ['ProgramFiles']
    vcpath = os.path.join(vcpath, 'Common Files', 'Microsoft',
        'Visual C++ for Python', '9.0', 'vcvarsall.bat')
    if os.path.isfile(vcpath):
        import distutils.msvc9compiler
        old_find = distutils.msvc9compiler.find_vcvarsall
        def new_find(version):
            path = old_find(version)
            if path is None:
                return vcpath
        distutils.msvc9compiler.find_vcvarsall = new_find
msg237662 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-03-09 15:18
> Very ugly :-)

And also incorrect on 64-bit interpreters (but not 32-bit interpreters on 64-bit machines, oddly enough), and machines where the user has installed the compiler just for themselves.

I'd seriously suggest copying the monkey patch from https://bitbucket.org/pypa/setuptools/src/18eceee63709bc70c06b51a609f3dbd2eab07ef4/setuptools/msvc9_support.py?at=default
msg237667 - (view) Author: Michael Clerx (Michael.Clerx) Date: 2015-03-09 15:52
Thanks!

If I do copy this patch, do I need to add anything to my license text other than a reference in the file itself?
msg237673 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-03-09 16:19
Hard to be sure. It was my contribution originally, and I'm okay to remain unattributed, but I've nosied Jason so he can answer authoritatively (especially since he rewrote most of my original contribution to make it work better :) ).
msg238068 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2015-03-14 01:55
The setuptools code is licensed under the ZPL and PSF licenses. If either of those licenses are insufficient, I'm happy to release my contribution in that file to the public domain.
msg245689 - (view) Author: Nathan Jensen (Nathan Jensen) Date: 2015-06-23 13:51
Is it possible to get the license issues resolved and get this fixed in 2.7.11 since that's the last 2.7 release that is currently scheduled?
msg245695 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-06-23 17:31
That's two separate questions:

> Is it possible to get the license issues resolved 

AFAICT they are resolved. Go ahead and copy the relevant code from setuptools.

> get this fixed in 2.7.11

Why do you need/want to build without setuptools?
msg245698 - (view) Author: Nathan Jensen (Nathan Jensen) Date: 2015-06-23 19:30
Ok, I was confused since the ticket was still marked as Open.  We are trying to make an open source CPython extension available for download and building and we don't want to add a dependency on setuptools.  We will patch our build commands based on the setuptools link provided in the message above.
msg283300 - (view) Author: Matteo Bertini (naufraghi) Date: 2016-12-15 13:42
I'm back on this issue with a minimal patch, and a longer motivation.

Distutils does not support Visual C++ for Python compiler, but it could, with a one-line patch.

The proposed workaround is to use `setuptools`. But, we are not alone in this World, am I supposed to fork each third party package still using `distutils` and upgrade them to `setuptools`?

No, someone will say, you can "simply" `import setuptools`, and it will monkey-patch `distutils` adding the support for VS for Python.

1) this is implicit, very very against the Python Zen
2) the modifications are not limited and back-compatible. Some packages, say `sip` from `PyQt`, are broken simply adding this `import setuptools`, and other are too.

That said, I think this minimal patch to the `find_vcvarsall` code, can save a lot of time to every Python 2.7 users, time I see better spent upgrading to Python 3.

Sorry for the rant-mode, but I very liked Python, and I still like Python more than other languages, but I don't think that providing half broken solutions and very limited support to all the developer still running Python 2.7 in some big old project is a good strategy to push people to Python 3.

Python 3 is already a better language, but Python as a language can be a better language only if Python 2.7 will be a first class citizen till 2020.
History
Date User Action Args
2016-12-15 14:57:52BreamoreBoysetnosy: - BreamoreBoy
2016-12-15 13:42:34naufraghisetfiles: + vsforpython.diff

nosy: + naufraghi
messages: + msg283300

keywords: + patch
type: enhancement
2015-06-23 20:06:11steve.dowersetstatus: open -> closed
resolution: not a bug
2015-06-23 19:30:12Nathan Jensensetmessages: + msg245698
2015-06-23 17:31:19steve.dowersetmessages: + msg245695
2015-06-23 13:51:48Nathan Jensensetnosy: + Nathan Jensen
messages: + msg245689
2015-04-06 20:30:15piotr.dobrogostsetnosy: + piotr.dobrogost
2015-03-14 01:55:54jason.coombssetmessages: + msg238068
2015-03-09 16:19:18steve.dowersetnosy: + jason.coombs
messages: + msg237673
2015-03-09 15:52:13Michael.Clerxsetmessages: + msg237667
2015-03-09 15:18:10steve.dowersetmessages: + msg237662
2015-03-09 15:15:34Michael.Clerxsetmessages: + msg237661
2015-03-09 15:15:28steve.dowersetmessages: + msg237660
2015-03-09 15:14:24steve.dowersetmessages: + msg237659
2015-03-09 15:01:37BreamoreBoysetmessages: + msg237655
2015-03-09 13:36:19Michael.Clerxsetmessages: + msg237647
2015-03-09 13:10:17steve.dowersetmessages: + msg237643
2015-03-09 11:00:43BreamoreBoysetnosy: + BreamoreBoy
messages: + msg237635
2015-03-09 10:52:07Michael.Clerxsetnosy: + Michael.Clerx
messages: + msg237634
2015-01-16 18:20:31steve.dowersetmessages: + msg234140
2015-01-16 17:48:19Gregory.Szorcsetmessages: + msg234137
2015-01-16 13:42:38steve.dowersetmessages: + msg234129
2015-01-16 04:44:04zach.waresetnosy: + steve.dower
2015-01-16 03:08:28Gregory.Szorcsetmessages: + msg234111
2015-01-16 02:59:53Gregory.Szorccreate