Win XP 64-bit
Python 2.6.4 64-bit

Getting distribution for 'pycrypto>=1.9'.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 1714, in main

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 1695, in with_ei_usage

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 1718, in <lambda>

  File "C:\Python26\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 236, in run

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 452, in easy_install

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 501, in install_item

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 680, in install_eggs

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 957, in build_and_install

  File "build\bdist.win-amd64\egg\setuptools\command\easy_install.py", 
line 946, in run_setup

  File "build\bdist.win-amd64\egg\setuptools\sandbox.py", line 29, in 
run_setup
  File "build\bdist.win-amd64\egg\setuptools\sandbox.py", line 70, in 
run
  File "build\bdist.win-amd64\egg\setuptools\sandbox.py", line 31, in 
<lambda>
  File "setup.py", line 163, in <module>

  File "C:\Python26\lib\distutils\core.py", line 152, in setup
    dist.run_commands()
  File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "build\bdist.win-amd64\egg\setuptools\command\bdist_egg.py", line 
174, in run
  File "build\bdist.win-amd64\egg\setuptools\command\bdist_egg.py", line 
161, in call_command
  File "C:\Python26\lib\distutils\cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "build\bdist.win-amd64\egg\setuptools\command\install_lib.py", 
line 20, in run
  File "C:\Python26\lib\distutils\command\install_lib.py", line 113, in 
build
    self.run_command('build_ext')
  File "C:\Python26\lib\distutils\cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "C:\Python26\lib\distutils\dist.py", line 995, in run_command
    cmd_obj.run()
  File "C:\Python26\lib\distutils\command\build_ext.py", line 340, in 
run
    self.build_extensions()
  File "setup.py", line 116, in build_extensions

  File "C:\Python26\lib\distutils\command\build_ext.py", line 449, in 
build_extensions
    self.build_extension(ext)
  File "C:\Python26\lib\distutils\command\build_ext.py", line 499, in 
build_extension
    depends=ext.depends)
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 448, in 
compile
    self.initialize()
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 358, in 
initialize
    vc_env = query_vcvarsall(VERSION, plat_spec)
  File "C:\Python26\lib\distutils\msvc9compiler.py", line 274, in 
query_vcvarsall
    raise ValueError(str(list(result.keys())))
ValueError: [u'path']
An error occured when trying to install pycrypto 2.0.1. Look above this 
message for any errors that were output by easy_
install.
While:
  Installing python.
  Getting distribution for 'pycrypto>=1.9'.
Error: Couldn't install: pycrypto 2.0.1

Most likely, the problem is that visual studio is not installed on your
system.

I have Visual C++ 2008 express edition installed.

Maybe it is a quirk of express edition as I was able to install pycrypto 
fine with the Visual Studio 2008 professional edition at work machine.

In any case, I believe this is a bug with distutils.

I have the same issue with the Express edition. You can work around it
by finding and executing vcvars32.bat or vcvars64.bat before running
setup.py. It would be nice if distutils took care of it though.

That's what distutils does : it runs vcvarsall.bat and pass as the
argument the architecture.

Can you run:

$ vcvarsall.bat arch

(replace arch by x86 or x64)

and paste me the result ?  (it looks like the 'path' variable misses
from this file.

I think we have two issues here:

First, the default install of VS Express does not support 64-bit tools,
so distutils cannot work with a 64-bit Python install. I verified that
it _does_ work with a 32-bit Python install.


Second, it is possible to install 64-bit tools from the SDK
(See:http://www.mathworks.com/support/solutions/en/data/1-6IJJ3L/index.html?solution=1-6IJJ3L),
but then some paths are broken in the vcvarsall.bat script:

The script is looking for bin\amd64\vcvarsamd64.bat, the actual path is
bin\vcvarsamd64.bat. The attached diff solves the problem for me, but
someone should verify that the paths are indeed broken when doing the
SDK install as described above.

I needed a new Windows VM image anyway, so I can now confirm that the
paths of a fresh VS Express + SDK 64-bit tools install are broken as
described above.

OK so bin\vcvars64.bat would be an alternative location to look at for
win32-64.

Your diff says "amd64", shouldn't it be testing for "x86_64", "amd64"
and "x64" ?

Ok, I created a new diff that considers all 64-bit values for 'arch'.
I think x64 cannot occur (See: PLAT_TO_VCVARS). Also, I use the
existence of bin\amd64\vcvarsamd64.bat as a test for VS Professional,
so in that case the new code is never executed.

Could someone confirm that VS Professional always has that file?

'ValueError' should be 'DistutilsPlatformError' in the diff.

I'm running into the same issue myself. I even installed Visual Studio 2008. Issue is it installed as a 32-bit application in the x86 directory, not sure why. I have 64-bit Windows 7. I'm installing the SDK to see if that helps.

Also I need this for pycrypto too. If anyone has a 64-bit AMD build for the latest crypto that'd be amazing.

Tarek, when you set the resolution to 'accepted', did you mean to close this? Resolutions are meant for closed issues.

Some committers have been using 'resolution: accepted' as a way to mark issues that are definitely going to be fixed/enhancement added.  For a critical bug I think this is a bit redundant (it is more useful for feature requests), but I'm pretty sure Tarek did it on purpose.

Yes I used 'accepted' to mark this issue as something I am working on.

I am confused about the resolution flag, I think it is a bit overlapping with the status flag, and overcomplex. 

Maybe we should replace "closed" with various flags like: "fixed", "wontfix",  "rejected". (My 2 cents of course ;) )

Remarks with the patch:

- we need to keep the find_vcvarsall() backward compatible. e.g. define a default value for arch.

- Next, since this fix is specific to 64bt and since *any value* in arch will be used as "not 64 bits", I think the patch should be using a boolean tag instead (vs64bits=False).

- Last, we need a unit test that runs exclusively in this environment, to demonstrate that youyr fix fixes the error.

Let me know if you want to work on the patch, or I can fix it directly.
I'd prefer of course if you modify the patch ;)

I agree with all points. Could you fix it directly if at all possible?
Of course eventually I'd do it, it might take some time until I get
around to it though.

Closed issue 9224 as a duplicate of this one;  merging nosy lists.

I've been looking at this again because of new duplicate reports.


I think we need to distinguish between amd64 and x86_amd64/x86_ia64.
But using arch="x86" as the default parameter would do fine.


Re unit test:

The test will only work if 64-bit tools from the SDK are present, and
I'm not sure how to test for that other than writing an independent
find_vcvarsall2 function that checks on the first one.

I have to say that I'm using Linux 99.9% of the time, so perhaps
a Windows expert could suggest a registry based solution.

New version of the patch with a default value for arch.

May I request we resolve this by folding this patch into the main source tree for the next bugfix release?

ipatrol: have you actually tested the patch? If so, which specific version, and on what specific Python version?

Actually, I forgot the 64 bit directions X-? It will take some time for everything to download. But I did apply the patch and it did give me the "64 bit SDK not installed" error. Also, I think the tow ValueErrors should be changed to DistutilsPlatformError and DistutilsExecError respectively.

ipatrol, thanks, I forgot to change ValueError. New patch attached.

Aren't there two ValueErrors in the file? One is a platform and (in the now-fixed version) the other would indicate a problem with I/O or vcvars execution (like permissions).

I noticed that the directions from mathworks are no longer correct. The only optional feature is a 64 bit SQL server. Yet I still can't get it to work correctly. Does MS no longer provide the compiler in their free editions?

HAHA DISREGARD THAT, WRONG DOWNLOAD. However, the C++ compiler `is` now installed by default. P.S: that second ValueError?

Stefan (sorry for the misattribution in my previous message), can you test with 3.2 and 3.1 and adjust versions if needed?

> Stefan (sorry for the misattribution in my previous message), can you test with 3.2 and 3.1 and adjust versions if needed?

Not easily at the moment. I just noticed that I replaced my only
Express installation with a Pro trial-edition.

Anyone else?

I came across this issue while trying to install mercurial using easy_install. I applied the vcvars4.diff to my distutils and this solved the problem.


Environment:

Python 2.6.6 (64 bit)

setuptools-0.6c11.tar.gz
installed using the following instructions:
http://stackoverflow.com/questions/309412/how-to-setup-setuptools-for-python-2-6-on-windows

pywin32-214.win-amd64-py2.6 (installed binaries)
py2exe-0.6.9.win64-py2.6.amd64 (installed binaries)


Windows 7 (64 bit)
Visual Studio Team System 2008 (installed everything)
Visual Studio 2008 SDK 1.1 (installed everything)

I can test this for 3.1, as all I have is the Express version of VC++. If you could point me towards a library that will work with 3.1 and has C components, that'll make the testing a lot easier. pycrypto and setuptools, the two libs mentioned in this issue, do not support Python 3.x.

Before anyone does any further testing:

Tarek, can this go into distutils 2.7/3.2 (after 3.2 has been released)?
Due to the popularity of this issue I think it might be worthwhile to
make an exception and ignore the distutils freeze.

IIUC, the issue is that people installing a 64-bit Python, and VS Express, and then wonder why they can't build extension modules. I'm not so sure that there is a bug in Python here - this setup is not supported (and that's really Microsoft's fault). Now, automatically finding the SDK would be a new feature, IMO: if you want to use SDK tools, you are supposed to set DISTUTILS_USE_SDK, after opening the respective build environment. IIUC, installing VS express would not have been necessary in this setup at all - just install the SDK.

Alternatively, people can install Visual Studio proper, or use a 32-bit Python.

You are right, this is not a bug in Python. The diff provides a workaround for a limitation in VC++ 2008 Express. This diff is a piece of user service.

An equally as workable workaround is for the user to copy VC\bin\vcvars64.bat into VC\bin\amd64\vcvarsamd64.bat. Once that is done, distutils works without the workaround provided in this diff.

Whether to provide a workaround within Python for an issue in VC++ 2008 Express can certainly be debated. I for one suggest to err on the side of being friendly, rather than insisting on purity.

Consider that users of Python do include hobbyists and students. For them, VC++ Pro may be out of reach. And there are plenty of reasons for building 64-bit binaries, among them learning exercises.

Purity shmurity. The point of distutils is largely to present a unified and simple interface. 'python setup.py install' should be all a user has to do on any platform. Unless you can come up with a better idea, MSVC is really the only big compiler on Windows.

Confirmed that this issue exists on Python 3.1 and 3.2b2. The exception thrown presents as:

ValueError: ['path', 'include', 'lib']

Going through more than a few registry keys, here are some that stood out for me:

"HKEY_CURRENT_USER\Software\Microsoft\Microsoft SDKs\Windows\v6.0A" InstallationFolder

"HKEY_CURRENT_USER\Software\Microsoft\Microsoft SDKs\Windows\v6.1" InstallationFolder

"HKEY_CURRENT_USER\Software\Microsoft\VCExpress\9.0\External Tools" ToolDir1

"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.0A" InstallationFolder

"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v6.1" InstallationFolder

"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\SxS\VC7" 9.0

"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Microsoft SDKs\Windows" CurrentInstallFolder

"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\9.0\Setup\VC" ProductDir

"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudio\SxS\VC7" 9.0


Maybe they can be read to tease out the install dirs?

regarding "hey this is an MS bug not Python", projects which feature optional C extensions are starting to apply workarounds for the issue on their end (I will need to commit a specific catch for this to SQLAlchemy) - users need to install our software and we need to detect compilation failures as a sign to move on without it.   I think it's preferable for Python distutils to work around an MS issue rather than N projects having to work around an MS issue exposed through distutils.  Seems like this bug has been out there a real long time...bump ?

Thorsten Behrens said:
> You are right, this is not a bug in Python. The diff provides a
> workaround for a limitation in VC++ 2008 Express. This diff is a
> piece of user service.

ipatrol added:
> Purity shmurity. The point of distutils is largely to present a
> unified and simple interface. 'python setup.py install' should be all
> a user has to do on any platform. Unless you can come up with a
> better idea, MSVC is really the only big compiler on Windows.

The feature freeze is distutils is not a conspiracy plotted by developers loving purity; it is a necessity caused by hard pragmatism.  Tarek would have loved to continue improving distutils instead of forking distutils2, but he could not because third-party code relying on bugs and workarounds was broken.  We have to be cautious, and it has nothing to do with purity.  Believe me, the freeze, fork and merge was not something fun for distutils developers, but we had to do it precisely for our users.

While I agree that in an ideal world, “'python setup.py install' should be all a user has to do on any platform”, it cannot be done in distutils.  Like Martin said:
> if you want to use SDK tools, you are supposed to set
> DISTUTILS_USE_SDK, after opening the respective build environment

For distutils2/packaging however, we can work on improving the compiler system, so that “pysetup run install” is all a user had to do on any platform.  See #12257 for that.

Hmm, in http://bugs.python.org/issue7511#msg106420 Tarek appeared to
be supportive of the patch.


Re DISTUTILS_USE_SDK:

I don't think many users are aware of this variable. Also, it is not
needed at all; it is sufficient to execute vcvars64.bat manually
before running setup.py.

mike bayer wrote:
> > 
> > mike bayer <mike_mp@zzzcomputing.com> added the comment:
> > 
> > regarding "hey this is an MS bug not Python", projects which feature optional C extensions are starting to apply workarounds for the issue on their end (I will need to commit a specific catch for this to SQLAlchemy) - users need to install our software and we need to detect compilation failures as a sign to move on without it.   I think it's preferable for Python distutils to work around an MS issue rather than N projects having to work around an MS issue exposed through distutils.  Seems like this bug has been out there a real long time...bump ?
This is not really an MS issue. Setting up the environment to be
able to compile extensions is a prerequisite on most platforms and
with most compilers.

MS VC++ supports having multiple compiler versions on the same
machine and allow compiling to x86, x64 and ia64 (at least
in more recent VC++ versions).

I think it's fair to ask the user to setup the environment correctly
before running "python setup.py install", since distutils doesn't
really know which compiler to use - e.g. you could be cross-compiling
for x64 on an x86 machine, or you may want to use VC 2008 instead of
a more recently installed VC 2010.

Wouldn't it be better to have distutils tell the user about the
possible options, instead of guessing and then possibly compiling
extensions which later on don't import or import, but don't work
as expected ?

Regarding the latest patch: This is not the right approach, since
find_vcvarsall() is supposed to return the path to the vcvarsall.bat
file and not an architecture specific setup file. It is later
called with the arch identifier, which the arch specific setup files
don't check or use.

Also note that vcvarsall.bat can take these options:

   x86 (default), x64, amd64, x86_amd64, ia64, x86_ia64

The x86_* options setup the cross compilers.

Marc-Andre Lemburg <report@bugs.python.org> wrote:
> Wouldn't it be better to have distutils tell the user about the
> possible options, instead of guessing and then possibly compiling
> extensions which later on don't import or import, but don't work
> as expected ?

That would be an option, yes.


> Regarding the latest patch: This is not the right approach, since
> find_vcvarsall() is supposed to return the path to the vcvarsall.bat
> file and not an architecture specific setup file. It is later
> called with the arch identifier, which the arch specific setup files
> don't check or use.

The patch does not change anything for Visual Studio Pro. In Visual Studio
Express (+SDK) vcvarsall.bat is broken, so the architecture specific setup
files have to be used (they also work with a superfluous parameter).


> Also note that vcvarsall.bat can take these options:
>
>    x86 (default), x64, amd64, x86_amd64, ia64, x86_ia64
>
> The x86_* options setup the cross compilers.

I think the patch covers all architecture specific files that are
present in the Visual Studio Express + SDK setup.

Visual Studio Pro is protected from all changes by checking for
the presence of the file bin\amd64\vcvarsamd64.bat. This
could probably be done more elegantly by using some obscure
registry value.



As Thorsten mentioned, another option would be to copy bin\vcvars64.bat
to bin\amd64\vcvarsamd64.bat if the latter is not present.

This is harmless, but it is perhaps not really the business of Python
to mess with existing installs.

> I think it's fair to ask the user to setup the environment correctly
before running "python setup.py install"

We're supporting automatic fallback to non-C install for those environments that don't support it.   We're just looking for a predictable CCompilerError to be raised for conditions like this rather than a ValueError where we have to parse the string message.

Stefan Krah wrote:
> > 
> > Stefan Krah <stefan-usenet@bytereef.org> added the comment:
> > 
> > Marc-Andre Lemburg <report@bugs.python.org> wrote:
>> >> Wouldn't it be better to have distutils tell the user about the
>> >> possible options, instead of guessing and then possibly compiling
>> >> extensions which later on don't import or import, but don't work
>> >> as expected ?
> > 
> > That would be an option, yes.
> > 
> > 
>> >> Regarding the latest patch: This is not the right approach, since
>> >> find_vcvarsall() is supposed to return the path to the vcvarsall.bat
>> >> file and not an architecture specific setup file. It is later
>> >> called with the arch identifier, which the arch specific setup files
>> >> don't check or use.
> > 
> > The patch does not change anything for Visual Studio Pro. In Visual Studio
> > Express (+SDK) vcvarsall.bat is broken, so the architecture specific setup
> > files have to be used (they also work with a superfluous parameter).

I guess what I wanted to say is that find_vcvarsall() should
return None for VC Express and code using it should then
revert to using a new find_vcvars() function, which takes the
architecture as parameter and returns the path to the correct
architecture setup file.

Hacking the support into find_vcvarsall() is not the right
approach. You have to add this support one level further up.

>> >> Also note that vcvarsall.bat can take these options:
>> >>
>> >>    x86 (default), x64, amd64, x86_amd64, ia64, x86_ia64
>> >>
>> >> The x86_* options setup the cross compilers.
> > 
> > I think the patch covers all architecture specific files that are
> > present in the Visual Studio Express + SDK setup.

Right, but it doesn't cover the ones available in VS Pro (see
above), which it should for completeness.

> > Visual Studio Pro is protected from all changes by checking for
> > the presence of the file bin\amd64\vcvarsamd64.bat. This
> > could probably be done more elegantly by using some obscure
> > registry value.
> > 
> > 
> > 
> > As Thorsten mentioned, another option would be to copy bin\vcvars64.bat
> > to bin\amd64\vcvarsamd64.bat if the latter is not present.
> > 
> > This is harmless, but it is perhaps not really the business of Python
> > to mess with existing installs.

Not a good idea :-)

PS: Changing the title, since I keep getting the following error messages from the email interface:

There were problems handling your subject line argument list:
- not of form [arg=value,value,...;arg=value,value,...]

Subject was: "Re: [issue7511] msvc9compiler.py: ValueError: [u'path']"

Marc-Andre Lemburg <report@bugs.python.org> wrote:
> >> >> Regarding the latest patch: This is not the right approach, since
> >> >> find_vcvarsall() is supposed to return the path to the vcvarsall.bat
> >> >> file and not an architecture specific setup file. It is later
> >> >> called with the arch identifier, which the arch specific setup files
> >> >> don't check or use.
> > > 
> > > The patch does not change anything for Visual Studio Pro. In Visual Studio
> > > Express (+SDK) vcvarsall.bat is broken, so the architecture specific setup
> > > files have to be used (they also work with a superfluous parameter).
> 
> I guess what I wanted to say is that find_vcvarsall() should
> return None for VC Express and code using it should then
> revert to using a new find_vcvars() function, which takes the
> architecture as parameter and returns the path to the correct
> architecture setup file.
> 
> Hacking the support into find_vcvarsall() is not the right
> approach. You have to add this support one level further up.

I agree that it is nicer if find_vcvarsall() actually returns vcvarsall.
The next level though is query_vcvarsall(), so it would still be wrong
to query vcvars64 in that function.

So unless query_vcvarsall() is renamed to something neutral like set_vcvars(),
I'm not sure where to add the support. But I imagine that renaming is out of
the question.

> Hmm, in http://bugs.python.org/issue7511#msg106420 Tarek appeared to
> be supportive of the patch.

I believe Martin has more knowledge about Windows.

Martin: They are looking for you to review this, in particular see:

http://bugs.python.org/issue7511#msg106420

My environment is Windows 7 64bits with VC++ 2008 Express and Python2.7.2.

I encountered same problem, I have tried all solution posted here, but non of them work.

Eventually, I add "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin" to my PATH and it works.

It appears that the command 

    "vcvarsall.bat" x86 & set

outputs different values depending on whether "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin" in PATH.

Victor, can you apply this patch and report that it works. In particular, if it does *not* work, can you please report the exact way of failing? (if you can, please also try to investigate why it fails).

Oddly, can't reproduce the problem anymore. I try to install win7 on virtual machine, but it works as well. I think the issue might caused by something else... not sure, so strange :S

Will look into detail once I encounter this issue again.

On 64-bit Windows with Visual Studio 2008 Professional, I also need to apply vcvars4.diff to avoid getting the ValueError. Is this something to be expected?

Just bumped into this issue when installing gevent 1.0b2 on Vista 64bit using pip. I think it would be worth to mention that although I'm not quite new to Python and in the past I already investigated issue with distutils not finding vcvarsall.bat (http://stackoverflow.com/a/5122521/95735), I didn't remember about DISTUTILS_USE_SDK environment variable. What I'm trying to say is that there are probably more people like me, having Visual Studio 2008 Express + Windows SDK installed but not knowing distutils. After seeing that Windows SDK installs 64bit compiler into VC\bin folder of Visual Studio's installation I thought that Python would pick it up automatically the same way it does with non Express versions of Visual Studio.

Python 2.7.3
WinXP Professional 2002 + SP3
Visual Studio C++ Express 2008

I bumped into this error trying to install fabric. One of the installation steps is compiling pycrypto.
After looking into C:\Program Files\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat I saw:
"%VS90COMNTOOLS%vsvars32.bat"

so I run:
set VS90COMNTOOLS=C:\Program Files\Microsoft Visual Studio 9.0\Common7\Tools 
directly before pip install fabric and it compiled successfully. Still need to see if it works.

FYI. I was able to build both 32bit/64bit.

Python 2.7.3 (32bit)
WinXP Professional SP3 (x86)
Visual Studio C++ Express 2008 SP1
Microsoft Windows SDK 2008

invoke Windows SDK's `CMD Shell` and do following:

C:\tmp> setenv /x64 /release
C:\tmp> set libpath=dummy
C:\tmp> python setup.py build --plat-name=win-amd64 build_ext --library_dirs=C:\Python27\libs-amd64

details is here: http://www.freia.jp/taka/blog/python-win32-binary-building-and-x64-cross-compiling-on-32bit-platform/index.html

This becomes more of an issue since VC++ 2008 Express is no longer available for download (unless you're an MSDN subscriber), though the compiler itself is still available as part of http://www.microsoft.com/en-us/download/details.aspx?id=3138 (for VC9)

I'll admit I haven't read all the discussion on other issues that relates to this, but I see no reason why the following maps couldn't be tried if vcvarsall.bat is not found. This would at least handle more cases and as far as I can tell is the only realistic option.

VCVARS_TO_WINSDK_BAT = {
    'x86' : 'bin\\vcvars32.bat',
    'amd64' : 'bin\\vcvars64.bat',
    'x86_amd64' : 'bin\\vcvarsx86_amd64.bat',
    'x86_ia64' : 'bin\\vcvarsx86_ia64.bat',
}

VCVARS_TO_VS_BAT = {
    'x86' : 'bin\\vcvars32.bat',
    'amd64' : 'bin\\amd64\\vcvars64.bat',
    'x86_amd64' : 'bin\\x86_amd64\\vcvarsx86_amd64.bat',
    'x86_ia64' : 'bin\\x86_ia64\\vcvarsx86_ia64.bat',
}

I have no idea why, but the SDK installs the batch files into different locations. VCVARS_TO_VS_BAT should be unnecessary, since vcvarsall.bat should always exist in this case, but I see no harm in having more fallback options.

All of these scripts (at least for VC9, VC10 and VC11) will ignore extra arguments.

('ignore extra arguments' meaning they could be returned from find_vcvarsall() and nobody will notice. Though it doesn't bother me if the fallback is handled at a higher level.)

I've dug deeper into VC10 and things are even worse there - without VS installed I don't think there's any way for distutils to work without completely replacing query_vcvarsall() (or msvc9compiler, for that matter).

I'm willing to write such a replacement, assuming I can get permission from my management. But I'd rather not do it if someone else is already working on it.

Hi Steve,

> This becomes more of an issue since VC++ 2008 Express is no longer available for download (unless you're an MSDN subscriber)

Here: http://www.microsoft.com/en-us/download/details.aspx?id=13276

>> This becomes more of an issue since VC++ 2008 Express is no longer available for download (unless you're an MSDN subscriber)
>
> Here: http://www.microsoft.com/en-us/download/details.aspx?id=13276

That's just the service pack and it won't install unless you already have VS installed. There is no way (other than being an MSDN subscriber) to get VS 2008 at this point - I've checked.

On 23.03.2013 16:39, Steve Dower wrote:
> 
> Steve Dower added the comment:
> 
>>> This becomes more of an issue since VC++ 2008 Express is no longer available for download (unless you're an MSDN subscriber)
>>
>> Here: http://www.microsoft.com/en-us/download/details.aspx?id=13276
> 
> That's just the service pack and it won't install unless you already have VS installed. There is no way (other than being an MSDN subscriber) to get VS 2008 at this point - I've checked.

There is, but you have to know what to look for :-)

http://download.microsoft.com/download/E/8/E/E8EEB394-7F42-4963-A2D8-29559B738298/VS2008ExpressWithSP1ENUX1504728.iso

>> That's just the service pack and it won't install unless you already have VS installed. There is no way (other than being an MSDN subscriber) to get VS 2008 at this point - I've checked.
>
> There is, but you have to know what to look for :-)

Fair enough, but I can't (as a Microsoft employee) endorse that link or guarantee that it won't go away :)

I still think it's worth fixing distutils to look in the right places for VC, since there are multiple ways of installing it that all include different batch files.

On 23.03.2013 22:33, Steve Dower wrote:
> 
> Steve Dower added the comment:
> 
>>> That's just the service pack and it won't install unless you already have VS installed. There is no way (other than being an MSDN subscriber) to get VS 2008 at this point - I've checked.
>>
>> There is, but you have to know what to look for :-)
> 
> Fair enough, but I can't (as a Microsoft employee) endorse that link or guarantee that it won't go away :)
> 
> I still think it's worth fixing distutils to look in the right places for VC, since there are multiple ways of installing it that all include different batch files.

Sure, no question about that. Adding support for more MS VC versions
is always a plus :-)

I've added a patch that replaces query_vcvarsall() with an implementation that does not try and use the batch file. Instead, the functionality of the batch file (effectively, reading the base path from the registry and appending constants) is reproduced in Python code.

(There is a second patch for 2.7, since winreg/_winreg is different between the two.)

There is no reason to expect that the installation layout of VC9 and VC10 will change at this stage, though future updates may be required for new versions (though I've already added the mapping for VC11 - not that we're building Python with that one yet).

I have left find_vcvarsall() in there, even though it is no longer used, in case other tools are monkeypatching to work around the existing issue.

Adding patch for 2.7.

Microsoft using Visual Studio 2012 (VC 11.0) now,
Errors happened again. 
I just want to install Cython.........

Error: Unable to find vcvarsall.bat
ValueError: [u'path', u'include', u'lib']
NameError: name 'KEY_BASE' is not defined
KeyError: '9.0'
error: command 'cl.exe' failed: No such file or directory

I must have missed something when I merged files to create the diffs. In any case, you'll still require VC9 or VC10 to be able to build something compatible with a CPython release (though maybe that doesn't matter for Cython? I don't know).

When I get a chance I'll fix up the changes and then ping Python-Dev about getting a review.

so i hit the same exception, here is how i solved it.

environment:
windows 7 enterprise 64 bit
python 2.7.5 64 bit
visual studio 2008 32 bit

issue:
pip installs failed with the above exception.

solution:
there are two parts.
1. install 64 bit components to vs2008
2. modify the PATH to point to 64 bit vs2008 bin folder

to confirm is this solution will help you:
check if there is an amd64 under your vs2008 bin folder:
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\bin
if yes, this solution will not help you.
if missing follow the steps below.

step 1.
control panel
select Microsoft Visual Studio 2008 Professional -ENU
window Visual Studio 2008 Maintenance Mode is opened
click through to and select Add or Remove Features
Under Language Tools->VC++ "X64 Compilers and Tools" should be unchecked.
if it is checked, skip to step 2.
check "X64 Compilers and Tools", Update.
you may have to provide the VS2008 installation CD.
confirm that update creates bin\amd64 subfolder.

step 2.
add 64 bit subfolder to your path.
VS2008_HOME=C:\Program Files (x86)\Microsoft Visual Studio 9.0
PATH=%PATH%;%VS2008_HOME%\VC\bin;%VS2008_HOME%\VC\bin\amd64;
open new command window to receive new PATH changes.
pip installations should no die with the ValueError.

About the "'KEY_BASE' is not defined" error in Steve Dower's diff, I was able to fix it by adding the following line before the KEY_BASE variable is referenced:

KEY_BASE = r"Software\Microsoft\\"

With this, I was finally able to use the patched msvccompiler9.py to build PyCrypto using VS Express 2010 with Windows SDK 7.1 and Python 3.3.2 (64-bit) installed.

If someone can confirm that this is the only thing missing in the 2 versions of the msvccompiler9.diff files, it would be great.

Hope it helps!

I believe that is all that is missing from the patches I posted, though I'd have thought that having Visual C++ 2010 Express installed would be sufficient without the patch (though you didn't mention C++, so maybe you have a different one?).

The speedups extension for MarkupSafe (which has a pure Python fallback) on Python 3.3.3 64-bit was happily compiled with `pip install markupsafe` after applying Steve's patch and Li Wah's definition for KEY_BASE.

The latest patch has an indentation error in an if-else clause, but I can't figure out what exactly was intended by the author.

Steve: Why are you providing a Python 3.3 patch? AFAICT, the issue does not exist with VS 2010 (express).

My patches (which I've backed away from now - between wheels and monkey patching Distutils from a setup.py, there are enough workarounds) also handled the fact that VS 2008 is no longer freely/officially available, and presumably VS 2010 will eventually go the same way. From my point of view, VC++ 2008 Express not including 64-bit compilers is less of a concern than VC++ 2008 Express not being available at all.

The Windows SDK is the only way that VC9 is distributed now, but it doesn't install the vcvarsall.bat file that Distutils needs. The same goes for the version of the SDK that includes VC10. These patches look in the registry for the actual paths, rather than just to find a batch file that isn't a component of the compiler.

(FWIW, the patches should be identical apart from some Python 2/3 differences - the version number is parameterised.)

I think getting this bug fixed would be really nice from a user experience point of view.

I've further into this for my own setup, which is Python 3.3 64-bit, Visual C++ Express 2010 and the Windows SDK v7.1 (plus service packs), which is probably a typical setup for a user looking to build Python extensions without paying for a compiler. Here's what I've noticed, apart from the fact that msvccompiler9.py is a mess.

query_vcvarsall executes "vcvarsall.bat amd64 & set". The first part, trying to delegate to another batch file that doesn't exist, will fail. So no environment variables will be set, but the PATH environment variable always exists, hence "ValueError: ['path']".

The DISTUTILS_USE_SDK environment variable is useless, even leaving aside the fact that MsSdk also needs to be defined. The idea is that distutils will trust that the user has set the appropriate environment variables (e.g. SetEnv.Cmd /Release). However, successful compilation of 64-bit extensions won't rely on this variable at all:

1. DISTUTILS_USE_SDK/MsSdk defined, SetEnv not called: "cl.exe" and friends can't be resolved and nothing works.

2. DISTUTILS_USE_SDK/MsSdk defined, SetEnv called: "cl.exe" and friends can be resolved and everything works.

3. DISTUTILS_USE_SDK/MsSdk undefined, SetEnv not called: "vcvarsall.bat amd64" will fail and we end up with "ValueError: ['path']" as above.

4. DISTUTILS_USE_SDK/MsSdk undefined, SetEnv called: in query_vcvarsall, "vcvarsall.bat amd64" still fails, but it doesn't matter at this point, because the environment variables INCLUDE, LIB, LIBPATH and PATH will exist, so parsing "vcvarsall.bat amd64 & set" works. This doesn't guarantee linking with the correct bitness, but one would hope so :)


I believe SetEnv.Cmd can be found using HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.1\WinSDKTools or HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Microsoft SDKs\Windows\v7.1\WinSDKTools.

In building pycrypto for python 3.4 I applied patch msvccompiler9_33.diff. After applying the patch there is an unassigned variable: KEY_BASE on line 67 of the patched file. 
After setting
KEY_BASE = "Software\\Wow6432Node\\Microsoft\\"
before its first use I was able to successfully build pycrypto.

I'm assuming that the 2.7 patch is still valid but have no idea whether or not any work is needed on 3.4 or 3.5.

There is other work going on that will make the patches unnecessary. One problem is that vcvarsall.bat isn't where msvc9compiler.py is looking, which can be fixed with monkey patching in setup.py or in setuptools. The other problem is that VC9 is hard to get hold of these days. I'm working on both problems and dealing with the big-company-inertia problem.

For 3.5 I'll update msvc9compiler.py as part of my VC14 support. 3.4 (and 3.5 in the meantime) don't need to be updated as the easiest way to get VC10 is still as part of VS 2010. However, when the VS 2010 download is removed then things will become more difficult.

In general, I think we're better off helping package developers make builds/wheels than we are helping users to get the compiler. Windows just doesn't have enough platform incompatibilities to make it worthwhile.

From the comments here, there seems to be a belief that this is somehow related to VC Express only.

I have hit this error when using the MS SDK (both 7.0 on Python 2.7, and 7.1 on Python 3.3 and 3.4) with DISTUTILS_USE_SDK set.

Unless I have somehow misconfigured my installation, this bug prevents me from having any means of building Python extensions using free tools for 64-bit Windows. It is possible I have misconfigured something, as I am working from various notes on the web, plus trial and error, because there is no official documentation I have found on how to set up the SDK for use with distutils in this manner.

Here's another patch fixing the Visual Studio Express edition issue
plus some minor related code-cleanup.

See the Hg repo at:
  https://bitbucket.org/jurko/cpython
branch:
  jurko/distutils_msvc_express_fix

direct link:
  https://bitbucket.org/jurko/cpython/commits/branch/jurko/distutils_msvc_express_fix

Basically - native 64-bit compiler is not available with Visual Studio
Express edition, but its 32-bit to 64-bit cross-compiler is, so the fix just attempts to configure the cross-compiler if configuring the native compiler fails to set all the necessary environment variables.

Tested to work correctly with Visual Studio Express edition installed, and no additional Platform SDK.

I'm also directly attaching a bundle containing the same commits.

I've split up the changes into cleanly separated commits so they should be real easy to review & verify, and there should be no risk of breakage in use cases not explicitly attacked by the proposed changes.

Hope this helps.

Best regards,
  Jurko Gospodnetić

The correct solution to this issue now is to install the Microsoft supplied "Visual C++ Compiler for Python 2.7" package (available as a free download from MSDN). That has all the components needed to build 32 and 64-bit extensions. Your setup.py needs to use setuptools (as there is a setuptools extension to recognise this compiler package) but otherwise the process should be seamless.

Note that in spite of the name, the process should also work for Python 3.2 and 3.3, which also use VS2008 (but I haven't tested that explicitly).

Paul Moore added the comment:
>
> Note that in spite of the name, the process should also work for Python
> 3.2 and 3.3, which also use VS2008 (but I haven't tested that explicitly).
>

Just 3.2; 3.3 was the first version to use VS2010.

Considering the consensus somewhere in the middle of this discussion that this is not a Python bug, the unavailability of VS2008 Express, and the availability of the Visual C++ Compiler for Python 2.7 package, I'm closing this as 'wont fix'.  If anyone just absolutely can't be satisfied by Visual C++ Compiler for Python 2.7, you can reopen.