I'm trying to install C extension modules inside a venv.
It works outside a venv and inside a virtualenv-1.8.2 but breaks inside the venv.

OS: Windows 7 Starter Edition SP1 (32-bit)
Python: 3.3.0 (python-3.3.0.msi)
Compiler: Microsoft Visual C++ 2010 Express SP1

Comparing the link command which works:

C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\python33\Libs /LIBPATH:C:\temp\venv2\libs /LIBPATH:C:\temp\venv2\PCbuild /EXPORT:PyInit__regex build\temp.win32-3.3\Release\Python3\_regex.obj build\temp.win32-3.3\Release\Python3\_regex_unicode.obj /OUT:build\lib.win32-3.3\_regex.pyd /IMPLIB:build\temp.win32-3.3\Release\Python3\_regex.lib /MANIFESTFILE:build\temp.win32-3.3\Release\Python3\_regex.pyd.manifest

with the one that fails:

C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\temp\venv\libs /LIBPATH:C:\Python33 /LIBPATH:C:\temp\venv
\PCbuild /EXPORT:PyInit__regex build\temp.win32-3.3\Release\Python3\_regex.obj build\temp.win32-3.3\Release\Python3\_regex_unicode.obj /OUT:build\lib.win32-3.3\
_regex.pyd /IMPLIB:build\temp.win32-3.3\Release\Python3\_regex.lib /MANIFESTFILE:build\temp.win32-3.3\Release\Python3\_regex.pyd.manifest

indicates that the link error happens because the library path is incorrectly specified as c:\Python33 rather than c:\Python33\libs.

I will investigate further.

A little more investigation suggests that the disparity is due to some code in virtualenv's version of distutils.__init__.py, which adds the
"\Libs" entry which allows the virtualenv version to work.

Adding Carl Meyer as nosy, as I'd like his advice on what the best approach to the fix would be.

On cursory inspection, I agree that this is precisely what the "if win32" block in `virtualenv_embedded/distutils-init.py` is intended to fix, and it seems to me the correct fix is likely to just make the equivalent fix directly in distutils: change the library_dirs-building code in `distutils.command.build_ext:finalize_options` (under the "if os.name == 'nt'" block) to build the path relative to `sys.base_exec_prefix` rather than `sys.exec_prefix`.

(Actually, to match virtualenv's fix it should add the paths based on both exec_prefix and base_exec_prefix, if they are different.)

Since I expect it may be some time before the next Python release is out, ISTM a temporary workaround would be to add a line following line 192 of Lib\distutils\command\build_ext.py, which reads

            self.library_dirs.append(os.path.join(sys.exec_prefix, 'libs'))

The new line should read

            self.library_dirs.append(os.path.join(sys.base_exec_prefix, 'libs'))

Which will result in adding both directories to the library search path. This is a little untidy in the case when you're not in a venv, but it should still work. With that change, the link command now looks like

C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\temp\venv\libs /LIBPATH:C:\Python33\libs /LIBPATH:C:\Python33 /LIBPATH:C:\temp\venv\PCbuild /EXPORT:PyInit__regex build\temp.win32-3.3\Release\Python3\_regex.obj build\temp.win32-3.3\Release\Python3\_regex_unicode.obj 
 /OUT:build\lib.win32-3.3\_regex.pyd /IMPLIB:build\temp.win32-3.3\Release\Python3\_regex.lib /MANIFESTFILE:build\temp.win32-3.3\Release\Python3\_regex.pyd.manifest

and the command succeeds.

If you like, you can make the added line conditional on "if sys.base_exec_prefix != sys.prefix", which is the form the actual fix is likely to take. Thus:

            if sys.base_exec_prefix != sys.prefix:
                self.library_dirs.append(os.path.join(sys.base_exec_prefix, 'libs'))

I have tested the workaround and it works correctly.
Please see attached log file.

btw, it seems to me that "-IC:\Python33\include -IC:\Python33\include" should be "-IC:\Users\msmhrt\mypython\3.3.0\include -IC:\Python33\include".
What do you think about it?

> I have tested the workaround and it works correctly.
> Please see attached log file.

Thanks for the update.

> btw, it seems to me that "-IC:\Python33\include -IC:\Python33\include"
> should be "-IC:\Users\msmhrt\mypython\3.3.0\include -IC:\Python33\include".

Could well be a similar problem; I'll look into it.

> Could well be a similar problem; I'll look into it.
Is there any progress?

Sorry, not yet - I haven't been able to spend much time looking at it, but hopefully I will be able to before too long.

Added fixes in my sandbox repo. Please verify.

I attached new log file. (generated with 3.3.0 + your patch)

So you agree that it's working? The include path has the venv's include in it, and the c:\Python33\libs inclusion allows the linking to work. I'll commit the changes to 3.3, 3.4 and close the issue shortly.

New changeset 43f537339107 by Vinay Sajip in branch '3.3':
Issue #16116: Now uses corrected include and library paths when building C extensions in a venv.
http://hg.python.org/cpython/rev/43f537339107

New changeset 5e9f656c3d67 by Vinay Sajip in branch 'default':
Closes #16116: Merged fix from 3.3.
http://hg.python.org/cpython/rev/5e9f656c3d67

>+- Issue #16116: Fix include and library paths to be correctwhen  building C

Spaces.

> So you agree that it's working?
Yes, the patch works correctly. Thank you!