classification
Title: Can not install C extension modules to inside a venv on Python 3.3.0 for Win32
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, carljm, msmhrt, python-dev, serhiy.storchaka, vinay.sajip
Priority: critical Keywords: patch

Created on 2012-10-03 09:09 by msmhrt, last changed 2012-10-23 22:40 by msmhrt. This issue is now closed.

Files
File name Uploaded Description Edit
Python33_with_venv.txt msmhrt, 2012-10-03 09:09 NG: Python 3.3.0 with venv
Python33_with_virtualenv.txt msmhrt, 2012-10-03 09:10 OK: Python 3.3.0 with virtualenv-1.8.2
Python33_without_venv.txt msmhrt, 2012-10-03 09:11 OK: Python 3.3.0 without venv
Python33_with_venv_2.txt msmhrt, 2012-10-05 09:34 OK: patched Python 3.3.0 with venv
attempted-fixes.diff vinay.sajip, 2012-10-23 12:55 review
Python33_with_venv_3.txt msmhrt, 2012-10-23 16:39 OK: patched Python 3.3.0 with venv
Repositories containing patches
http://hg.python.org/sandbox/vsajip#fix16116
Messages (18)
msg171875 - (view) Author: Masami HIRATA (msmhrt) Date: 2012-10-03 09:09
I'm trying to install C extension modules inside a venv.
It works outside a venv and inside a virtualenv-1.8.2 but breaks inside the venv.

OS: Windows 7 Starter Edition SP1 (32-bit)
Python: 3.3.0 (python-3.3.0.msi)
Compiler: Microsoft Visual C++ 2010 Express SP1
msg171953 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-04 14:23
Comparing the link command which works:

C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:c:\python33\Libs /LIBPATH:C:\temp\venv2\libs /LIBPATH:C:\temp\venv2\PCbuild /EXPORT:PyInit__regex build\temp.win32-3.3\Release\Python3\_regex.obj build\temp.win32-3.3\Release\Python3\_regex_unicode.obj /OUT:build\lib.win32-3.3\_regex.pyd /IMPLIB:build\temp.win32-3.3\Release\Python3\_regex.lib /MANIFESTFILE:build\temp.win32-3.3\Release\Python3\_regex.pyd.manifest

with the one that fails:

C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\temp\venv\libs /LIBPATH:C:\Python33 /LIBPATH:C:\temp\venv
\PCbuild /EXPORT:PyInit__regex build\temp.win32-3.3\Release\Python3\_regex.obj build\temp.win32-3.3\Release\Python3\_regex_unicode.obj /OUT:build\lib.win32-3.3\
_regex.pyd /IMPLIB:build\temp.win32-3.3\Release\Python3\_regex.lib /MANIFESTFILE:build\temp.win32-3.3\Release\Python3\_regex.pyd.manifest

indicates that the link error happens because the library path is incorrectly specified as c:\Python33 rather than c:\Python33\libs.

I will investigate further.
msg171960 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-04 15:07
A little more investigation suggests that the disparity is due to some code in virtualenv's version of distutils.__init__.py, which adds the
"\Libs" entry which allows the virtualenv version to work.

Adding Carl Meyer as nosy, as I'd like his advice on what the best approach to the fix would be.
msg171961 - (view) Author: Carl Meyer (carljm) * Date: 2012-10-04 15:20
On cursory inspection, I agree that this is precisely what the "if win32" block in `virtualenv_embedded/distutils-init.py` is intended to fix, and it seems to me the correct fix is likely to just make the equivalent fix directly in distutils: change the library_dirs-building code in `distutils.command.build_ext:finalize_options` (under the "if os.name == 'nt'" block) to build the path relative to `sys.base_exec_prefix` rather than `sys.exec_prefix`.
msg171962 - (view) Author: Carl Meyer (carljm) * Date: 2012-10-04 15:21
(Actually, to match virtualenv's fix it should add the paths based on both exec_prefix and base_exec_prefix, if they are different.)
msg171970 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-04 16:29
Since I expect it may be some time before the next Python release is out, ISTM a temporary workaround would be to add a line following line 192 of Lib\distutils\command\build_ext.py, which reads

            self.library_dirs.append(os.path.join(sys.exec_prefix, 'libs'))

The new line should read

            self.library_dirs.append(os.path.join(sys.base_exec_prefix, 'libs'))

Which will result in adding both directories to the library search path. This is a little untidy in the case when you're not in a venv, but it should still work. With that change, the link command now looks like

C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\temp\venv\libs /LIBPATH:C:\Python33\libs /LIBPATH:C:\Python33 /LIBPATH:C:\temp\venv\PCbuild /EXPORT:PyInit__regex build\temp.win32-3.3\Release\Python3\_regex.obj build\temp.win32-3.3\Release\Python3\_regex_unicode.obj 
 /OUT:build\lib.win32-3.3\_regex.pyd /IMPLIB:build\temp.win32-3.3\Release\Python3\_regex.lib /MANIFESTFILE:build\temp.win32-3.3\Release\Python3\_regex.pyd.manifest

and the command succeeds.
msg171972 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-04 16:33
If you like, you can make the added line conditional on "if sys.base_exec_prefix != sys.prefix", which is the form the actual fix is likely to take. Thus:

            if sys.base_exec_prefix != sys.prefix:
                self.library_dirs.append(os.path.join(sys.base_exec_prefix, 'libs'))
msg172063 - (view) Author: Masami HIRATA (msmhrt) Date: 2012-10-05 09:34
I have tested the workaround and it works correctly.
Please see attached log file.
msg172068 - (view) Author: Masami HIRATA (msmhrt) Date: 2012-10-05 10:28
btw, it seems to me that "-IC:\Python33\include -IC:\Python33\include" should be "-IC:\Users\msmhrt\mypython\3.3.0\include -IC:\Python33\include".
What do you think about it?
msg172094 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-05 16:00
> I have tested the workaround and it works correctly.
> Please see attached log file.

Thanks for the update.

> btw, it seems to me that "-IC:\Python33\include -IC:\Python33\include"
> should be "-IC:\Users\msmhrt\mypython\3.3.0\include -IC:\Python33\include".

Could well be a similar problem; I'll look into it.
msg173570 - (view) Author: Masami HIRATA (msmhrt) Date: 2012-10-22 23:12
> Could well be a similar problem; I'll look into it.
Is there any progress?
msg173584 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-23 08:45
Sorry, not yet - I haven't been able to spend much time looking at it, but hopefully I will be able to before too long.
msg173607 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-23 12:49
Added fixes in my sandbox repo. Please verify.
msg173624 - (view) Author: Masami HIRATA (msmhrt) Date: 2012-10-23 16:39
I attached new log file. (generated with 3.3.0 + your patch)
msg173627 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2012-10-23 18:06
So you agree that it's working? The include path has the venv's include in it, and the c:\Python33\libs inclusion allows the linking to work. I'll commit the changes to 3.3, 3.4 and close the issue shortly.
msg173635 - (view) Author: Roundup Robot (python-dev) Date: 2012-10-23 19:27
New changeset 43f537339107 by Vinay Sajip in branch '3.3':
Issue #16116: Now uses corrected include and library paths when building C extensions in a venv.
http://hg.python.org/cpython/rev/43f537339107

New changeset 5e9f656c3d67 by Vinay Sajip in branch 'default':
Closes #16116: Merged fix from 3.3.
http://hg.python.org/cpython/rev/5e9f656c3d67
msg173636 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-10-23 20:11
>+- Issue #16116: Fix include and library paths to be correctwhen  building C

Spaces.
msg173645 - (view) Author: Masami HIRATA (msmhrt) Date: 2012-10-23 22:40
> So you agree that it's working?
Yes, the patch works correctly. Thank you!
History
Date User Action Args
2012-10-23 22:40:29msmhrtsetmessages: + msg173645
2012-10-23 20:11:48serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg173636
2012-10-23 19:27:55python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg173635

resolution: fixed
stage: resolved
2012-10-23 18:06:18vinay.sajipsetmessages: + msg173627
2012-10-23 16:39:35msmhrtsetfiles: + Python33_with_venv_3.txt

messages: + msg173624
2012-10-23 12:55:30vinay.sajipsetfiles: + attempted-fixes.diff
keywords: + patch
2012-10-23 12:49:27vinay.sajipsethgrepos: + hgrepo155
messages: + msg173607
2012-10-23 08:45:13vinay.sajipsetmessages: + msg173584
2012-10-22 23:12:20msmhrtsetmessages: + msg173570
2012-10-05 16:16:21asvetlovsetnosy: + asvetlov
2012-10-05 16:00:07vinay.sajipsetmessages: + msg172094
2012-10-05 10:28:19msmhrtsetmessages: + msg172068
2012-10-05 09:34:45msmhrtsetfiles: + Python33_with_venv_2.txt

messages: + msg172063
2012-10-04 16:33:52vinay.sajipsetmessages: + msg171972
2012-10-04 16:29:40vinay.sajipsetmessages: + msg171970
2012-10-04 15:21:54carljmsetmessages: + msg171962
2012-10-04 15:20:01carljmsetmessages: + msg171961
2012-10-04 15:07:19vinay.sajipsetnosy: + carljm
messages: + msg171960
2012-10-04 14:23:15vinay.sajipsetmessages: + msg171953
2012-10-03 19:11:39pitrousetpriority: normal -> critical
nosy: + vinay.sajip

components: + Windows
versions: + Python 3.4
2012-10-03 09:11:16msmhrtsetfiles: + Python33_without_venv.txt
2012-10-03 09:10:23msmhrtsetfiles: + Python33_with_virtualenv.txt
2012-10-03 09:09:08msmhrtcreate