classification
Title: Distutils does not put quotes around paths that contain spaces when compiling with MSVC
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: almar, eric.araujo, mhammond, tarek
Priority: normal Keywords: needs review, patch

Created on 2012-01-11 10:53 by almar, last changed 2012-01-13 15:19 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
distutils_spaces_in_libdir.patch almar, 2012-01-11 10:53 Patch review
Messages (6)
msg151054 - (view) Author: Almar Klein (almar) Date: 2012-01-11 10:53
I found an easy to solve bug in distutils, which is causing problems with compiling Cython code on Windows. I have reproduced this on Python 2.6 and Python 3.2 (32 bit).

The problem occurs with the native msvc compiler. Using gcc (MinGW) works fine.

The problem is that the command to link the libraries does not put double quotes around paths that have spaces in them. Unfortunately, the path where I have Python installed has spaces in it ("c:/program files/python26"). Small example of part of a link command: 

/LIBPATH:C:\Program Files (x86)\python32\libs. 

Note that the include_dirs DO have double quotes around them.

The problem is easily solved (I confirmed this) by a small change in msvc9compiler.py and msvccompiler.py (see also the patch):

def library_dir_option(self, dir):  # OLD VERSION
     return "/LIBPATH:" + dir

def library_dir_option(self, dir): # FIXED VERSION
    if ' ' in dir and not dir.startswith('"'):
        dir = '"%s"' % dir
    return "/LIBPATH:" + dir

I tried to see if it would be nicer to apply a change elsewhere, e.g. where library_dir_option() is called. However, it is called nowhere from within the distutils package. In my case I suspect numpy.distutils or a part of Cython calls it. Anyway, in my opinion you should be able to pass this function a dirname that has a space in it, and get the right (partial) command. For the record, this change is quite safe because it checks whether quotes are already present. 


===== Below follows a minimal Cython example and traceback =====

===== test_.pyx
def foo():
print('hello')


===== setup.py
import os, sys
from Cython.Distutils import build_ext
from distutils.core import setup
from distutils.extension import Extension
from numpy.distutils.misc_util import get_numpy_include_dirs

# Ugly hack so I can run setup.py in my IDE
sys.argv = ['setup.py', 'build_ext', '--inplace']

# Init include dirs
include_dirs = ['.']
include_dirs.extend(get_numpy_include_dirs())

# Creat Extensions
ext_modules = [
     Extension('test_', ['test_.pyx'],
        include_dirs=include_dirs,
        ),
     ]

# Compile
setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = ext_modules,
    )

print('Successfully compiled cython file: test_')


===== output when running setup.py
running build_ext
No module named msvccompiler in numpy.distutils; trying from distutils
cythoning test_.pyx to test_.c
building 'test_' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -I. -I"C:\Program Files (x86)\python32\lib\site-packages\numpy\core\include" -I"C:\Program Files (x86)\python32\include" -I"C:\Program Files (x86)\python32\PC" /Tctest_.c /Fobuild\temp.win32-3.2\Release\test_.obj
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Program Files (x86)\python32\libs /LIBPATH:C:\Program Files (x86)\python32\PCbuild /EXPORT:PyInit_test_ build\temp.win32-3.2\Release\test_.obj /OUT:C:\almar\projects\py\cmu1394\test_.pyd /IMPLIB:build\temp.win32-3.2\Release\test_.lib /MANIFESTFILE:build\temp.win32-3.2\Release\test_.pyd.manifest
Found executable C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\link.exe
LINK : fatal error LNK1181: cannot open input file 'Files.obj'
msg151135 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-12 16:29
Thanks for the report and patch.  How can we test it?  Do we need to install a Python on a path with embedded spaces?
msg151136 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-12 16:30
Oh and BTW it would be very helpful if you could adapt your test script to be pure-distutils, without Cython or Numpy extensions.
msg151174 - (view) Author: Almar Klein (almar) Date: 2012-01-13 14:52
Ok, I went to prepare a minimal example that does not use Cython nor Numpy. And then the problem was gone. Even more so, my fix would cause a problem, because somewhere quotes are placed around the entire command:

...link.exe /DLL "/LIBPATH:C:\Program Files (x86)\python32\libs" etc.

This somewhere is in spawn(), which calls nt_quote_args().

So I went on to search for the cause in Cython, and after that in Numpy. I have no traced it to Numpy, because it does NOT use the nt_quote_args() function of distutils.

So I think we can close the issue. I will open an issue at numpy.
msg151175 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-13 15:10
OK, thanks.  Please add a link to the Numpy or Cython bug report here.
msg151176 - (view) Author: Almar Klein (almar) Date: 2012-01-13 15:14
This issue is posted at http://projects.scipy.org/numpy/ticket/2018
History
Date User Action Args
2012-01-13 15:19:37eric.araujosetstatus: open -> closed
resolution: not a bug
2012-01-13 15:14:54almarsetstatus: closed -> open
resolution: not a bug -> (no value)
messages: + msg151176
2012-01-13 15:10:53eric.araujosetstatus: open -> closed
messages: + msg151175

assignee: tarek -> eric.araujo
resolution: not a bug
stage: patch review -> resolved
2012-01-13 14:52:49almarsetmessages: + msg151174
2012-01-12 16:30:23eric.araujosetmessages: + msg151136
2012-01-12 16:29:19eric.araujosetkeywords: + needs review

stage: patch review
messages: + msg151135
versions: + Python 2.7, Python 3.3, - Python 2.6
2012-01-11 10:53:26almarcreate