classification
Title: Distutils fails to build extension in path with spaces
Type: behavior Stage: resolved
Components: Distutils, Windows Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: distutils compiler not handling spaces in path to output/src files
View: 4508
Assigned To: Nosy List: BreamoreBoy, Ramchandra Apte, eric.araujo, klo.uo, loewis, serhiy.storchaka, tarek, zach.ware
Priority: normal Keywords:

Created on 2013-01-09 12:22 by klo.uo, last changed 2016-01-18 21:59 by zach.ware. This issue is now closed.

Messages (10)
msg179434 - (view) Author: klo uo (klo.uo) Date: 2013-01-09 12:22
I noticed this issue, while trying to compile Cython extension, when source file is in path with spaces. Extension wouldn't compile because LIBDIR is passed to MinGW g++ (though not to gcc) or MSVC compilers unquoted. I tracked the problem to "distutils/build_ext.py" in get_ext_fullpath() function.

I patched it, this way:

========================================
--- build_ext.bak
+++ build_ext.py
@@ -647,6 +647,11 @@
         package = '.'.join(modpath[0:-1])
         build_py = self.get_finalized_command('build_py')
         package_dir = os.path.abspath(build_py.get_package_dir(package))
+        try:
+            from win32api import GetShortPathName
+            package_dir = GetShortPathName(package_dir)
+        except:
+            pass
         # returning
         #   package_dir/filename
         return os.path.join(package_dir, filename)
========================================

which is just one way to do it.
msg179439 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2013-01-09 13:17
Can you fix the bare except?
msg179440 - (view) Author: klo uo (klo.uo) Date: 2013-01-09 13:22
> Can you fix the bare except?

========================================
...
except ImportError:
    pass
========================================
msg179444 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-09 14:24
get_ext_fullpath() has no relations to this bug. The bug is probably in spawn() functions, perhaps in _nt_quote_args() or something like. It looks as Windows specific bug (Posix passes command arguments as a list, not joining it into string).
msg179473 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2013-01-09 16:27
klo.uo: can you kindly provide a working (or, rather, failing) example? A trivial "hello-world" kind of package could do, along with a report what path you unpacked it in, and what error you get.

Your patch is not applicable to Python, since it requires the win32 extensions, which we cannot assume to be present.
msg179482 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-01-09 18:06
I think this is a duplicate report; could you please search for an existing open report for the same bug?
msg179484 - (view) Author: klo uo (klo.uo) Date: 2013-01-09 18:15
> klo.uo: can you kindly provide a working (or, rather, failing)
> example? A trivial "hello-world" kind of package could do, along
> with a report what path you unpacked it in, and what error you get.

As mentioned in opening thread, this doesn't happen with Cython's 
"hello world" example, as `gcc` somehow doesn't trigger this problem. 
Maybe because it's handled by Cython's own distutils copies, maybe it's 
handled by Numpy's distutils version or it could be by Python's 
distutils.

I noticed this issue while using Numpy with Cython. Here is simple example:

C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyx:
========================================
cimport numpy

def sum(x):
    cdef numpy.ndarray[int, ndim=1] arr = x
    cdef int i, s = 0
    for i in range(arr.shape[0]):
        s += arr[i]
    return s
========================================

C:\Documents and Settings\klo\My Documents\code\python\misc\setup.py:
========================================
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from numpy.distutils.misc_util import get_numpy_include_dirs

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("test", ["test.pyx"], include_dirs=get_numpy_include_dirs())]
    )
========================================

command line: `python setup.py build_ext --inplace`

MinGW result:
========================================
...
g++ -shared build\temp.win32-2.7\Release\test.o -LC:\Python27\libs -LC:\Python27\PCbuild -lpython27 -lmsvcr90 -o C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd
Found executable C:\MinGW\bin\g++.exe
g++.exe: error: and: No such file or directory
g++.exe: error: Settings\klo\My: No such file or directory
g++.exe: error: Documents\code\python\misc\test.pyd: No such file or directory
error: Command "g++ -shared build\temp.win32-2.7\Release\test.o -LC:\Python27\libs -LC:\Python27\PCbuild -lpython27 -lmsvcr90 -o C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd" failed with exit status 1
========================================

Similar result if I use MSVC compiler:

MSVC result:
========================================
building 'test' extension
c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\lib\site-packages\numpy\core\include -IC:\Python27\include -IC:\Python27\PC /Tctest.c /Fobuild\temp.win32-2.7\Release\test.obj
Found executable c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe
c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:inittest build\temp.win32-2.7\Release\test.obj /OUT:C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd /IMPLIB:build\temp.win32-2.7\Release\test.lib /MANIFESTFILE:build\temp.win32-2.7\Release\test.pyd.manifest /MANIFEST
Found executable c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe
LINK : fatal error LNK1181: cannot open input file 'and.obj'
error: Command "c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:inittest build\temp.win32-2.7\Release\test.obj /OUT:C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd /IMPLIB:build\temp.win32-2.7\Release\test.lib /MANIFESTFILE:build\temp.win32-2.7\Release\test.pyd.manifest /MANIFEST" failed with exit status 1181
========================================

So issue is with unquoted path for output file, and not for LIBDIR as I wrote
previously (perhaps I misread the logs, as I was trying to fix similar issue
for Theano, at the same time)

> Your patch is not applicable to Python, since it requires the win32
> extensions, which we cannot assume to be present.

Yes, I provided patch that worked for me temporarily, as I'm not 
familiar with distutils, and that's as far as I went.
But source issue is probably elsewhere, as Serhiy suggested.
msg179485 - (view) Author: klo uo (klo.uo) Date: 2013-01-09 18:21
I found two similar issues:

1. distutils compiler not handling spaces in path to output/src files: http://bugs.python.org/issue4508

with patches for unix compiler and cygwin compiler

2. Distutils does not put quotes around paths that contain spaces when compiling with MSVC: http://bugs.python.org/issue13765

with some patch for MSVC

Problem seems to be deeper than provided patches, as evident also from my report
msg224333 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-30 18:03
Of the two issues mentioned in msg179485 #4508 is still open but #13765 has been closed "not a bug".
msg258553 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-01-18 21:59
This isn't obviously a distutils bug, but sounds strikingly similar to #4508.  Closing as a duplicate of #4508.
History
Date User Action Args
2016-01-18 21:59:58zach.waresetstatus: open -> closed

superseder: distutils compiler not handling spaces in path to output/src files
assignee: eric.araujo ->

nosy: + zach.ware
messages: + msg258553
resolution: duplicate
stage: needs patch -> resolved
2014-07-30 18:03:59BreamoreBoysetnosy: + BreamoreBoy

messages: + msg224333
versions: + Python 3.4, Python 3.5
2013-01-09 18:21:32klo.uosetmessages: + msg179485
2013-01-09 18:15:50klo.uosetmessages: + msg179484
2013-01-09 18:06:22eric.araujosetmessages: + msg179482
2013-01-09 16:27:58loewissetnosy: + loewis
messages: + msg179473
2013-01-09 14:24:33serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg179444

components: + Windows
type: behavior
stage: needs patch
2013-01-09 13:22:14klo.uosetmessages: + msg179440
2013-01-09 13:17:22Ramchandra Aptesetnosy: + Ramchandra Apte
messages: + msg179439
2013-01-09 12:22:42klo.uocreate