I found this bug when using scipy's weave and having some inline C/C++
code. Weave uses distutils to compile the source in its own directory,
python was installed under "program files". Unfortunately it doesn't
seem to handle spaces in the path to either the output or the temporary
source files.
I checked that this was the only problem by copying the weave directory
from site-packages and put it in another directory without spaces in the
path - weave and distutils then worked fine.
I then tried the g++ command seen in the stack trace at the command line
and got the same error - so altered the command by correctly quoting the
 -c "path/to source/a.c" and the -o objects.
This worked so set out changing it in distutils.

The two files I altered were unixcompiler.py (surprising as I am under
windows!) and cygwinccompiler.py 

This was clearly an older stable build so I checked out the python trunk
r67510 on windows (32bit), made a patch, and tested with my weave
dependant code.
(I am not familiar enough with distutils to make a standalone test).

The same method to cause a problem has the same effect in ubuntu - the
build directory seems to default to users home, but as the source path
has a space in it g++ gets confused.

The patch hasn't been accepted for the trunk, yet, so it is not a
candidate for a 2.5 backport.

Hi. I am not familiar with weave. Could you provide a small sample of
code that raises this issue. This way, I will be able to write the
standalone test we can integrate in distutils together with your fix.

Ok - this bug has nothing to do with weave itself. I think any extension
written in C or C++ that has a space in the filename will raise the
error in both windows and gnu/linux.
Here is a simple c extension with and without spaces. The one with
spaces in the filename fails to build on standard python. Whereas the
one without should build fine. Did for me on windows 2000 on both python
2.5 and 2.6 with and without spaces in their paths.

I couldn't reproduce the problem yet, using your sample.

I tried to compile on Mac OSX (Python 2.5 GCC) and Windows XP (Python
2.5 , MinGW 5.1.4 and SWIG 1.2.3) and it worked well.

The output was right, (the .o file has a space, but the pyd/so files
come out properly), and I could use the gcd() function without any problem.

The call to the compiler quotes the c file correctly:

    ...
    c:\Program Files\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall
-Ic:\Python25\include -Ic:\Python25\PC -c "ex ample.c" -o
"build\temp.win32-2.5\Release\ex ample.o"
    ...

This is done automatically with distutils.spawn._nt_quote_args under
Windows for instance. And this is basically what your patch is doing.

When does it crash exactly for you ? Do you have a traceback ?

I also found this bug while using SciPy's weave on Windows XP to compile
some inline c/c++ code.

Thorney's distutils_compiler_quoting.patch fixed the problem for me.


Here is the relevant part of the traceback (I hope):
"""
Found executable C:\Program Files\Topographica\python_topo\mingw\bin\g++.exe
g++.exe:
c:\docume~1\ceball\locals~1\temp\ceball\python25_intermediate\compiler_
894ad5ed761bb51736c6d2b7872dc212\Release\program: No such file or directory
g++.exe:
files\topographica\python_topo\lib\site-packages\weave\scxx\weave_imp.o:
 No such file or directory
"""

Chris

Comments on the SciPy-Dev mailing list:

>>>
While testing Scipy's weave on several different Windows installations, I came
across some problems with spaces in paths that often prevent weave from working.
I can see a change that could probably get weave working on most Windows
installations, but it is a quick hack. Someone knowledgeable about distutils
(and numpy.distutils?) might be able to help me fix this properly. Below I
describe three common problems with weave on Windows, in the hope that this
information helps others, or allows someone to suggest how to fix the spaces-in-
paths problem properly.

I think there are three common problems that stop weave from working on Windows.
The first is not having a C compiler. Both Python(x,y) and EPD provide a C
compiler that seems to work fine, which is great!

The second problem is that if weave is installed to a location with a space in
the path, linking fails. There is already a scipy bug report about this
(http://projects.scipy.org/scipy/ticket/809). I've just commented on that
report, saying the problem appears to be with distutils, and there is already a
Python bug report about it (http://bugs.python.org/issue4508). Maybe someone
could close this scipy bug, or link it to the Python one somehow? In any case,
when using Python(x,y) or EPD, this bug will not show up if the default
installation locations are accepted. So, that's also good news!

The third problem is that if the Windows user name has a space in it (which in
my experience is quite common), compilation fails. Weave uses the user name to
create a path for its "intermediate" and "compiled" files. When the compilation
command is issued, the path with the space in it is also not quoted. Presumably
that is another error in distutils (or numpy.distutils)? Unfortunately I wasn't
able to pinpoint what function is failing to quote strings properly, because I
couldn't figure out the chain that leads to the compiler being called. However,
I can avoid the problem by removing spaces from the user name in weave itself
(catalog.py):

def whoami():
   """return a string identifying the user."""
   return (os.environ.get("USER") or os.environ.get("USERNAME") or
"unknown").replace(" ","")

(where I have added .replace(" ","") to the existing code).
<<<

>>>
As well as that, I have another problem on another computer (which I
don't have access to right now) where I have a user name with a space in
it, and that causes several problems (although I think these problems
are the fault of distutils and nothing to do with scipy). I found a way
to work around that though by changing the environment variable for
USERNAME from "Dan Goodman" to "DanGoo~1".
<<<

>>>
Unfortunately, it is still often necessary to patch weave to get it to work on Windows. If a Windows user name includes a space, Weave tries to use paths with spaces for its "intermediate" and "compiled" files. I have no idea if this is a problem with distutils (not quoting strings correctly), or if Weave itself should do something different, but I myself hack weave installations on Windows.
<<<

I tried using a space in a directory rather than in the file name and it worked. Thorney, can you repost your example as plain text files (separate source files or a diff against an empty dir) or at worst in an archive format readable by free software? Thank you.

(Don’t remember why I changed component, reverting.)

Sorry, here is the example as a diff against an empty directory. 

(I'm 100% sure the Ubuntu archiving program that I just used is free software though...)

I can’t open the archive with unrar-free, and can’t apply the diff with patch (tried context mode and ed mode). Would you mind making it a unified diff this time?

Hopefully this works! Otherwise would a tar.gz or a zip archive be ok?

Thanks, this time it worked. I had to remove the example.i source file in setup.py since I don’t have swig installed, but apart from that, compiling “ex ample.c” or “ex ample/ex ample.c” worked.

Next steps:
- Write something similar to your example (can’t just copy it since Python is not under GPL) as a unit test for distutils; I’ll do it if nobody does it shortly.
- Confirm that the bug is present on windows and hunt for it (paths translation? argument quoting?)

Ok, I found a similar problem with MMTK. I am currently altering my distutils package to add a function called nt_quote_dir that adds quotes to paths with spaces and then applies it to each path if the platform is win32. When I'm done I will submit a diff after confirming that it works.

> Ok, I found a similar problem with MMTK.
I don’t know what that is.

> I am currently altering my distutils package to add a function called nt_quote_dir
> that adds quotes to paths with spaces and then applies it to each path if the platform
> is win32.
You shouldn’t have to do that; there’s already a similar function in distutils.spawn.

> When I'm done I will submit a diff after confirming that it works.
What is needed is a diff for distutils 2.7 or 3.2 adding a test with a file containing a space.  I can’t do anything with diffs for other projects.

i have to report that this issue also happens on Linux and Mac as well.
no matter wether python 2.6 or 2.7 is used.

Thanks for the confirmation.  Note that the versions field in this bug tracker indicates the versions that will get fixed, not all versions with the bug; 2.6 only gets security fixes now.

Are you interested in updating the patch with a test (see my previous messages)?

I've put together a patch adding the test requested. There is no problem on my Ubuntu machine with python 3.3.
There is a comment in the file saying Don't load the xx module more than once, I am unsure whether my patch (using a renamed c file) violates this?


One extremely minor nitpick that I'll mention is inconsistency with spaces in the module name. One can create a python file "my file.py" and can import it with __import__("my file"). I couldn't do the same for a C extension.

Building a C extension with distutils:

  module_name = 'ex ample'
  example_ext = Extension(module_name, [target_c])
  dist = Distribution({'name': module_name, 'ext_modules': [example_ext]})

Doesn't work due to the import mechanism looks for a PyInit function with the new module name (including a space):

  ImportError: dynamic module does not define init function (PyInit_ex ample)

I don't think this is worth making a ticket over.

I am having the same issue on Debian 7 x86-64 (3.2.0-4-amd64)
Python 2.7.3 (default, Jan  2 2013, 13:56:14) // [GCC 4.7.2] on linux2

Not noticing this bug I've opened a duplicate, for which the interesting comment can be found here: http://bugs.python.org/issue20824#msg212548

Please let me know if you need any files and I will try hard to provide some in my free time (also unittest from what I've read in the comments posted here).

Thanks Brian, let’s try and get this fixed.

> I've put together a patch adding the test requested. There is no problem on
> my Ubuntu machine with python 3.3.

Are you saying the test does not reproduce the bug discussed here?

> There is a comment in the file saying Don't load the xx module more than
> once, I am unsure whether my patch (using a renamed c file) violates this?

Hm I’m not quite sure if it’s enough that the extensions use different file names, or if they should also have different names inside the code.  Existing tests already create and import xx multiple times though…

> One can create a python file "my file.py" and can import it with
> __import__("my file"). I couldn't do the same for a C extension.

One can’t do “import my file” though, so I would sweep this under the rug as an obscure corner case :‑)

Confirmed still broken in 3.6.3.

I ran into this when running CFFI. It does very wrong things with the path to cl.exe and include files with path spaces.

My workaround is to monkeypatch distutils.ccompiler.gen_preprocess_options such that include paths are formed as -I"path" and not -Ipath; and to monkeypatch distutils.spawn._nt_quote_args so that it leaves all of the arguments unquoted except the first arg (path to cl.exe), which I only got working by using tilde-mangled short names.

Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools