classification
Title: distutils compiler not handling spaces in path to output/src files
Type: behavior Stage: test needed
Components: Distutils, Distutils2 Versions: Python 3.3, Python 3.2, Python 2.7, 3rd party
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Thorney, alexis, ceball, drunax, eric.araujo, fishdude, ipatrol, loewis, tarek
Priority: normal Keywords: patch

Created on 2008-12-03 21:58 by Thorney, last changed 2014-03-07 10:37 by eric.araujo.

Files
File name Uploaded Description Edit
distutils_compiler_quoting.patch Thorney, 2008-12-03 21:58 Patch to quote src and object args in distutils c compiler review
python_extension_bug.rar Thorney, 2009-01-12 01:20 Example use of distutils that fails with spaces in the source code filename.
4508example.diff Thorney, 2010-08-25 02:01
distutils_space_test.patch Thorney, 2012-08-05 10:58 Distutils Test (Diff against 5abea8a43f19) review
Messages (19)
msg76848 - (view) Author: Brian Thorne (Thorney) Date: 2008-12-03 21:58
I found this bug when using scipy's weave and having some inline C/C++
code. Weave uses distutils to compile the source in its own directory,
python was installed under "program files". Unfortunately it doesn't
seem to handle spaces in the path to either the output or the temporary
source files.
I checked that this was the only problem by copying the weave directory
from site-packages and put it in another directory without spaces in the
path - weave and distutils then worked fine.
I then tried the g++ command seen in the stack trace at the command line
and got the same error - so altered the command by correctly quoting the
 -c "path/to source/a.c" and the -o objects.
This worked so set out changing it in distutils.

The two files I altered were unixcompiler.py (surprising as I am under
windows!) and cygwinccompiler.py 

This was clearly an older stable build so I checked out the python trunk
r67510 on windows (32bit), made a patch, and tested with my weave
dependant code.
(I am not familiar enough with distutils to make a standalone test).

The same method to cause a problem has the same effect in ubuntu - the
build directory seems to default to users home, but as the source path
has a space in it g++ gets confused.
msg77510 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-12-10 08:54
The patch hasn't been accepted for the trunk, yet, so it is not a
candidate for a 2.5 backport.
msg78632 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2008-12-31 16:46
Hi. I am not familiar with weave. Could you provide a small sample of
code that raises this issue. This way, I will be able to write the
standalone test we can integrate in distutils together with your fix.
msg79641 - (view) Author: Brian Thorne (Thorney) Date: 2009-01-12 01:20
Ok - this bug has nothing to do with weave itself. I think any extension
written in C or C++ that has a space in the filename will raise the
error in both windows and gnu/linux.
Here is a simple c extension with and without spaces. The one with
spaces in the filename fails to build on standard python. Whereas the
one without should build fine. Did for me on windows 2000 on both python
2.5 and 2.6 with and without spaces in their paths.
msg80066 - (view) Author: Tarek Ziadé (tarek) * (Python committer) Date: 2009-01-18 00:45
I couldn't reproduce the problem yet, using your sample.

I tried to compile on Mac OSX (Python 2.5 GCC) and Windows XP (Python
2.5 , MinGW 5.1.4 and SWIG 1.2.3) and it worked well.

The output was right, (the .o file has a space, but the pyd/so files
come out properly), and I could use the gcd() function without any problem.

The call to the compiler quotes the c file correctly:

    ...
    c:\Program Files\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall
-Ic:\Python25\include -Ic:\Python25\PC -c "ex ample.c" -o
"build\temp.win32-2.5\Release\ex ample.o"
    ...

This is done automatically with distutils.spawn._nt_quote_args under
Windows for instance. And this is basically what your patch is doing.

When does it crash exactly for you ? Do you have a traceback ?
msg83748 - (view) Author: C. E. Ball (ceball) Date: 2009-03-18 13:57
I also found this bug while using SciPy's weave on Windows XP to compile
some inline c/c++ code.

Thorney's distutils_compiler_quoting.patch fixed the problem for me.


Here is the relevant part of the traceback (I hope):
"""
Found executable C:\Program Files\Topographica\python_topo\mingw\bin\g++.exe
g++.exe:
c:\docume~1\ceball\locals~1\temp\ceball\python25_intermediate\compiler_
894ad5ed761bb51736c6d2b7872dc212\Release\program: No such file or directory
g++.exe:
files\topographica\python_topo\lib\site-packages\weave\scxx\weave_imp.o:
 No such file or directory
"""

Chris
msg114867 - (view) Author: Brian Thorne (Thorney) Date: 2010-08-25 01:04
Comments on the SciPy-Dev mailing list:

>>>
While testing Scipy's weave on several different Windows installations, I came
across some problems with spaces in paths that often prevent weave from working.
I can see a change that could probably get weave working on most Windows
installations, but it is a quick hack. Someone knowledgeable about distutils
(and numpy.distutils?) might be able to help me fix this properly. Below I
describe three common problems with weave on Windows, in the hope that this
information helps others, or allows someone to suggest how to fix the spaces-in-
paths problem properly.

I think there are three common problems that stop weave from working on Windows.
The first is not having a C compiler. Both Python(x,y) and EPD provide a C
compiler that seems to work fine, which is great!

The second problem is that if weave is installed to a location with a space in
the path, linking fails. There is already a scipy bug report about this
(http://projects.scipy.org/scipy/ticket/809). I've just commented on that
report, saying the problem appears to be with distutils, and there is already a
Python bug report about it (http://bugs.python.org/issue4508). Maybe someone
could close this scipy bug, or link it to the Python one somehow? In any case,
when using Python(x,y) or EPD, this bug will not show up if the default
installation locations are accepted. So, that's also good news!

The third problem is that if the Windows user name has a space in it (which in
my experience is quite common), compilation fails. Weave uses the user name to
create a path for its "intermediate" and "compiled" files. When the compilation
command is issued, the path with the space in it is also not quoted. Presumably
that is another error in distutils (or numpy.distutils)? Unfortunately I wasn't
able to pinpoint what function is failing to quote strings properly, because I
couldn't figure out the chain that leads to the compiler being called. However,
I can avoid the problem by removing spaces from the user name in weave itself
(catalog.py):

def whoami():
   """return a string identifying the user."""
   return (os.environ.get("USER") or os.environ.get("USERNAME") or
"unknown").replace(" ","")

(where I have added .replace(" ","") to the existing code).
<<<

>>>
As well as that, I have another problem on another computer (which I
don't have access to right now) where I have a user name with a space in
it, and that causes several problems (although I think these problems
are the fault of distutils and nothing to do with scipy). I found a way
to work around that though by changing the environment variable for
USERNAME from "Dan Goodman" to "DanGoo~1".
<<<

>>>
Unfortunately, it is still often necessary to patch weave to get it to work on Windows. If a Windows user name includes a space, Weave tries to use paths with spaces for its "intermediate" and "compiled" files. I have no idea if this is a problem with distutils (not quoting strings correctly), or if Weave itself should do something different, but I myself hack weave installations on Windows.
<<<
msg114868 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-25 01:18
I tried using a space in a directory rather than in the file name and it worked. Thorney, can you repost your example as plain text files (separate source files or a diff against an empty dir) or at worst in an archive format readable by free software? Thank you.

(Don’t remember why I changed component, reverting.)
msg114870 - (view) Author: Brian Thorne (Thorney) Date: 2010-08-25 01:30
Sorry, here is the example as a diff against an empty directory. 

(I'm 100% sure the Ubuntu archiving program that I just used is free software though...)
msg114872 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-25 01:47
I can’t open the archive with unrar-free, and can’t apply the diff with patch (tried context mode and ed mode). Would you mind making it a unified diff this time?
msg114875 - (view) Author: Brian Thorne (Thorney) Date: 2010-08-25 02:01
Hopefully this works! Otherwise would a tar.gz or a zip archive be ok?
msg114877 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-08-25 02:14
Thanks, this time it worked. I had to remove the example.i source file in setup.py since I don’t have swig installed, but apart from that, compiling “ex ample.c” or “ex ample/ex ample.c” worked.

Next steps:
- Write something similar to your example (can’t just copy it since Python is not under GPL) as a unit test for distutils; I’ll do it if nobody does it shortly.
- Confirm that the bug is present on windows and hunt for it (paths translation? argument quoting?)
msg147768 - (view) Author: (ipatrol) Date: 2011-11-16 14:02
Ok, I found a similar problem with MMTK. I am currently altering my distutils package to add a function called nt_quote_dir that adds quotes to paths with spaces and then applies it to each path if the platform is win32. When I'm done I will submit a diff after confirming that it works.
msg147865 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-11-18 12:41
> Ok, I found a similar problem with MMTK.
I don’t know what that is.

> I am currently altering my distutils package to add a function called nt_quote_dir
> that adds quotes to paths with spaces and then applies it to each path if the platform
> is win32.
You shouldn’t have to do that; there’s already a similar function in distutils.spawn.

> When I'm done I will submit a diff after confirming that it works.
What is needed is a diff for distutils 2.7 or 3.2 adding a test with a file containing a space.  I can’t do anything with diffs for other projects.
msg160151 - (view) Author: fishdude (fishdude) Date: 2012-05-07 15:27
i have to report that this issue also happens on Linux and Mac as well.
no matter wether python 2.6 or 2.7 is used.
msg160156 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-07 16:07
Thanks for the confirmation.  Note that the versions field in this bug tracker indicates the versions that will get fixed, not all versions with the bug; 2.6 only gets security fixes now.

Are you interested in updating the patch with a test (see my previous messages)?
msg167484 - (view) Author: Brian Thorne (Thorney) Date: 2012-08-05 10:58
I've put together a patch adding the test requested. There is no problem on my Ubuntu machine with python 3.3.
There is a comment in the file saying Don't load the xx module more than once, I am unsure whether my patch (using a renamed c file) violates this?


One extremely minor nitpick that I'll mention is inconsistency with spaces in the module name. One can create a python file "my file.py" and can import it with __import__("my file"). I couldn't do the same for a C extension.

Building a C extension with distutils:

  module_name = 'ex ample'
  example_ext = Extension(module_name, [target_c])
  dist = Distribution({'name': module_name, 'ext_modules': [example_ext]})

Doesn't work due to the import mechanism looks for a PyInit function with the new module name (including a space):

  ImportError: dynamic module does not define init function (PyInit_ex ample)

I don't think this is worth making a ticket over.
msg212662 - (view) Author: Alexandru Gheorghe (drunax) Date: 2014-03-03 19:45
I am having the same issue on Debian 7 x86-64 (3.2.0-4-amd64)
Python 2.7.3 (default, Jan  2 2013, 13:56:14) // [GCC 4.7.2] on linux2

Not noticing this bug I've opened a duplicate, for which the interesting comment can be found here: http://bugs.python.org/issue20824#msg212548

Please let me know if you need any files and I will try hard to provide some in my free time (also unittest from what I've read in the comments posted here).
msg212872 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-07 10:37
Thanks Brian, let’s try and get this fixed.

> I've put together a patch adding the test requested. There is no problem on
> my Ubuntu machine with python 3.3.

Are you saying the test does not reproduce the bug discussed here?

> There is a comment in the file saying Don't load the xx module more than
> once, I am unsure whether my patch (using a renamed c file) violates this?

Hm I’m not quite sure if it’s enough that the extensions use different file names, or if they should also have different names inside the code.  Existing tests already create and import xx multiple times though…

> One can create a python file "my file.py" and can import it with
> __import__("my file"). I couldn't do the same for a C extension.

One can’t do “import my file” though, so I would sweep this under the rug as an obscure corner case :‑)
History
Date User Action Args
2014-03-07 10:37:50eric.araujosetmessages: + msg212872
2014-03-03 19:45:26drunaxsetnosy: + drunax
messages: + msg212662
2014-03-02 20:07:18ned.deilylinkissue20824 superseder
2012-08-05 10:58:46Thorneysetfiles: + distutils_space_test.patch

messages: + msg167484
2012-05-07 16:07:47eric.araujosetversions: + 3rd party, - Python 2.6
nosy: + alexis

messages: + msg160156

assignee: tarek -> eric.araujo
components: + Distutils2, - Extension Modules, Macintosh, Windows
2012-05-07 15:27:05fishdudesetversions: + Python 2.6
nosy: + fishdude

messages: + msg160151

components: + Macintosh
2011-11-18 12:41:10eric.araujosetmessages: + msg147865
versions: + Python 3.3, - Python 3.1
2011-11-16 14:02:29ipatrolsetnosy: + ipatrol
messages: + msg147768
components: + Extension Modules, Windows
2010-08-25 02:14:22eric.araujosetmessages: + msg114877
2010-08-25 02:01:32Thorneysetfiles: + 4508example.diff

messages: + msg114875
2010-08-25 02:00:11Thorneysetfiles: - example.diff
2010-08-25 01:47:56eric.araujosetmessages: + msg114872
2010-08-25 01:30:56Thorneysetfiles: + example.diff

messages: + msg114870
2010-08-25 01:18:57eric.araujosetversions: - Python 2.6, Python 2.5
nosy: loewis, ceball, tarek, eric.araujo, Thorney
messages: + msg114868

components: + Distutils, - Distutils2
stage: test needed
2010-08-25 01:04:35Thorneysetmessages: + msg114867
2010-05-15 23:56:48eric.araujosetmessages: - msg105840
2010-05-15 23:56:30eric.araujosetassignee: sjoerd -> tarek

messages: + msg105840
nosy: - sjoerd
2010-05-15 23:55:26eric.araujosetassignee: tarek -> sjoerd
versions: + Python 2.5, Python 3.2, - Python 3.0
nosy: + sjoerd, eric.araujo
components: + Distutils2, - Distutils
2009-03-18 13:57:09ceballsetnosy: + ceball
messages: + msg83748
2009-01-18 00:45:37tareksetmessages: + msg80066
2009-01-12 01:20:39Thorneysetfiles: + python_extension_bug.rar
messages: + msg79641
2008-12-31 16:46:30tareksetpriority: normal
assignee: tarek
messages: + msg78632
nosy: + tarek
2008-12-10 08:54:57loewissetnosy: + loewis
messages: + msg77510
versions: - Python 2.5, Python 2.4, Python 2.3, Python 2.2.3, Python 2.2.2, Python 2.2.1, Python 2.2, Python 2.1.2, Python 2.1.1, Python 2.5.3
2008-12-03 21:58:56Thorneycreate