when installing python for windows and running it from a msys or cygwin
shell, python does not notice that the path separator is backslash "/"
instead of forward slash "\".

can this be configured somehow, so the outputs are done like the current
shell accepts it? like checking in
http://docs.python.org/library/os.path.html what the parent process accepts?

I'm not sure I understand the quesiton.  The cygwin path separator is
forward slash, isn't it?  Beyond that, I'm not clear on what behavior
you think is incorrect.  What "outputs"?

if one installes python for windows with the provided installer, and 
then run this python from mingw/msys or cygwin, python prints 
backslash as path separator instead of forward slash.

it would be nice if python would notice that it was started out of 
bash and this determines the path separator vs "output", e.g. print to 
the console.

That would mean that python's notion of which OS it was running on
(windows or cygwin) would have to change depending on which shell it was
lanched from. This affects far more than the path seperator, and as far
as I know is not practical with the current python design (you are
welcome to try to work up a patch, though).

If you want cygwin behavior from python, run the cygwin python.

to give an example case, running mercurial, which we do for a couple 
of years now with success. one install, starting it either from cmd, 
or mingw/msys bash:
$ hg status
M src\com\file.txt
$ hg co -m "different path now" src/com/file.txt

apart from the backslash in the printed paths, we are very happy on 
how neatly python handles this case. 

it is running on windows, using the standard libraries, .. therefor 
everything else is really "windows". it would be quite an exceptional 
case if anything else would be affected. could you come up with an 
example which you were thinking on?

if you point us to some location in the code which would be best to 
start reading i'd be thankful.

So is this a cosmetic issue or a functional issue?  Either way, it is a
feature request and I've updated the issue to reflect that.

If you want to look at the code, ntpath.py is probably the relevant module.

Rupert, your original posts mislabels slash '/' and backslach '\' as the
opposite.  I have no idea what change you are requesting.  Please post a
minimal (line or two) code example with actual output and desired output.

I do not know if changing os.sep changes output.  I suspect that what is
printed may be controlled somewhat by the OS.

I'm not sure Python *can* change its behaviour in this case. How would
it know which shell (cmd or msys/cygwin) it was launched from?

Also, even if it could figure that out, how would it know whether a
particular filename "stringification" with os.path.join() was intended
for display to the user or to be passed to a Windows API?

the parent process would be "sh.exe" in the msys case, contrary to the 
windows standard cmd.exe, explorer.exe, system, system idle 
process, ...

an example is the mercurial "status" command, see 
http://selenic.com/repo/index.cgi/hg/file/8bf6eb68ddaf/mercurial/comman
ds.py#l2752

which uses 
http://selenic.com/repo/index.cgi/hg/file/8bf6eb68ddaf/mercurial/util.p
y#l210 (pathto, ), and normpath = os.path.normcase(path).

which does at the end:
 sys.stdout.write(str(a))

when string the string goes out on stdout or stderr it is not known 
any more it was a file path. hmm.

> So is this a cosmetic issue or a functional issue? 

It's a cosmetic issue.

> Also, even if it could figure that out, how would it know whether
> a particular filename "stringification" with os.path.join() was 
> intended for display to the user or to be passed to a Windows API?

The Windows API's (at least every one I've ever called) take either
slashes or backslashes, so it wouldn't matter.

I'm +0 on this request, if we could reliably figure out which separator
we wanted to use based on the shell (or maybe an environment variable).
It would be one more way to let me forget I'm using Windows instead of Unix.

My use case is mostly copying path's that have been print()'d from
within Python, then pasting them into a cygwin bash shell. All of the
backslashes need to be manually escaped (or the whole string quoted). Of
course this doesn't help with other characters that also need escaping
(like spaces). That and the fragile nature of the "which shell am I
running" check are why I'm +0.

Eric Smith wrote:
> Eric Smith <eric@trueblade.com> added the comment:
> 
>> So is this a cosmetic issue or a functional issue? 
> 
> It's a cosmetic issue.
> 
>> Also, even if it could figure that out, how would it know whether
>> a particular filename "stringification" with os.path.join() was 
>> intended for display to the user or to be passed to a Windows API?
> 
> The Windows API's (at least every one I've ever called) take either
> slashes or backslashes, so it wouldn't matter.

Just for information's sake, the shell APIs usually only accept backslashes.

TJG

Tim Golden wrote:
> Just for information's sake, the shell APIs usually only accept backslashes.

That's good to know. Do you have any specific examples?

CreateFile and the like, which is where my experience is, take either. 
90% of my Windows Python programs use slashes exclusively. About the 
only time I get a backslash is when using os.path.join(), and then I end 
up with mixed slashes and backslashes. Windows handles these mixed paths 
correctly, as far as I can tell.

Can this issue to confined to the small round filing cabinet, mainly on the grounds that I find some of the wording undecipherable?

I say close it. Any "shell detection" is likely to be fragile, and any changes are likely to break something. It's not worth the risk.

The resolution should be 'This is a gawd-awful mess created by Microsoft 30 years ago and we already do the best we sensibly can to deal with it' ;-) (and sensible people know it would be a mess from the first day).

To put is another way, the answer to the original question 'can this be configured somehow' seems to be 'not obviously'.

In my opinion, the decision to cross your arms and pretend there is no issue is unprofessional.

Summary of your arguments:
- It's Microsoft's fault.
- Windows accepts backslashes anyway.
- Shell detection is difficult.
- It's a complex issue.

None of this is relevant to addressing the issue ThurnerRupert was facing. He asked to be able to CONFIGURE the path seperator. This needs not involve shell detection (which is indeed brittle). This could be handled simply by the user setting 'os.sep' and having os.path.join() and friends use that.

I'm using SCons on MinGW and the build is failing due to the messed-up paths.

<rant>
I'm less than thrilled about learning that this won't get fixed. I feel that one of the important strength of Python is that it "just works" on many platforms and this amply justifies this issue not being treated with such frivolity.
</rant>

How can you rant about something "won't get fixed" when it's an enhancement request?  If it's that important and/or so simple the OP or anyone else for that matter can provide a patch, including tests and doc changes.  Given that the issue is nearly 4 1/2 years old I'm not holding my breath.

I think both of you could stand to turn down the rhetoric :)

Laurent: we're not saying there's no issue, we're saying we don't see an acceptable fix for the issue.

So the bottom line is that for this situation to improve someone is going to have to go through the hassle of figuring out *how* to make it better (working code, not just words, I think), and convincing people that it is a good idea.  The better the solution is, the easier it will be to get it committed.  I don't think a "stop gap" kind of solution will be good enough (such as making the global os.sep value settable...I don't think that kind of global state change will fly, though I could be wrong).

There are other people interested in SCons/mingw support, and this could conceivably be a configure time option for such support, but *that* is a whole other area of discussion.  For that to get off the ground, there has to be someone willing to commit to supporting it (more than one, someone, I suspect) who can work well enough with the community to eventually become part of the core team.  (There are patches in this tracker relevant to that, and at least one someone working on making them better, but admit I haven't been following that activity and don't know if any of those patches have gotten reviewed yet.)

There's also the fact that *Cygwin's* Python will behave like a *nix Python
and use backslashes. It's only the Windows Python that follows Windows
platform conventions.

So there's already a reasonable workaround in the cygwin case: use the
version which relies on the POSIX compatibility layer, not the Windows
native one.

As David pointed out, unless/until someone that is affected by the issue
can come up with a coherent design proposal for what the behaviour *should*
be that is a better solution than "just use the Cygwin Python instead if
you actually want POSIX behaviour", this isn't going to progress.

david, you mentioned working code. i am the opposite of an expert in the source code of python, but i thought i would be able to at least find where the code is for sys.stdout.write and sys.stderr.write, where i thought \ should be replaced by / when running in cygwin or msys/mingw. embarrassing for me, i did not find it.

You certainly can't replace *all* occurrences of backslash characters with forward slashes when running in a cygwin or msys shell anyway. Backslashes have many uses besides being (annoyingly) Windows path separators.

Agreed with Nick. I think it's clear than any change to the behavior will have to take place inside os.path. I just don't know what that change would be.

As noted above, it's still not clear the change *can* be implemented at the standard library level - we don't have the context to determine if a Windows native path or a POSIX path is more appropriate when a Windows build is being used in a POSIX environment.

From the interpreter's point of view, the launch environment makes no difference, the only thing that matters is the target platform for the build. If they don't match, you're going to get weird glitches like this.

First, I apologize for my overly harsh tone. A dismissive comment above struck a nerve but that doesn't excuse it.


@Lawrence: from my perspective it is a bug: the Python interpreter doesn't handle paths correctly on MinGW as I'd expect.


As Nick said, the ideal scenario would be to come up with a solution that actually fixes the problem on MinGW. I realize that very few people use MinGW and are impacted by this, so it seems like much effort for little gain. Nevertheless, perhaps the issue could be left open until someone decides to bite the bullet and fix it properly? I'm the one complaining and it's fair that I spend the effort to fix it, but like everyone I have limited time on my hands, and I can't do this now.

FWIW I see three potential solutions. I'm not involved in Python development and I don't know how applicable they are.

1) Modifying os.sep as described above.

Pros: it behaves as expected from the user point-of-view. It makes sense that the separator in this variable is used internally by the module for manipulating the paths. It may be straightforward to implement by making this a property and propagating the change to the Python code and/or interpreter code whenever it is set.

Cons: AFAICT there aren't obvious reasons why someone would modify os.sep currently, so it *should* be fine for backward compatibility for most users, but there is no guarantee.

2) Adding the function os.set_sep(separator). 

Like 1), but it's less transparent to the user, who might set os.sep naively like I did and be surprised that it doesn't work. I think that eliminates all potential backward compatibility issues though.

3) Detect MinGW.

It's brittle but it makes Python behaves as expected for the user -- who might still want/depend on the current behavior however? I'm not a fan of this but it's doable. If the MinGW detection fails then the behavior falls back to the standard Windows behavior. The reverse (assuming MinGW incorrectly) is way more problematic, but I think that the odds of that happening are very low. So essentially if the detection fails then only the MinGW users are impacted.


Nick, correct me if I'm wrong but the target platform for MinGW is mostly Windows itself, aside of this particular issue? The rest of the standard library seems to be working fine on MinGW as it is.


Thank you.

We can't change os.sep globally for the process because the Windows APIs
don't reliably accept forward slashes. Changing it when the Windows binary
is run through msys would thus mean potentially breaking currently working
applications.

Using the Cygwin Python instead is safer, since all the affected OS API
calls will be trapped by Cygwin's POSIX API implementation and translated
appropriately for Windows.

Terry's description when first closing this issue remains accurate: MS
created this mess through an atrociously poor choice of path separator
decades ago, and until *all* their APIs natively accept forward slashes as
path separators (which seems unlikely to ever happen), the only backwards
compatible answer currently appears to be to use a full POSIX interface
layer like Cygwin.

However, coming up with a way to detect that msys is in use, and publishing
that information for use by applications like Mercurial, or creating a
catalogue of which Windows APIs still don't accept forward slashes as path
separators could help the issue progress.

There's just no magic fix we can apply at the interpreter or standard
library level to make the problem go away - it's *precisely* the kind of
problem that exists in the space between the "POSIX shell on Windows"
provided by msys and the full POSIX API provided by Cygwin.

I agree with the "no magic bullet" point and that Microsoft is fully to blame for the situation.


About the catalogue of Windows APIs that accept slashes. I've read in various places that the only problematic case is the legacy shell API. The power shell seems to accept forward slashes.

http://stackoverflow.com/questions/18464038/is-it-possible-to-make-vim-use-forward-slashes-on-windows

http://stackoverflow.com/questions/10523708/why-does-the-cmd-exe-shell-on-windows-fail-with-paths-using-a-forward-slash


Is it reasonable to believe that most Python programs don't care about the legacy shell API? I assume that the Python library uses CreateProcess() under the hood without touching cmd.exe. Some odd Python programs may invoke cmd.exe explicitly but hopefully that is a rare case.

If those assumptions hold, then changing os.sep *from the user script* would not break backward compatibility and allow Python to work as expected on MinGW. Arguably it's a simpler solution than requiring Python projects like Mercurial to tackle that issue themselves. Even if it's not the perfect solution, it's a step-up from "Non-Cygwin Python corrupts paths on MinGW and you can't make it stop".

subprocess has a 'shell=True' option, although its use is discouraged unless really necessary. To pursue this, I suggest running the test suite on Windows (including xp) with os.sep changed.

On 14/11/2013 00:21, Laurent Birtz wrote:
> Is it reasonable to believe that most Python programs don't care
> about the legacy shell API? 

No more than it is to believe that most Python programs don't care about
MSys or Cygwin ;)

For information, cmd.exe will happily run, eg, "c:/windows/notepad.exe".
The problem is that command-line programs built in to cmd.exe (such as
"type" and "dir") expect the slash to prefix a command-line switch. And
they don't seem to have particularly good escape- or quote-handling.
External command-line tools, like "xcopy", will happily act on
forward-slash pathnames as long as they're double-quoted.

Now certain of the Windows shell API (and here "shell" means: the visual
elements, *not* the cmd/PS command shell) will only operate on
backslash-separated paths. For example: SHILCreateFromPath:

http://msdn.microsoft.com/en-us/library/windows/desktop/bb762210(v=vs.85).aspx

I'm not exactly how much that affects the current discussion, but my
point is that there *are* places in Windows where the backslash is the
only acceptable separator.

Changing os.sep globally is real no-no. it has the potential to break a
*lot* of code and we're in no position to assess how it's been used.
Applying a change conditionally might be acceptable, but somone would
have to work out how we knew what environment we were in.

TJG

Thank you for the clarifications on the usage of cmd.exe and shell=True. I tend to forget that the shell support in subprocess isn't just about UNIX.

The point of running msys is getting programs to work as if they were in a limited UNIX environment without the emulation layer provided by Cygwin. It's nice if programs unaware of msys work out-of-the-box, but for use cases like mine what is needed is the possibility to write a program that works correctly on msys with a minimum of fuss.

Given that, conditionally setting os.dep on program entry and wrapping the calls to cmd.exe manually seems to fit the bill. I haven't tested if SCons plays nice with that change, so I may be in for a disappointment. Even if it doesn't work right, it would still be useful for me to temporarily change os.dep and join my paths correctly until I return control to SCons.

On my system, print os.environ["MSYSTEM"] yields MINGW32. That's all I need for detecting msys currently. If that ever break I'll find another method.