Issue6208
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2009-06-05 18:21 by ThurnerRupert, last changed 2022-04-11 14:56 by admin. This issue is now closed.
Messages (30) | |||
---|---|---|---|
msg88958 - (view) | Author: (ThurnerRupert) | Date: 2009-06-05 18:21 | |
when installing python for windows and running it from a msys or cygwin shell, python does not notice that the path separator is backslash "/" instead of forward slash "\". can this be configured somehow, so the outputs are done like the current shell accepts it? like checking in http://docs.python.org/library/os.path.html what the parent process accepts? |
|||
msg88961 - (view) | Author: R. David Murray (r.david.murray) * | Date: 2009-06-05 18:37 | |
I'm not sure I understand the quesiton. The cygwin path separator is forward slash, isn't it? Beyond that, I'm not clear on what behavior you think is incorrect. What "outputs"? |
|||
msg89071 - (view) | Author: (ThurnerRupert) | Date: 2009-06-08 08:52 | |
if one installes python for windows with the provided installer, and then run this python from mingw/msys or cygwin, python prints backslash as path separator instead of forward slash. it would be nice if python would notice that it was started out of bash and this determines the path separator vs "output", e.g. print to the console. |
|||
msg89082 - (view) | Author: R. David Murray (r.david.murray) * | Date: 2009-06-08 15:34 | |
That would mean that python's notion of which OS it was running on (windows or cygwin) would have to change depending on which shell it was lanched from. This affects far more than the path seperator, and as far as I know is not practical with the current python design (you are welcome to try to work up a patch, though). If you want cygwin behavior from python, run the cygwin python. |
|||
msg89087 - (view) | Author: (ThurnerRupert) | Date: 2009-06-08 16:09 | |
to give an example case, running mercurial, which we do for a couple of years now with success. one install, starting it either from cmd, or mingw/msys bash: $ hg status M src\com\file.txt $ hg co -m "different path now" src/com/file.txt apart from the backslash in the printed paths, we are very happy on how neatly python handles this case. it is running on windows, using the standard libraries, .. therefor everything else is really "windows". it would be quite an exceptional case if anything else would be affected. could you come up with an example which you were thinking on? if you point us to some location in the code which would be best to start reading i'd be thankful. |
|||
msg89096 - (view) | Author: R. David Murray (r.david.murray) * | Date: 2009-06-08 18:53 | |
So is this a cosmetic issue or a functional issue? Either way, it is a feature request and I've updated the issue to reflect that. If you want to look at the code, ntpath.py is probably the relevant module. |
|||
msg89290 - (view) | Author: Terry J. Reedy (terry.reedy) * | Date: 2009-06-12 16:46 | |
Rupert, your original posts mislabels slash '/' and backslach '\' as the opposite. I have no idea what change you are requesting. Please post a minimal (line or two) code example with actual output and desired output. I do not know if changing os.sep changes output. I suspect that what is printed may be controlled somewhat by the OS. |
|||
msg89308 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2009-06-13 01:35 | |
I'm not sure Python *can* change its behaviour in this case. How would it know which shell (cmd or msys/cygwin) it was launched from? Also, even if it could figure that out, how would it know whether a particular filename "stringification" with os.path.join() was intended for display to the user or to be passed to a Windows API? |
|||
msg89360 - (view) | Author: (ThurnerRupert) | Date: 2009-06-14 17:51 | |
the parent process would be "sh.exe" in the msys case, contrary to the windows standard cmd.exe, explorer.exe, system, system idle process, ... an example is the mercurial "status" command, see http://selenic.com/repo/index.cgi/hg/file/8bf6eb68ddaf/mercurial/comman ds.py#l2752 which uses http://selenic.com/repo/index.cgi/hg/file/8bf6eb68ddaf/mercurial/util.p y#l210 (pathto, ), and normpath = os.path.normcase(path). which does at the end: sys.stdout.write(str(a)) when string the string goes out on stdout or stderr it is not known any more it was a file path. hmm. |
|||
msg89403 - (view) | Author: Eric V. Smith (eric.smith) * | Date: 2009-06-15 14:50 | |
> So is this a cosmetic issue or a functional issue? It's a cosmetic issue. > Also, even if it could figure that out, how would it know whether > a particular filename "stringification" with os.path.join() was > intended for display to the user or to be passed to a Windows API? The Windows API's (at least every one I've ever called) take either slashes or backslashes, so it wouldn't matter. I'm +0 on this request, if we could reliably figure out which separator we wanted to use based on the shell (or maybe an environment variable). It would be one more way to let me forget I'm using Windows instead of Unix. My use case is mostly copying path's that have been print()'d from within Python, then pasting them into a cygwin bash shell. All of the backslashes need to be manually escaped (or the whole string quoted). Of course this doesn't help with other characters that also need escaping (like spaces). That and the fragile nature of the "which shell am I running" check are why I'm +0. |
|||
msg89404 - (view) | Author: Tim Golden (tim.golden) * | Date: 2009-06-15 14:54 | |
Eric Smith wrote: > Eric Smith <eric@trueblade.com> added the comment: > >> So is this a cosmetic issue or a functional issue? > > It's a cosmetic issue. > >> Also, even if it could figure that out, how would it know whether >> a particular filename "stringification" with os.path.join() was >> intended for display to the user or to be passed to a Windows API? > > The Windows API's (at least every one I've ever called) take either > slashes or backslashes, so it wouldn't matter. Just for information's sake, the shell APIs usually only accept backslashes. TJG |
|||
msg89405 - (view) | Author: Eric V. Smith (eric.smith) * | Date: 2009-06-15 15:06 | |
Tim Golden wrote: > Just for information's sake, the shell APIs usually only accept backslashes. That's good to know. Do you have any specific examples? CreateFile and the like, which is where my experience is, take either. 90% of my Windows Python programs use slashes exclusively. About the only time I get a backslash is when using os.path.join(), and then I end up with mixed slashes and backslashes. Windows handles these mixed paths correctly, as far as I can tell. |
|||
msg189166 - (view) | Author: Mark Lawrence (BreamoreBoy) * | Date: 2013-05-13 18:53 | |
Can this issue to confined to the small round filing cabinet, mainly on the grounds that I find some of the wording undecipherable? |
|||
msg189172 - (view) | Author: Eric V. Smith (eric.smith) * | Date: 2013-05-13 20:05 | |
I say close it. Any "shell detection" is likely to be fragile, and any changes are likely to break something. It's not worth the risk. |
|||
msg189176 - (view) | Author: Terry J. Reedy (terry.reedy) * | Date: 2013-05-13 20:44 | |
The resolution should be 'This is a gawd-awful mess created by Microsoft 30 years ago and we already do the best we sensibly can to deal with it' ;-) (and sensible people know it would be a mess from the first day). To put is another way, the answer to the original question 'can this be configured somehow' seems to be 'not obviously'. |
|||
msg202707 - (view) | Author: Laurent Birtz (laurent.birtz) | Date: 2013-11-12 17:34 | |
In my opinion, the decision to cross your arms and pretend there is no issue is unprofessional. Summary of your arguments: - It's Microsoft's fault. - Windows accepts backslashes anyway. - Shell detection is difficult. - It's a complex issue. None of this is relevant to addressing the issue ThurnerRupert was facing. He asked to be able to CONFIGURE the path seperator. This needs not involve shell detection (which is indeed brittle). This could be handled simply by the user setting 'os.sep' and having os.path.join() and friends use that. I'm using SCons on MinGW and the build is failing due to the messed-up paths. <rant> I'm less than thrilled about learning that this won't get fixed. I feel that one of the important strength of Python is that it "just works" on many platforms and this amply justifies this issue not being treated with such frivolity. </rant> |
|||
msg202709 - (view) | Author: Mark Lawrence (BreamoreBoy) * | Date: 2013-11-12 17:44 | |
How can you rant about something "won't get fixed" when it's an enhancement request? If it's that important and/or so simple the OP or anyone else for that matter can provide a patch, including tests and doc changes. Given that the issue is nearly 4 1/2 years old I'm not holding my breath. |
|||
msg202713 - (view) | Author: R. David Murray (r.david.murray) * | Date: 2013-11-12 19:15 | |
I think both of you could stand to turn down the rhetoric :) Laurent: we're not saying there's no issue, we're saying we don't see an acceptable fix for the issue. So the bottom line is that for this situation to improve someone is going to have to go through the hassle of figuring out *how* to make it better (working code, not just words, I think), and convincing people that it is a good idea. The better the solution is, the easier it will be to get it committed. I don't think a "stop gap" kind of solution will be good enough (such as making the global os.sep value settable...I don't think that kind of global state change will fly, though I could be wrong). There are other people interested in SCons/mingw support, and this could conceivably be a configure time option for such support, but *that* is a whole other area of discussion. For that to get off the ground, there has to be someone willing to commit to supporting it (more than one, someone, I suspect) who can work well enough with the community to eventually become part of the core team. (There are patches in this tracker relevant to that, and at least one someone working on making them better, but admit I haven't been following that activity and don't know if any of those patches have gotten reviewed yet.) |
|||
msg202721 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2013-11-12 21:10 | |
There's also the fact that *Cygwin's* Python will behave like a *nix Python and use backslashes. It's only the Windows Python that follows Windows platform conventions. So there's already a reasonable workaround in the cygwin case: use the version which relies on the POSIX compatibility layer, not the Windows native one. As David pointed out, unless/until someone that is affected by the issue can come up with a coherent design proposal for what the behaviour *should* be that is a better solution than "just use the Cygwin Python instead if you actually want POSIX behaviour", this isn't going to progress. |
|||
msg202733 - (view) | Author: (ThurnerRupert) | Date: 2013-11-13 10:14 | |
david, you mentioned working code. i am the opposite of an expert in the source code of python, but i thought i would be able to at least find where the code is for sys.stdout.write and sys.stderr.write, where i thought \ should be replaced by / when running in cygwin or msys/mingw. embarrassing for me, i did not find it. |
|||
msg202737 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2013-11-13 12:20 | |
You certainly can't replace *all* occurrences of backslash characters with forward slashes when running in a cygwin or msys shell anyway. Backslashes have many uses besides being (annoyingly) Windows path separators. |
|||
msg202739 - (view) | Author: Eric V. Smith (eric.smith) * | Date: 2013-11-13 12:34 | |
Agreed with Nick. I think it's clear than any change to the behavior will have to take place inside os.path. I just don't know what that change would be. |
|||
msg202750 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2013-11-13 13:56 | |
As noted above, it's still not clear the change *can* be implemented at the standard library level - we don't have the context to determine if a Windows native path or a POSIX path is more appropriate when a Windows build is being used in a POSIX environment. From the interpreter's point of view, the launch environment makes no difference, the only thing that matters is the target platform for the build. If they don't match, you're going to get weird glitches like this. |
|||
msg202766 - (view) | Author: Laurent Birtz (laurent.birtz) | Date: 2013-11-13 18:37 | |
First, I apologize for my overly harsh tone. A dismissive comment above struck a nerve but that doesn't excuse it. @Lawrence: from my perspective it is a bug: the Python interpreter doesn't handle paths correctly on MinGW as I'd expect. As Nick said, the ideal scenario would be to come up with a solution that actually fixes the problem on MinGW. I realize that very few people use MinGW and are impacted by this, so it seems like much effort for little gain. Nevertheless, perhaps the issue could be left open until someone decides to bite the bullet and fix it properly? I'm the one complaining and it's fair that I spend the effort to fix it, but like everyone I have limited time on my hands, and I can't do this now. FWIW I see three potential solutions. I'm not involved in Python development and I don't know how applicable they are. 1) Modifying os.sep as described above. Pros: it behaves as expected from the user point-of-view. It makes sense that the separator in this variable is used internally by the module for manipulating the paths. It may be straightforward to implement by making this a property and propagating the change to the Python code and/or interpreter code whenever it is set. Cons: AFAICT there aren't obvious reasons why someone would modify os.sep currently, so it *should* be fine for backward compatibility for most users, but there is no guarantee. 2) Adding the function os.set_sep(separator). Like 1), but it's less transparent to the user, who might set os.sep naively like I did and be surprised that it doesn't work. I think that eliminates all potential backward compatibility issues though. 3) Detect MinGW. It's brittle but it makes Python behaves as expected for the user -- who might still want/depend on the current behavior however? I'm not a fan of this but it's doable. If the MinGW detection fails then the behavior falls back to the standard Windows behavior. The reverse (assuming MinGW incorrectly) is way more problematic, but I think that the odds of that happening are very low. So essentially if the detection fails then only the MinGW users are impacted. Nick, correct me if I'm wrong but the target platform for MinGW is mostly Windows itself, aside of this particular issue? The rest of the standard library seems to be working fine on MinGW as it is. Thank you. |
|||
msg202792 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2013-11-13 22:30 | |
We can't change os.sep globally for the process because the Windows APIs don't reliably accept forward slashes. Changing it when the Windows binary is run through msys would thus mean potentially breaking currently working applications. Using the Cygwin Python instead is safer, since all the affected OS API calls will be trapped by Cygwin's POSIX API implementation and translated appropriately for Windows. Terry's description when first closing this issue remains accurate: MS created this mess through an atrociously poor choice of path separator decades ago, and until *all* their APIs natively accept forward slashes as path separators (which seems unlikely to ever happen), the only backwards compatible answer currently appears to be to use a full POSIX interface layer like Cygwin. |
|||
msg202794 - (view) | Author: Nick Coghlan (ncoghlan) * | Date: 2013-11-13 22:37 | |
However, coming up with a way to detect that msys is in use, and publishing that information for use by applications like Mercurial, or creating a catalogue of which Windows APIs still don't accept forward slashes as path separators could help the issue progress. There's just no magic fix we can apply at the interpreter or standard library level to make the problem go away - it's *precisely* the kind of problem that exists in the space between the "POSIX shell on Windows" provided by msys and the full POSIX API provided by Cygwin. |
|||
msg202805 - (view) | Author: Laurent Birtz (laurent.birtz) | Date: 2013-11-14 00:21 | |
I agree with the "no magic bullet" point and that Microsoft is fully to blame for the situation. About the catalogue of Windows APIs that accept slashes. I've read in various places that the only problematic case is the legacy shell API. The power shell seems to accept forward slashes. http://stackoverflow.com/questions/18464038/is-it-possible-to-make-vim-use-forward-slashes-on-windows http://stackoverflow.com/questions/10523708/why-does-the-cmd-exe-shell-on-windows-fail-with-paths-using-a-forward-slash Is it reasonable to believe that most Python programs don't care about the legacy shell API? I assume that the Python library uses CreateProcess() under the hood without touching cmd.exe. Some odd Python programs may invoke cmd.exe explicitly but hopefully that is a rare case. If those assumptions hold, then changing os.sep *from the user script* would not break backward compatibility and allow Python to work as expected on MinGW. Arguably it's a simpler solution than requiring Python projects like Mercurial to tackle that issue themselves. Even if it's not the perfect solution, it's a step-up from "Non-Cygwin Python corrupts paths on MinGW and you can't make it stop". |
|||
msg202814 - (view) | Author: Terry J. Reedy (terry.reedy) * | Date: 2013-11-14 01:20 | |
subprocess has a 'shell=True' option, although its use is discouraged unless really necessary. To pursue this, I suggest running the test suite on Windows (including xp) with os.sep changed. |
|||
msg202822 - (view) | Author: Tim Golden (tim.golden) * | Date: 2013-11-14 09:27 | |
On 14/11/2013 00:21, Laurent Birtz wrote: > Is it reasonable to believe that most Python programs don't care > about the legacy shell API? No more than it is to believe that most Python programs don't care about MSys or Cygwin ;) For information, cmd.exe will happily run, eg, "c:/windows/notepad.exe". The problem is that command-line programs built in to cmd.exe (such as "type" and "dir") expect the slash to prefix a command-line switch. And they don't seem to have particularly good escape- or quote-handling. External command-line tools, like "xcopy", will happily act on forward-slash pathnames as long as they're double-quoted. Now certain of the Windows shell API (and here "shell" means: the visual elements, *not* the cmd/PS command shell) will only operate on backslash-separated paths. For example: SHILCreateFromPath: http://msdn.microsoft.com/en-us/library/windows/desktop/bb762210(v=vs.85).aspx I'm not exactly how much that affects the current discussion, but my point is that there *are* places in Windows where the backslash is the only acceptable separator. Changing os.sep globally is real no-no. it has the potential to break a *lot* of code and we're in no position to assess how it's been used. Applying a change conditionally might be acceptable, but somone would have to work out how we knew what environment we were in. TJG |
|||
msg202858 - (view) | Author: Laurent Birtz (laurent.birtz) | Date: 2013-11-14 16:46 | |
Thank you for the clarifications on the usage of cmd.exe and shell=True. I tend to forget that the shell support in subprocess isn't just about UNIX. The point of running msys is getting programs to work as if they were in a limited UNIX environment without the emulation layer provided by Cygwin. It's nice if programs unaware of msys work out-of-the-box, but for use cases like mine what is needed is the possibility to write a program that works correctly on msys with a minimum of fuss. Given that, conditionally setting os.dep on program entry and wrapping the calls to cmd.exe manually seems to fit the bill. I haven't tested if SCons plays nice with that change, so I may be in for a disappointment. Even if it doesn't work right, it would still be useful for me to temporarily change os.dep and join my paths correctly until I return control to SCons. On my system, print os.environ["MSYSTEM"] yields MINGW32. That's all I need for detecting msys currently. If that ever break I'll find another method. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:56:49 | admin | set | github: 50457 |
2013-11-14 16:46:02 | laurent.birtz | set | messages: + msg202858 |
2013-11-14 09:27:28 | tim.golden | set | messages: + msg202822 |
2013-11-14 01:20:33 | terry.reedy | set | messages: + msg202814 |
2013-11-14 00:21:09 | laurent.birtz | set | messages: + msg202805 |
2013-11-13 22:37:40 | ncoghlan | set | messages: + msg202794 |
2013-11-13 22:30:37 | ncoghlan | set | messages: + msg202792 |
2013-11-13 18:37:26 | laurent.birtz | set | messages: + msg202766 |
2013-11-13 13:56:48 | ncoghlan | set | messages: + msg202750 |
2013-11-13 12:34:47 | eric.smith | set | messages: + msg202739 |
2013-11-13 12:20:54 | ncoghlan | set | messages: + msg202737 |
2013-11-13 10:14:43 | ThurnerRupert | set | messages: + msg202733 |
2013-11-12 21:10:03 | ncoghlan | set | messages:
+ msg202721 title: path separator output ignores shell's path separator: / instead of \ -> path separator output ignores shell's path separator: / instead of \ |
2013-11-12 19:15:03 | r.david.murray | set | messages: + msg202713 |
2013-11-12 17:44:59 | BreamoreBoy | set | messages: + msg202709 |
2013-11-12 17:34:14 | laurent.birtz | set | nosy:
+ laurent.birtz messages: + msg202707 |
2013-05-13 20:44:20 | terry.reedy | set | status: open -> closed resolution: works for me messages: + msg189176 |
2013-05-13 20:05:00 | eric.smith | set | messages: + msg189172 |
2013-05-13 18:53:13 | BreamoreBoy | set | nosy:
+ BreamoreBoy messages: + msg189166 |
2009-06-15 15:06:59 | eric.smith | set | messages: + msg89405 |
2009-06-15 14:54:27 | tim.golden | set | nosy:
+ tim.golden title: path separator output ignores shell's path separator: / instead of \ -> path separator output ignores shell's path separator: / instead of \ messages: + msg89404 |
2009-06-15 14:50:29 | eric.smith | set | nosy:
+ eric.smith messages: + msg89403 |
2009-06-14 17:51:42 | ThurnerRupert | set | messages: + msg89360 |
2009-06-13 01:35:38 | ncoghlan | set | nosy:
+ ncoghlan messages: + msg89308 |
2009-06-12 16:46:39 | terry.reedy | set | nosy:
+ terry.reedy messages: + msg89290 versions: - Python 2.6, Python 3.1 |
2009-06-08 18:53:15 | r.david.murray | set | priority: normal -> low versions: + Python 2.7, Python 3.2 type: behavior -> enhancement messages: + msg89096 resolution: not a bug -> (no value) stage: resolved -> |
2009-06-08 16:09:53 | ThurnerRupert | set | messages: + msg89087 |
2009-06-08 15:34:26 | r.david.murray | set | resolution: not a bug messages: + msg89082 stage: resolved |
2009-06-08 08:52:26 | ThurnerRupert | set | messages: + msg89071 |
2009-06-05 18:37:09 | r.david.murray | set | priority: normal nosy: + r.david.murray messages: + msg88961 components: + Installation, - IO type: behavior |
2009-06-05 18:22:19 | ThurnerRupert | set | components: + Windows, IO |
2009-06-05 18:21:49 | ThurnerRupert | create |