New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot install package with unicode module names on Windows #78602
Comments
Please provide more details. How to reproduce your issue? What you got, and what you expect to get? Seems the code just before lines modified by your PR are purposed to solve this issue. Why it doesn't work? |
Hello, Yes, it does seem odd that that code does not work. On my Windows machine (WIndows 7, 64 bits, running 32-bit Python) I checked and it seems that the code in the if block immediately preceding my PR does not run at all, whereby the error. For a reproducible example, my Taqdir package, mostly consisting of unicode packages and modules, runs into this issue (and installs successfully after my proposed fix here combined with a separate PR in pip). Perhaps the most easily accessible example would be the Appveyor build (https://ci.appveyor.com/project/julienmalard/Tinamit) for my TInamit project, which has Taqdir as a dependency. Thanks! -Julien Malard દ્વારા: Serhiy Storchaka <report@bugs.python.org> New submission from Serhiy Storchaka <storchaka+cpython@gmail.com>: Please provide more details. How to reproduce your issue? What you got, and what you expect to get? Seems the code just before lines modified by your PR are purposed to solve this issue. Why it doesn't work? ---------- Python tracker <report@bugs.python.org> |
I would prefer to use the backslashreplace error handler rather of the unicode-escape codec. Just as few lines above, but with ASCII encoding. msg = msg.encode('ascii', 'backslashreplace').decode('ascii') It is still not clear to me why the current code purposed to handle this problem doesn't work in this case. We need to find the cause and fix the existing solution. |
The existing re-code solution is being triggered, as the Here, pip is using subprocess.Popen() to have Python run setup.py. During execution, a filename, 'taqdir\\\u0634\u0645\u0627\u0631.py', which has characters not encodable in cp1252. I think that here, Python is not configuring its stdin/stdout/stderr streams correctly when run as a subprocess connected to pipes. Or, at least, subprocess.Popen() isn't passing the right (or enough) information to Python to get itself configured. There should ultimately be a way to have Python (in a subprocess, on Windows) pass through Unicode untouched to its calling process. I suppose it would mean setting the PYTHONIOENCODING envvar when using subprocess. After all that, it seems that:
|
PR 9126 makes distutils.log using "backslashreplace" instead of "unicode-escape" and simplifies the code (it is more efficient now, although the performance of logging is not critical). "unicode-escape" escapes all non-ASCII characters, even encodable. It also escapes control characters like \t, \b, \r or \x1a (which starts control sequences for ANSI compatible terminals), this can be not desirable. |
Hello, Thanks for the insights and better fixes. Regarding (1), do you have any pointers on how or where to fix pip? I have an inprogress pull request there (pypa/pip#5712) to fix a related unicode error during installation and could perhaps combine both solutions. Thanks! -Julien |
For pip, in call_subprocess() (given here in rough pseudo-code) is_python = (cmd[0] == sys.executable)
kwds = {}
if is_python:
env['PYTHONIOENCODING'] = 'utf8'
kwds['encoding'] = 'utf8'
proc = Popen(..., **kwds)
.
.
.
if stdout is not None:
while True:
line = proc.stdout.readline()
# When running Python, the output is already Unicode
if not is_python:
line = console_to_str(line)
if not line:
break Hopefully, there is enough context to figure out the exact placement. |
Thanks! Will give it a try and reference this conversation here as background. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: