Please provide more details. How to reproduce your issue? What you got, and what you expect to get?

Seems the code just before lines modified by your PR are purposed to solve this issue. Why it doesn't work?

Hello,

Yes, it does seem odd that that code does not work. On my Windows machine (WIndows 7, 64 bits, running 32-bit Python) I checked and it seems that the code in the if block immediately preceding my PR does not run at all, whereby the error.

For a reproducible example, my Taqdir package, mostly consisting of unicode packages and modules, runs into this issue (and installs successfully after my proposed fix here combined with a separate PR in pip). Perhaps the most easily accessible example would be the Appveyor build (https://ci.appveyor.com/project/julienmalard/Tinamit) for my TInamit project, which has Taqdir as a dependency.

Thanks!

-Julien Malard

દ્વારા: Serhiy Storchaka <report@bugs.python.org>
મોકલ્યું: 18 ઑગસ્ટ 2018 06:32
પ્રતિ: Julien Malard
વિષય: [bpo-34421] Cannot install package with unicode module names on Windows

New submission from Serhiy Storchaka <storchaka+cpython@gmail.com>:

Please provide more details. How to reproduce your issue? What you got, and what you expect to get?

Seems the code just before lines modified by your PR are purposed to solve this issue. Why it doesn't work?

----------
nosy: +serhiy.storchaka

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue34421\>

New changeset 0afada1 by Éric Araujo (Julien Malard) in branch 'master':
bpo-34421 avoid unicode error in distutils logging (GH-8799)
0afada1

New changeset 3b36642 by Miss Islington (bot) in branch '3.6':
bpo-34421 avoid unicode error in distutils logging (GH-8799)
3b36642

New changeset 77b92b1 by Miss Islington (bot) in branch '3.7':
bpo-34421 avoid unicode error in distutils logging (GH-8799)
77b92b1

I would prefer to use the backslashreplace error handler rather of the unicode-escape codec. Just as few lines above, but with ASCII encoding.

    msg = msg.encode('ascii', 'backslashreplace').decode('ascii')

It is still not clear to me why the current code purposed to handle this problem doesn't work in this case. We need to find the cause and fix the existing solution.

The existing re-code solution is being triggered, as the errors in this case is 'surrogateescape' with an encoding of 'cp1252'.

Here, pip is using subprocess.Popen() to have Python run setup.py. During execution, a filename, 'taqdir\\\u0634\u0645\u0627\u0631.py', which has characters not encodable in cp1252.

I think that here, Python is not configuring its stdin/stdout/stderr streams correctly when run as a subprocess connected to pipes. Or, at least, subprocess.Popen() isn't passing the right (or enough) information to Python to get itself configured.

There should ultimately be a way to have Python (in a subprocess, on Windows) pass through Unicode untouched to its calling process. I suppose it would mean setting the PYTHONIOENCODING envvar when using subprocess.

After all that, it seems that:

1. pip needs to be changed to support calling Python subprocesses to enable lossless unicode transmission,
2. change the errors check in distutils.log to include 'surrogateescape'? (the heart of this issue)

PR 9126 makes distutils.log using "backslashreplace" instead of "unicode-escape" and simplifies the code (it is more efficient now, although the performance of logging is not critical).

"unicode-escape" escapes all non-ASCII characters, even encodable. It also escapes control characters like \t, \b, \r or \x1a (which starts control sequences for ANSI compatible terminals), this can be not desirable.

Hello,

Thanks for the insights and better fixes. Regarding (1), do you have any pointers on how or where to fix pip? I have an inprogress pull request there (pypa/pip#5712) to fix a related unicode error during installation and could perhaps combine both solutions.

Thanks!

-Julien

For pip, in call_subprocess() (given here in rough pseudo-code)

is_python = (cmd[0] == sys.executable)
kwds = {}
if is_python:
    env['PYTHONIOENCODING'] = 'utf8'
    kwds['encoding'] = 'utf8'
proc = Popen(..., **kwds)
.
.
.
if stdout is not None:
    while True:
        line = proc.stdout.readline()
        # When running Python, the output is already Unicode
        if not is_python:
            line = console_to_str(line)
        if not line:
            break

Hopefully, there is enough context to figure out the exact placement.

Thanks! Will give it a try and reference this conversation here as background.

New changeset 4b860fd by Serhiy Storchaka in branch 'master':
bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126)
4b860fd

New changeset c73df53 by Serhiy Storchaka in branch '3.7':
bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126) (GH-9506)
c73df53

New changeset 0b67995 by Miss Islington (bot) in branch '3.6':
bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126) (GH-9506)
0b67995