classification
Title: [Windows] distutils.util.byte_compile() fails indirect byte compiling with non-ASCII full-path
Type: behavior Stage: resolved
Components: Distutils, Unicode, Windows Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Suzumizaki, aldwinaldwin, dstufft, eric.araujo, ezio.melotti, methane, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2019-07-12 02:28 by Suzumizaki, last changed 2021-02-03 18:10 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
batch.diff Suzumizaki, 2019-07-12 02:28 Patch against distutils.util.py
my_test_case.py Suzumizaki, 2019-07-12 02:31
Messages (8)
msg347718 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 02:28
`distutils.util.byte_compile` fails _indirect_ byte compiling when the
full-path of the .py file contains non-ASCII characters.

Because there is the project (cx_Freeze) which directly uses
`distutils.util.byte_compile`, the problem would happen while installing it
under the non-ASCII path.

Fortunately, it is easy to fix.
 
Ofcourse, the files included in external libraries should be named ASCII only,
but `distutils.util.byte_compile` makes and uses full path of them. I learned
many of libraries uses setuptools instead of distutils, but please consider
the following situations:

1) The case that venv creates the virtual environment under or with
   non-ASCII named folder.
2) And special cases of 1), on Windows, the name of end-users OFTEN be
   composed with non-ASCII characters, this means including theirs
   Documents, Desktop, or Downloads etc.

The more they are not professional,they make easily the situation above.
msg347719 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 02:31
Here's test code.

Note: I marked just Python 3.8 and 3.7 I tested, but maybe all versions.
msg347721 - (view) Author: Aldwin Pollefeyt (aldwinaldwin) * Date: 2019-07-12 04:30
All 'The byte compiling test seems succeeded.' when i tested your script on ubuntu 18.04 on latest 3.7.4rc2+, 3.8.0b1+ and 3.9.0a0.

What are the exact python versions and the OS version you see the issue?
msg347723 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 05:24
Terribly Sorry! I forgot to write this is only Windows Issue! Thanks!
msg347724 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 05:35
... and please check "batch.diff" (Oops, that's typo of 'patch'). I just added "encoding" parameter to calling open() function, and added the temp file to 'utf-8' code declaration. This bug could happen where the default encoding of open() function is not utf-8(may not only Windows).
msg347732 - (view) Author: Aldwin Pollefeyt (aldwinaldwin) * Date: 2019-07-12 09:19
FYI, PEP597 is created to avoid these issues (i think), target Python 3.9.

https://www.python.org/dev/peps/pep-0597/

https://discuss.python.org/t/use-utf-8-as-default-text-file-encoding/1785
msg347749 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 15:41
Thanks, but Those cannot resolve this issue.

This is install time problem. In other words, whatever the default encoding is, even some users change it, `distutils.util.byte_compile` must always succeed. 

There's no relation between using non-ASCII path and the default encoding. 
No logically reason to fail.
msg386287 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-02-03 18:10
Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools
History
Date User Action Args
2021-02-03 18:10:55steve.dowersetstatus: open -> closed
resolution: out of date
messages: + msg386287

stage: resolved
2019-07-14 09:17:26vstinnersetnosy: + methane
2019-07-14 09:17:13vstinnersettitle: `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path. -> [Windows] distutils.util.byte_compile() fails indirect byte compiling with non-ASCII full-path
2019-07-12 15:41:42Suzumizakisetmessages: + msg347749
2019-07-12 09:19:00aldwinaldwinsetmessages: + msg347732
2019-07-12 05:35:10Suzumizakisetmessages: + msg347724
2019-07-12 05:24:09Suzumizakisetnosy: + vstinner, paul.moore, tim.golden, ezio.melotti, zach.ware, steve.dower
messages: + msg347723
components: + Unicode, Windows
2019-07-12 04:30:29aldwinaldwinsetnosy: + aldwinaldwin
messages: + msg347721
2019-07-12 02:31:26Suzumizakisetfiles: + my_test_case.py

messages: + msg347719
2019-07-12 02:28:51Suzumizakicreate