classification
Title: [Windows] distutils.util.byte_compile() fails indirect byte compiling with non-ASCII full-path
Type: behavior Stage:
Components: Distutils, Unicode, Windows Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Suzumizaki, aldwinaldwin, dstufft, eric.araujo, ezio.melotti, inada.naoki, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2019-07-12 02:28 by Suzumizaki, last changed 2019-07-14 09:17 by vstinner.

Files
File name Uploaded Description Edit
batch.diff Suzumizaki, 2019-07-12 02:28 Patch against distutils.util.py
my_test_case.py Suzumizaki, 2019-07-12 02:31
Messages (7)
msg347718 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 02:28
`distutils.util.byte_compile` fails _indirect_ byte compiling when the
full-path of the .py file contains non-ASCII characters.

Because there is the project (cx_Freeze) which directly uses
`distutils.util.byte_compile`, the problem would happen while installing it
under the non-ASCII path.

Fortunately, it is easy to fix.
 
Ofcourse, the files included in external libraries should be named ASCII only,
but `distutils.util.byte_compile` makes and uses full path of them. I learned
many of libraries uses setuptools instead of distutils, but please consider
the following situations:

1) The case that venv creates the virtual environment under or with
   non-ASCII named folder.
2) And special cases of 1), on Windows, the name of end-users OFTEN be
   composed with non-ASCII characters, this means including theirs
   Documents, Desktop, or Downloads etc.

The more they are not professional,they make easily the situation above.
msg347719 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 02:31
Here's test code.

Note: I marked just Python 3.8 and 3.7 I tested, but maybe all versions.
msg347721 - (view) Author: Aldwin Pollefeyt (aldwinaldwin) * Date: 2019-07-12 04:30
All 'The byte compiling test seems succeeded.' when i tested your script on ubuntu 18.04 on latest 3.7.4rc2+, 3.8.0b1+ and 3.9.0a0.

What are the exact python versions and the OS version you see the issue?
msg347723 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 05:24
Terribly Sorry! I forgot to write this is only Windows Issue! Thanks!
msg347724 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 05:35
... and please check "batch.diff" (Oops, that's typo of 'patch'). I just added "encoding" parameter to calling open() function, and added the temp file to 'utf-8' code declaration. This bug could happen where the default encoding of open() function is not utf-8(may not only Windows).
msg347732 - (view) Author: Aldwin Pollefeyt (aldwinaldwin) * Date: 2019-07-12 09:19
FYI, PEP597 is created to avoid these issues (i think), target Python 3.9.

https://www.python.org/dev/peps/pep-0597/

https://discuss.python.org/t/use-utf-8-as-default-text-file-encoding/1785
msg347749 - (view) Author: Suzumizaki (Suzumizaki) Date: 2019-07-12 15:41
Thanks, but Those cannot resolve this issue.

This is install time problem. In other words, whatever the default encoding is, even some users change it, `distutils.util.byte_compile` must always succeed. 

There's no relation between using non-ASCII path and the default encoding. 
No logically reason to fail.
History
Date User Action Args
2019-07-14 09:17:26vstinnersetnosy: + inada.naoki
2019-07-14 09:17:13vstinnersettitle: `distutils.util.byte_compile` fails indirect byte compiling with non-ASCII full-path. -> [Windows] distutils.util.byte_compile() fails indirect byte compiling with non-ASCII full-path
2019-07-12 15:41:42Suzumizakisetmessages: + msg347749
2019-07-12 09:19:00aldwinaldwinsetmessages: + msg347732
2019-07-12 05:35:10Suzumizakisetmessages: + msg347724
2019-07-12 05:24:09Suzumizakisetnosy: + vstinner, paul.moore, tim.golden, ezio.melotti, zach.ware, steve.dower
messages: + msg347723
components: + Unicode, Windows
2019-07-12 04:30:29aldwinaldwinsetnosy: + aldwinaldwin
messages: + msg347721
2019-07-12 02:31:26Suzumizakisetfiles: + my_test_case.py

messages: + msg347719
2019-07-12 02:28:51Suzumizakicreate