➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python Launcher, Windows, fails on scripts w/ non-latin names
Type: behavior Stage: resolved
Components: Unicode, Windows Versions: Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, ezio.melotti, loewis, mdengler, steve.dower, terry.reedy, tim.golden, vinay.sajip, vstinner, zach.ware, zart
Priority: normal Keywords: patch

Created on 2013-12-21 14:10 by zart, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
pylauncher-fix-launcing-unicode-filenames.patch zart, 2013-12-21 14:10 Patch to fix unicode filename handling on command-line review
pylauncher-fix-launcing-unicode-filenames.patch zart, 2013-12-21 14:13 Patch to fix unicode filename handling on command-line review
Messages (12)
msg206741 - (view) Author: Konstantin Zemlyak (zart) Date: 2013-12-21 14:10 
Running `py.exe юникод.py` in cmd window fails:

E:\>set PYLAUNCH_DEBUG=1

E:\>py юникод.py
launcher build: 32bit
launcher executable: Console
File 'C:\Users\Zart\AppData\Local\py.ini' non-existent
Using global configuration file 'C:\Windows\py.ini'
Called with command line: .pymaybe_handle_shebang: read 211 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 64bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files\Python27\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\amd64\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files\Python32\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files\Python33\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locating Pythons in native registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files (x86)\Python27\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\amd64\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files (x86)\Python32\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files (x86)\Python33\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: ?????????????? ?????? ? ????? ?????, ????? ????? ??? ????? ????.
found configured value 'python3=3.2-32' in environment
search for Python version '3.2-32' found '"C:\Program Files (x86)\Python32\python.exe"'
run_child: about to run '"C:\Program Files (x86)\Python32\python.exe" .py'
C:\Program Files (x86)\Python32\python.exe: can't open file '.py': [Errno 2] No such file or directory
child process exit code: 2


Note "Called with command line: .py" in output shows that filename was mangled very early on. Invoking `юникод.py` (which is associated with py.exe) directly works fine though.

The problem lies in Windows handling of command-line arguments and fix to this is simple but non-obvious. Patch attached.
msg206743 - (view) Author: Konstantin Zemlyak (zart) Date: 2013-12-21 14:13 
Sorry, fixed whitespaces in the patch.
msg206744 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-12-21 14:19 
It looks like the wide character strings (wchar_t*) are misused. For example:

error(RC_NO_PYTHON, L"Requested Python version (%s) ...", &p[1]);
fwprintf(stdout, L"usage: %s ...\n\n", argv[0]);

The %s formatter is for byte string (char*), "%ls" should be used instead.

+	_setmode(_fileno(stdout), _O_WTEXT);

Extract of wprintf() documentation:

"The wprintf() and vwprintf() functions perform wide-character output to stdout.  stdout must not be byte oriented;  see  fwide(3)  for more information."

So _setmode() or fwide() should be used if I understood correctly. Or wprintf() should be replaced with printf() (still with "%ls" format)?

wprintf("%ls") replaces unencodable character string arguments by ? (U+003F), whereas printf("%ls") and wprintf("%s") truncates the output at the first undecodable/unencodable character:
http://unicodebook.readthedocs.org/en/latest/programming_languages.html#printf-functions-family

So wprintf() is probably better here.
msg206745 - (view) Author: Konstantin Zemlyak (zart) Date: 2013-12-21 14:22 
I don't care much about debug output though it probably should be fixed.

The point is that changing text mode of stdout has a weird side effect of fixing command-line arguments when invoking interactively from cmd.exe.
msg206760 - (view) Author: Konstantin Zemlyak (zart) Date: 2013-12-21 15:46 
There is something weird with my proposed fix. Right after submitting a bug with patch I've updated pythons on my system - 2.7.5 to 2.7.6, 3.3.2 to 3.3.3, and installed 3.4.0b1 - both 32- and 64-bit. Then my fixed py.exe stopped working.

Then I've added _setmode for stdin/stdout/stderr and rebuilt both debug/release and x86/x64 versions:

E:\>set PYLAUNCH_DEBUG=1

E:\>e:\cpython\PCbuild\py.exe юникод.py
launcher build: 32bit
launcher executable: Console
launcher charset: Multi-byte
File 'C:\Users\Zart\AppData\Local\py.ini' non-existent
File 'e:\cpython\PCbuild\py.ini' non-existent
Called with command line: .py
maybe_handle_shebang: read 211 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 64bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files\Python27\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python32\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python33\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python34\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locating Pythons in native registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files (x86)\Python27\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python32\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python33\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python34\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
found configured value 'python3=3.3-32' in environment
search for Python version '3.3-32' found '"C:\Program Files (x86)\Python33\python.exe"'
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" .py'
C:\Program Files (x86)\Python33\python.exe: can't open file '.py': [Errno 2] No such file or directory
child process exit code: 2

E:\>e:\cpython\PCbuild\py_d.exe юникод.py
launcher build: 32bit
launcher executable: Console
launcher charset: Multi-byte
File 'C:\Users\Zart\AppData\Local\py.ini' non-existent
File 'e:\cpython\PCbuild\py.ini' non-existent
Called with command line: юникод.py
maybe_handle_shebang: read 211 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 64bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files\Python27\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python32\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python33\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python34\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locating Pythons in native registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files (x86)\Python27\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python32\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python33\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python34\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
found configured value 'python3=3.3-32' in environment
search for Python version '3.3-32' found '"C:\Program Files (x86)\Python33\python.exe"'
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" юникод.py'
Привет
child process exit code: 0

E:\>e:\cpython\PCbuild\amd64\py.exe юникод.py
launcher build: 64bit
launcher executable: Console
launcher charset: Multi-byte
File 'C:\Users\Zart\AppData\Local\py.ini' non-existent
File 'e:\cpython\PCbuild\amd64\py.ini' non-existent
Called with command line: юникод.py
maybe_handle_shebang: read 211 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 32bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files (x86)\Python27\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python32\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python33\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python34\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locating Pythons in native registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files\Python27\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python32\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python33\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python34\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
found configured value 'python3=3.3-32' in environment
search for Python version '3.3-32' found '"C:\Program Files (x86)\Python33\python.exe"'
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" юникод.py'
Привет
child process exit code: 0

E:\>e:\cpython\PCbuild\amd64\py_d.exe юникод.py
launcher build: 64bit
launcher executable: Console
launcher charset: Multi-byte
File 'C:\Users\Zart\AppData\Local\py.ini' non-existent
File 'e:\cpython\PCbuild\amd64\py.ini' non-existent
Called with command line: никод.py
maybe_handle_shebang: read 211 bytes
maybe_handle_shebang: BOM not found, using UTF-8
parse_shebang: found command: python3
locating Pythons in 32bit registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files (x86)\Python27\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python32\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python33\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python34\python.exe" is a 32bit executable
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files (x86)\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locating Pythons in native registry
locate_pythons_for_key: unable to open PythonCore key in HKCU
locate_pythons_for_key: "C:\Program Files\Python27\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python2\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python32\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python33\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python34\python.exe" is a 64bit executable
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
locate_pythons_for_key: "C:\Program Files\Python3\PCBuild\amd64\python.exe: Синтаксическая ошибка в имени файла, имени папки или метке тома.
found configured value 'python3=3.3-32' in environment
search for Python version '3.3-32' found '"C:\Program Files (x86)\Python33\python.exe"'
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" никод.py'
C:\Program Files (x86)\Python33\python.exe: can't open file '<unprintable file name>': [Errno 2] No such file or directory
child process exit code: 2




Setting wide mode for stderr had fixed error messages in debug output. And looks like x64 debug build has off-by-one error and CRT behavior is wonky regarding command-line handling. So my patch doesn't really fix original problem yet exhibits some underlying crt bug.
msg206763 - (view) Author: Konstantin Zemlyak (zart) Date: 2013-12-21 16:38 
Some more fun stuff with command-line (I'm cutting output to few essential lines for easier reading):


e:\cpython\PCbuild\py.exe юникод.py
...
Called with command line: .py
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" .py'
C:\Program Files (x86)\Python33\python.exe: can't open file '.py': [Errno 2] No such file or directory
child process exit code: 2

e:\cpython\PCbuild\py.exe e:\юникод.py
...
Called with command line: e:\юникод.py
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" e:\юникод.py'
child process exit code: 0


E:\>e:\cpython\PCbuild\py.exe тест\unicode.py
...
Called with command line: ест\unicode.py
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" ест\unicode.py'
C:\Program Files (x86)\Python33\python.exe: can't open file '<unprintable file name>': [Errno 2] No such file or directory
child process exit code: 2

E:\>e:\cpython\PCbuild\py.exe e:\тест\unicode.py
...
Called with command line: e:\тест\unicode.py
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" e:\тест\unicode.py'
child process exit code: 0


E:\>e:\cpython\PCbuild\py.exe "юникод.py"
Called with command line: "юникод.py"
run_child: about to run '"C:\Program Files (x86)\Python33\python.exe" "юникод.py"'
child process exit code: 0




IOW, so long as command-line starts with ASCII character everything is fine. If not, then one or more characters gets mangled. Now I'm not sure whether it's a cmd.exe bug or C runtime one, and whether it's possible to workaround about it.
msg225162 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-08-10 20:23 
Is this actually fixable?  I only ask as we seem to have a whole lot of fun with anything involving cmd.exe as epitomized on #1602.
msg225496 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2014-08-18 16:02 
It should be fixable. In general, Unicode in the console is fine, but the CRT doesn't handle it well (as shown by the _setmode extension being able to fix it).

The 'correct' fix for Unicode in the console is at http://www.siao2.com/2010/04/07/9989346.aspx and it basically comes down to "use the Windows API and not the CRT". It's certainly fixable here, though the general fix for Python itself is more difficult because we want/need to expose the bytes interface as well (that said, #1602 seems to have a good fix right now that just happens to be easily distributable as pure Python code, so there's little motivation to merge it in, especially since it will break back-compat).

I don't know entirely whether _setmode is a correct fix here, or if the attached patch is sufficient, but it can be fixed.
msg225529 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2014-08-19 11:12 
The problem is skip_whitespace mistakenly calls isspace instead of iswspace. 

http://hg.python.org/cpython/file/c0e311e010fc/PC/launcher.c#l48

isspace has undefined behavior when the argument is "not EOF or in the range of 0 through 0xFF":

http://msdn.microsoft.com/en-us/library/y13z34da%28v=vs.100%29.aspx

The display of debug messages should be handled in its own issue. IMO, setting stderr to _O_WTEXT mode or _O_U16TEXT mode looks reasonable. The launcher already uses wide-character strings and the wprintf family. It's just the default _O_TEXT mode ends up encoding to the console codepage. 

Regarding msg206744, %s in wide-character format strings is OK for VC++ 10:

http://msdn.microsoft.com/en-us/library/hf4y5e3w%28v=vs.100%29.aspx

This will be a legacy mode in VC++ 14 (CPython 3.5?): 

http://blogs.msdn.com/b/vcblog/archive/2014/06/18/crt-features-fixes-and-breaking-changes-in-visual-studio-14-ctp1.aspx

P.S.
In case someone wants an easy issue, I noticed a bug. There should only be one %s in the format string that starts on line 262:

http://hg.python.org/cpython/file/c0e311e010fc/PC/launcher.c#l262
msg241891 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-04-23 21:34 
The bug reported in msg225529 has been fixed, but there's another one a few lines up https://hg.python.org/cpython/file/bd656916586f/PC/launcher.c#l265 as there's only one % but two parameters.  Although IIRC we'd get away with this the way C works, shouldn't we fix it anyway?
msg297792 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-07-06 01:26 
Is this still relevant or should it be closed?

On Win10, I created a short script юникод.py using Save As from IDLE.

py -2 юникод.py produces
C:\Programs\Python27\python.exe: can't open file '??????.py': [Errno 22] Invalid argument
If the patch fixes 2.7 and looks okay, maybe we should apply. If 'no' or 'no', maybe we should forget 2.7 running files with such names.

On 3.5 and 3.6, the file runs without issue.  The issue was opened with 3.3; 3.5 switched to a much more recent compiler, and I did not see any indication in the messages that this was tested on 3.5 before it was added.  So perhaps for 3.5+, this is out-of-date.
msg297801 - (view) Author: Konstantin Zemlyak (zart) Date: 2017-07-06 04:37 
Terry J. Reedy wrote:

> Is this still relevant or should it be closed?

Should be closed.

> On Win10, I created a short script юникод.py using Save As from IDLE.
>
> py -2 юникод.py produces
> C:\Programs\Python27\python.exe: can't open file '??????.py': [Errno 22] Invalid argument
> If the patch fixes 2.7 and looks okay, maybe we should apply. If 'no' or 'no', maybe we should forget 2.7 running files with such names.

This outcome is expected and error comes from python itself, not from launcher.

> On 3.5 and 3.6, the file runs without issue.  The issue was opened with 3.3; 3.5 switched to a much more recent compiler, and I did not see any indication in the messages that this was tested on 3.5 before it was added.  So perhaps for 3.5+, this is out-of-date.

Launcher works fine now from my testing.
History
Date User Action Args
2022-04-11 14:57:55adminsetgithub: 64241
2017-07-06 05:54:42terry.reedysetstatus: open -> closed
resolution: out of date
stage: patch review -> resolved
2017-07-06 04:37:35zartsetmessages: + msg297801
2017-07-06 01:26:45terry.reedysetnosy: + terry.reedy, - BreamoreBoy
messages: + msg297792
versions: + Python 3.6, Python 3.7, - Python 3.4, Python 3.5
2015-04-23 21:34:37BreamoreBoysetmessages: + msg241891
2014-08-19 11:12:24eryksunsetnosy: + eryksun
messages: + msg225529
2014-08-18 16:02:38steve.dowersetmessages: + msg225496
2014-08-10 20:23:36BreamoreBoysetnosy: + BreamoreBoy, vinay.sajip, zach.ware, steve.dower, - brian.curtin

messages: + msg225162
versions: + Python 2.7, Python 3.4, Python 3.5, - Python 3.3
2014-02-28 11:52:36mdenglersetnosy: + mdengler
2013-12-27 21:27:15terry.reedysettitle: Python Launcher for Windows fails to invoke scripts with non-latin names -> Python Launcher, Windows, fails on scripts w/ non-latin names
2013-12-21 16:38:32zartsetmessages: + msg206763
2013-12-21 15:46:44zartsetmessages: + msg206760
versions: - Python 3.4
2013-12-21 14:59:47pitrousetstage: patch review
versions: + Python 3.4
2013-12-21 14:22:53zartsetmessages: + msg206745
2013-12-21 14:20:22vstinnersetnosy: + loewis, tim.golden, brian.curtin
2013-12-21 14:19:50vstinnersetmessages: + msg206744
2013-12-21 14:13:37zartsetfiles: + pylauncher-fix-launcing-unicode-filenames.patch

messages: + msg206743
2013-12-21 14:10:03zartcreate