have a problem that python3.6 can not reopen .pyc file with Chinese path, and python3.5 can reopen the same pyc file. As shown in the picture

run_file encodes the file path via PyUnicode_EncodeFSDefault, which encodes as UTF-8 in Windows, starting with 3.6. PyRun_SimpleFileExFlags subsequently tries to open this encoded path via _Py_fopen, which calls fopen. The CRT expects an ANSI encoded path, so only the common ASCII subset will work. Non-ASCII paths will fail.

This could be addressed in _Py_fopen by decoding the path and calling _wfopen instead of fopen. 

Executing a .pyc also fails in 3.5 if the wide-character path can't be encoded as ANSI, but the 3.5 branch only accepts security fixes.

Thanks a lot. What should I do to  reopen .pyc file with non-ASCII path use
python3.6 in cmd？Could you give me* some **code examples*.Thank you again,
and I look forward to hearing from you

2017-12-20 15:35 GMT+08:00 Eryk Sun <report@bugs.python.org>:

>
> Eryk Sun <eryksun@gmail.com> added the comment:
>
> run_file encodes the file path via PyUnicode_EncodeFSDefault, which
> encodes as UTF-8 in Windows, starting with 3.6. PyRun_SimpleFileExFlags
> subsequently tries to open this encoded path via _Py_fopen, which calls
> fopen. The CRT expects an ANSI encoded path, so only the common ASCII
> subset will work. Non-ASCII paths will fail.
>
> This could be addressed in _Py_fopen by decoding the path and calling
> _wfopen instead of fopen.
>
> Executing a .pyc also fails in 3.5 if the wide-character path can't be
> encoded as ANSI, but the 3.5 branch only accepts security fixes.
>
> ----------
> components: +Interpreter Core, Unicode
> nosy: +eryksun, ezio.melotti, vstinner
> stage:  -> test needed
> title: python3.6 can not reopen .pyc file with Chinese path -> Python 3.6
> cannot reopen .pyc file with non-ASCII path
> type: compile error -> behavior
> versions: +Python 3.7
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32381>
> _______________________________________
>

Workarounds: (1) force 3.6 to use the legacy ANSI filesystem encoding by setting the environment variable PYTHONLEGACYWINDOWSFSENCODING. (2) Use 8.3 DOS names, if creating them is enabled on the volume. You can check their value in CMD via `dir /x`. (3) Create alternative directory symbolic links or junctions with ASCII names via CMD's `mklink` command.

run_file() gets a wchar_t* string which comes from wmain() argv.

run_file() encodes the wchar_t* using PyUnicode_EncodeFSDefault().

Later, PyRun_SimpleFileExFlags() calls indirectly fopen() with the encoded filename.

> This could be addressed in _Py_fopen by decoding the path and calling _wfopen instead of fopen. 

I agree that it's the correct fix.

I would make _Py_fopen() more compatible with the PEP 529.

> I would make _Py_fopen() more compatible with the PEP 529.

Typo: It* would

Hum. In fact, this problem can be fixed differently: modify PyRun_xxx() functions to pass the filename as an Unicode string. Maybe pass it as a wchar_t* string or even a Python str object.

Thanks, Eryk, for catching the dup, I missed it somehow.

@ZackerySpytz: do you plan to proceed with your PR? If not, I can pick it up -- this issue broke the software I develop after upgrade to 3.8.

I filed issue 42569 to hopefully clarify the status of _Py_fopen() which became murky to me.

I can reproduce the issue on Python 3.10 with a script called 北京市.py which contains: print("hello").

c:\> python 北京市.py
hello

c:\>python __pycache__\北京市.cpython-310.pyc
python: Can't reopen .pyc file

And with my PR 23642 fix, it works as expected:

C:\>python __pycache__\北京市.cpython-310.pyc
hello

bpo-42568 is marked as a duplicate of this issue.

Thanks for the patch, Victor, it looks good.

Just so it doesn't get lost: the problem with the contract of PyErr_ProgramText() which I mentioned in my dup 42568 is still there.

> Just so it doesn't get lost: the problem with the contract of PyErr_ProgramText() which I mentioned in my dup 42568 is still there.

It seems like PyErr_ProgramText() is no longer used in Python. PyErr_ProgramTextObject() is used and it pass the filename as Python object to _Py_fopen_obj().

> It seems like PyErr_ProgramText() is no longer used in Python.

Isn't it a part of the public API? I can't find it in the docs, but it seems to be declared in the public header.

> Isn't it a part of the public API? I can't find it in the docs, but it seems to be declared in the public header.

The Python C API has a strange history...

New changeset b6d98c10fff6f320f8fdf595c3f9a05d8be4e31d by Victor Stinner in branch 'master':
bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642)
https://github.com/python/cpython/commit/b6d98c10fff6f320f8fdf595c3f9a05d8be4e31d

New changeset f0e42ae03c41ec32fcb3064772f46ff7f2c5ff3b by Victor Stinner in branch '3.9':
bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642) (GH-23692)
https://github.com/python/cpython/commit/f0e42ae03c41ec32fcb3064772f46ff7f2c5ff3b

New changeset b5cf308de8b19bf8f77053013f7e8a944159e1aa by Victor Stinner in branch '3.8':
bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642) (GH-23692) (GH-23696)
https://github.com/python/cpython/commit/b5cf308de8b19bf8f77053013f7e8a944159e1aa

It's now fixed in 3.8, 3.9 and master branches.

Thanks for the bug report tianjg.

Thanks for the fix and backports!

New changeset 815506d852daabc40e14ff0987c1142c0205fbe7 by Victor Stinner in branch 'master':
bpo-32381: Rewrite PyErr_ProgramText() (GH-23700)
https://github.com/python/cpython/commit/815506d852daabc40e14ff0987c1142c0205fbe7

New changeset 550e4673be538d98b6ddf5550b3922539cf5c4b2 by Victor Stinner in branch 'master':
bpo-32381: Add _PyRun_SimpleFileObject() (GH-23709)
https://github.com/python/cpython/commit/550e4673be538d98b6ddf5550b3922539cf5c4b2

New changeset ca064402079f889226cb107b26b329891431c319 by Victor Stinner in branch 'master':
bpo-32381: Remove unused _Py_fopen() function (GH-23711)
https://github.com/python/cpython/commit/ca064402079f889226cb107b26b329891431c319

New changeset a82f63f5af027a0eab0f0812d750b804368cbd25 by Victor Stinner in branch 'master':
bpo-32381: Add _PyRun_AnyFileObject() (GH-23723)
https://github.com/python/cpython/commit/a82f63f5af027a0eab0f0812d750b804368cbd25

My PR 23778 fix the encoding/error handler when writing the filename into stderr, when the file does not exist:

$ LANG= PYTHONCOERCECLOCALE=0 ./python -X utf8=0 héllo.py
./python: can't open file '/home/vstinner/python/master/h\udcc3\udca9llo.py': [Errno 2] No such file or directory

New changeset ceb420251c1d635520049fbb7b5269a73d63fb58 by Victor Stinner in branch 'master':
bpo-32381: pymain_run_file() uses PySys_FormatStderr() (GH-23778)
https://github.com/python/cpython/commit/ceb420251c1d635520049fbb7b5269a73d63fb58

New changeset a12491681f08a33abcca843f5150330740c91111 by Victor Stinner in branch 'master':
bpo-32381: pymain_run_command() uses PyCF_IGNORE_COOKIE (GH-23724)
https://github.com/python/cpython/commit/a12491681f08a33abcca843f5150330740c91111

boost-python was using the removed private _Py_fopen() function, I proposed https://github.com/boostorg/python/pull/344 to replace _Py_fopen() with fopen().