New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add PYTHONFSENCODING environment variable #52868
Comments
As discussed on bpo-8610, we need a way to override the automatic detection of the file system encoding - for much the same reasons we also do for the I/O encoding: the detection mechanism isn't fail-safe. We should add a new environment variable with the same functionality as PYTHONIOENCODING:
|
STINNER Victor wrote:
Thanks for the patch. A couple of notes:
We should also add a new sys.setfilesystemencoding() |
done
I copied the PYTHONIOENCODING doc which doesn't mention that. Does Python re-read other environment variables at runtime? Anyway, I changed the doc to: + If this is set before running the intepreter, it overrides the encoding used I also changed PYTHONIOENCODING doc. Is it better?
Ok, done. I patched also get_codeset() and get_codec_name() to always set a Python error.
Good idea, done.
It does already write a message to stderr, but it doesn't explain why it failed. I changed initfsencoding() to display two messages on get_codeset() error. First explain why get_codeset() failed (with the Python error) and then say that we fallback to utf-8. Full example (PYTHONFSENCODING error and simulated get_codeset() error):
No, I plan to REMOVE this function. sys.setfilesystemencoding() is dangerous because it introduces a lot of inconsistencies: this function is unable to reencode all filenames in all objects (eg. Python is unable to find filenames in user objects or 3rd party libraries). Eg. if you change the filesystem from utf8 to ascii, it will not be possible to use existing non-ascii (unicode) filenames: they will raise UnicodeEncodeError. As sys.setdefaultencoding() in Python2, I think that sys.setfilesystemencoding() is the root of evil :-) At startup, initfsencoding() sets the filesystem encoding using the locale encoding. Even for the startup process (with very few objects), it's very hard to find all filenames:
See bpo-9630 for the details. To remove sys.setfilesystemencoding(), I already patched PEP-383 tests (r84170) and I will open a new issue. But it's maybe better to commit both changes (remove the function and PYTHONFSENCODING) at the same time. |
done, issue bpo-9632 |
STINNER Victor wrote:
Yes, thanks.
Looks good !
Sorry, I wasn't aware we had such a function (and was looking at the
|
Commited to 3.2 as r84182. |
Oh, I realized that PYTHONFSENCODING is ignored on Windows and Mac OS X. r84201 and r84202 fix test_sys, and r84203 fixes the documentation and Python usage (hide PYTHONFSENCODING variable in Python help on Windows and Mac OS X). We might allow to override the filesystem encoding on Windows, but I don't think that it is a good idea because third party libraries will use anyway the mbcs encoding. |
STINNER Victor wrote:
This has to be changed: The env var needs to be respected on all |
I don't think so. On Mac OS X, you cannot create a file with an invalid utf-8 name. The VFS uses Use a different encoding will raise error for the first non-ascii filename. -- About Windows, Python3 uses the wide character API of Windows, except in some |
STINNER Victor wrote:
>
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
>
>>> Oh, I realized that PYTHONFSENCODING is ignored on Windows and Mac OS X.
>>> r84201 and r84202 fix test_sys, and r84203 fixes the documentation and
>>> Python usage (hide PYTHONFSENCODING variable in Python help on Windows
>>> and Mac OS X).
>>
>> This has to be changed: The env var needs to be respected on all
>> platforms.
>
> I don't think so.
>
> On Mac OS X, you cannot create a file with an invalid utf-8 name. The VFS uses
> utf-8:
> http://developer.apple.com/mac/library/qa/qa2001/qa1173.html
>
> Use a different encoding will raise error for the first non-ascii filename.
>
> --
>
> About Windows, Python3 uses the wide character API of Windows, except in some
> functions using third party libraries only providing a bytes API (eg.
> openssl). filenames are stored as unicode, even on removable media like CD-Rom
> or USB keys. I don't get the usecase here. Why would you like to change the
> filesystem encoding on Windows? Ok, point taken. Just please make sure that on other platforms such as BSD, Solaris, |
Le jeudi 19 août 2010 22:40:53, vous avez écrit :
I added much more tests on the filesystem encoding:
These tests are skipped on Windows and Mac OS X. I also patched the doc |
This is still an issue on some buildbots:
The issue was fixed in r84201, r84202, r84203 for OS X buildbots only, but since r84224 it is failing again. |
I'm working on a fix for test_sys failure. test_os should not fail anymore. |
In an up to date checkout of py3k on Gentoo linux with LC_CTYPE=en_US.UTF-8, I get a failure in test_sys: ====================================================================== Traceback (most recent call last):
File "/home/rdmurray/python/py3k/Lib/test/test_sys.py", line 605, in test_pythonfsencoding
self.check_fsencoding(get_fsencoding(env), 'ascii')
File "/home/rdmurray/python/py3k/Lib/test/test_sys.py", line 573, in check_fsencoding
self.assertEqual(fs_encoding, expected)
AssertionError: 'utf-8' != 'ascii'
- utf-8
+ ascii |
Setting LC_ALL instead of LANG in the test fixes the problem. |
r84308 should fix the last problems on Mac OS X, FreeBSD and Solaris. The last failure on test_sys is on Windows with test_undecodable_code (TypeError: Type str doesn't support the buffer API), which is unrelated. Reopen the issue if you see new failures. |
test_sys is still failing on my system where LC_CTYPE only is set to utf-8. Victor, do you want me to apply the LANG->LC_ALL change to the test? |
Oh yes, test_sys fails if LC_ALL or LC_CTYPE is a locale using a different encoding than ascii (eg. LC_ALL=fr_FR.utf8). Fixed by r84314. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: