classification
Title: test.support.FS_NONASCII returns incorrect result in Windows with non-US locale
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Ivan.Pozdeev, miss-islington, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2018-05-31 08:54 by Ivan.Pozdeev, last changed 2018-12-06 09:52 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 7278 merged Ivan.Pozdeev, 2018-05-31 08:56
PR 7279 merged Ivan.Pozdeev, 2018-05-31 09:24
PR 10981 merged miss-islington, 2018-12-06 07:05
PR 10982 merged miss-islington, 2018-12-06 07:05
Messages (7)
msg318271 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-05-31 08:54
This causes test_ntpath and test_posixpath to fail in subj.

Sample failure:

======================================================================
FAIL: test_expandvars_nonascii (__main__.NtCommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\Sasha\Documents\cpython\lib\test\test_genericpath.py", line 230
, in test_expandvars_nonascii
    check(u'$spam bar', u'%s bar' % unonascii)
  File "c:\Users\Sasha\Documents\cpython\lib\test\test_genericpath.py", line 214
, in check
    self.assertEqual(expandvars(value), expected)
AssertionError: u'? bar' != u'\xe6 bar'
- ? bar
? ^
+ \xe6 bar
? ^


Cause:

if sys.getfilesystemencoding()=='mbcs', encoding Unicode characters that are missing in the current locale succeeds but produces '?'.

So, test.support.FS_NONASCII's test fails to detect if a character is present in the current locale.
msg318308 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-31 13:32
> FAIL: test_expandvars_nonascii (__main__.NtCommonTest)

What is your Python version?

Python uses UTF-8 to encode paths on Windows since Python 3.6:
https://vstinner.github.io/python36-utf8-windows.html
msg318481 - (view) Author: Ivan Pozdeev (Ivan.Pozdeev) * Date: 2018-06-02 01:47
In 3.x, it turns out, this doesn't result in test failures in stock configuration.

It does though if PYTHONLEGACYWINDOWSFSENCODING is in system environment.

I was diagnosing failures in 2.x and saw that 3.x has the same logic, so it was a no-brainer to replicate the change...
msg331207 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-06 07:04
New changeset 8752dfbd1f0c96ca09cdacabaf0d0f8c3895b6ce by Serhiy Storchaka (native-api) in branch 'master':
bpo-33709: test_ntpath and test_posixpath fail in Windows with ACP!=1252. (GH-7278)
https://github.com/python/cpython/commit/8752dfbd1f0c96ca09cdacabaf0d0f8c3895b6ce
msg331208 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 07:22
New changeset b1438c0d376e1d438a11927e2698e3317da0d854 by Miss Islington (bot) in branch '3.7':
bpo-33709: test_ntpath and test_posixpath fail in Windows with ACP!=1252. (GH-7278)
https://github.com/python/cpython/commit/b1438c0d376e1d438a11927e2698e3317da0d854
msg331209 - (view) Author: miss-islington (miss-islington) Date: 2018-12-06 07:26
New changeset af31228650d30f02a283d291ba106e84275a04c1 by Miss Islington (bot) in branch '3.6':
bpo-33709: test_ntpath and test_posixpath fail in Windows with ACP!=1252. (GH-7278)
https://github.com/python/cpython/commit/af31228650d30f02a283d291ba106e84275a04c1
msg331217 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-06 09:43
New changeset 29a4cbff92862207eb9df9a970b3636b8b06ff5d by Serhiy Storchaka (native-api) in branch '2.7':
[2.7] bpo-33709: test_ntpath and test_posixpath fail in Windows with ACP!=1252. (GH-7278) (GH-7279)
https://github.com/python/cpython/commit/29a4cbff92862207eb9df9a970b3636b8b06ff5d
History
Date User Action Args
2018-12-06 09:52:10serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-12-06 09:43:39serhiy.storchakasetmessages: + msg331217
2018-12-06 07:26:53miss-islingtonsetmessages: + msg331209
2018-12-06 07:22:20miss-islingtonsetnosy: + miss-islington
messages: + msg331208
2018-12-06 07:05:26miss-islingtonsetpull_requests: + pull_request10223
2018-12-06 07:05:12miss-islingtonsetpull_requests: + pull_request10222
2018-12-06 07:04:38serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg331207
2018-06-02 01:47:15Ivan.Pozdeevsetmessages: + msg318481
2018-05-31 13:32:02vstinnersetmessages: + msg318308
2018-05-31 09:35:36serhiy.storchakasetnosy: + vstinner
2018-05-31 09:24:27Ivan.Pozdeevsetpull_requests: + pull_request6904
2018-05-31 08:56:13Ivan.Pozdeevsetkeywords: + patch
stage: patch review
pull_requests: + pull_request6903
2018-05-31 08:54:10Ivan.Pozdeevcreate