Attached patch changes how support.TESTFN_UNDECODABLE is computed on UNIX: use the filesystem encoding in *strict* mode, not using the surrogateescape error handler.

So we can use support.TESTFN_UNDECODABLE to check if a function uses correctly the surrogateescape error handler and/or check if it behaves correctly with non-ASCII characters.

The patch uses also support.TESTFN_UNDECODABLE (only on UNIX) in test_cmd_line_script.test_non_ascii() to also check that the fix for bpo-16218 works with UTF-8 locale encoding.

Please test the patch on UNIX, Windows and Mac OS X.

We may also use support.TESTFN_UNDECODABLE in test_cmd_line_script.test_non_ascii() on Windows, I will check.

Windows has some strange behaviour with undecodable characters: some of them are replaced a character with a similar glyph.

The patch contains two print to help debugging the patch itself, these print statements must be removed later.

+print("TESTFN_UNDECODABLE = %a" % TESTFN_UNDECODABLE)
+print("TESTFN_NONASCII = %a" % TESTFN_NONASCII)

We may also use support.TESTFN_UNDECODABLE
in test_cmd_line_script.test_non_ascii() on Windows

Oh, subprocess doesn't support passing bytes arguments to a program anymore (since Python 3.0).
http://bugs.python.org/issue4036#msg100376

So it's better to use TESTFN_NONASCII instead for this test ;-) It confirms that we need two constants depending on the context. It dependson the platform and how the data is read/written: sometimes undecodable characters are supported on any platform (ex: base64 encoder), sometimes undecodable characters are not supported (ex: distutils expects valid metadata), sometimes it depends on the platform (ex: this test).

Please test the patch on UNIX, Windows and Mac OS X.

The full test suite pass on:

• Linux with UTF-8 locale encoding
• Linux with ASCII locale encoding
• Windows with cp932 ANSI code page
• Mac OS 10.8 with ASCII locale encoding (and utf-8/surrogateescape for the filesystem encoding) ($LANG, $LC_ALL, $LC_CTYPE are not set)

Try b'\x81', b'\x98', b'\xae', b'\xd5', b'\xff'. They are undecodable in all 1-byte encodings.

b'\x81' : shift_jis_2004 shift_jis shift_jisx0213 cp869 cp874 cp932 cp1250 cp1252 cp1253 cp1254 cp1255 cp1257 cp1258
b'\x98' : shift_jis_2004 shift_jis shift_jisx0213 cp874 cp932 cp1250 cp1251 cp1253 cp1257
b'\xae' : iso8859-3 iso8859-6 iso8859-7 cp424
b'\xd5' : iso8859-8 cp856 cp857
b'\xff' : hp-roman8 iso8859-6 iso8859-7 iso8859-8 iso8859-11 shift_jis_2004 shift_jis shift_jisx0213 tis-620 cp864 cp874 cp1253 cp1255

Try b'\xed\xb2\x80' and b'\xed\xb4\x80' for UTF-8 (on Unix and Mac OS X).

b'\xed\xb2\x80' is b'\x80'.decode('utf-8', 'surrogateescape').encode('utf-8', 'surrogatepass').
b'\xed\xb4\x80' is '\udd00'.encode('utf-8', 'surrogatepass') and '\udd00' can't be encoded with surrogateescape.

The full test suite pass on:

The matter is not only in the fact that tests passed. They should fail if the original bug occurs again. Have you tried to restore the bugs?

The matter is not only in the fact that tests passed.

Right, but I don't want to introduce a regression :-)

They should fail if the original bug occurs again. Have you tried to restore the bugs?

test_cmd_line_script.test_non_ascii() comes from the issue bpo-16218, changeset 23ebe277e982. I checked this issue: support_undecodable.patch checks for non-regression with UTF-8 (and ASCI and ISO-8859-1) locale encoding on UNIX.

test_genericpath.test_non_ascii() comes from the issue bpo-3426, this fix comes from the issue bpo-3187, changeset 8a7c930abab6. I don't want to spend time on trying the new test on this issue because this 8a7c930abab6 is a major change, I don't see how to revert it just to test the issue. I consider the issue has fixed, and the new test should not reduce the test coverage, but just increase it ;-)

New changeset 6b8a8bc6ba9c by Victor Stinner in branch 'default':
Issue bpo-16444, bpo-16218: Use TESTFN_UNDECODABLE on UNIX
http://hg.python.org/cpython/rev/6b8a8bc6ba9c

TESTFN_UNDECODABLE is not detected for cp1250, cp1251, cp1252, cp1254, cp1257 and cp1258. Just add b'\x81\x98\xae\xd5\xff', at leas one of this bytes undecodable in some encoding which has any undecodable bytes.

I suppose you noticed you broke a bunch of buildbots :)

New changeset 398f8770bf0d by Victor Stinner in branch 'default':
Issue bpo-16444: disable undecodable characters in test_non_ascii() test until
http://hg.python.org/cpython/rev/398f8770bf0d

TESTFN_UNDECODABLE is not detected for cp1250, cp1251, cp1252, cp1254, cp1257 and cp1258.

The Python encoding and the real codec used by Windows are different: Python fails to decode bytes 0x80-0x9f, but Windows does decode them. I prefer to avoid these bytes to not rely too much on the Python codec.

These encodings used not only on Windows.

I suppose you noticed you broke a bunch of buildbots :)

Failures occur on FreeBSD, OpenIndiana and some other buildbots which don't set a locale and so use the "C" locale. main() decodes command line arguments from the locale encoding using _Py_char2wchar(). On these OSes, the "C" locale uses the ISO-8859-1 encoding, but the problem is that nl_langinfo(CODESET) announces ASCII :-/ test_cmd_line.test_undecodable_code() handles this case. Extract of a comment:

# _Py_char2wchar() decoded b'\xff' as '\xff' even if the locale is
# C and the locale encoding is ASCII. It occurs on FreeBSD, Solaris
# and Mac OS X.

Mac OS X is now using UTF-8 to decode the command line arguments.

I just created the issue bpo-16455 to fix FreeBSD and OpenIndiana.

I propose to close this issue because I consider it as fixed (bpo-16455 will reenable TESTFN_UNDECODABLE in test_cmd_line_script).

These encodings used not only on Windows.

You can uses cpXXX encodings explictly to read or write a file, but these encodings are not used for sys.getfilesystemencoding() (or sys.stdout.encoding).

New changeset 6017f09ead53 by Victor Stinner in branch '3.3':
Issue bpo-16218, bpo-16444: Backport improvment on tests for non-ASCII characters
http://hg.python.org/cpython/rev/6017f09ead53

You can uses cpXXX encodings explictly to read or write a file, but these
encodings are not used for sys.getfilesystemencoding() (or
sys.stdout.encoding).

At least CP1251 has been used for many cyrillic locales in before-UTF8 age (I use it sometimes still). For now CP1251 is the default encoding for Byelorussian and Bulgarian:

$ grep CP /usr/share/i18n/SUPPORTED
be_BY CP1251
bg_BG CP1251
ru_RU.CP1251 CP1251
yi_US CP1255

Ping.

New changeset ed0ff4b3d1c4 by Victor Stinner in branch 'default':
Issue bpo-16444: test more bytes in support.TESTFN_UNDECODABLE to support more Windows code pages
http://hg.python.org/cpython/rev/ed0ff4b3d1c4

Ooook, all remaining issues about undecodable bytes should now be fixed (until someone opens a new one? :-))

New changeset 41658a4fb3cc by Victor Stinner in branch '3.2':
Issue bpo-16218, bpo-16414, bpo-16444: Backport FS_NONASCII, TESTFN_UNDECODABLE,
http://hg.python.org/cpython/rev/41658a4fb3cc

New changeset 4d40c1ce8566 by Victor Stinner in branch '3.3':
(Merge 3.2) Issue bpo-16218, bpo-16414, bpo-16444: Backport FS_NONASCII,
http://hg.python.org/cpython/rev/4d40c1ce8566