$ PYTHONFSENCODING=latin-1 ./python Lib/test/test_warnings.py 
...

======================================================================
FAIL: test_nonascii (main.CEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:DeprecaciónWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcf3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (main.PyEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:DeprecaciónWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\\udcf3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

The problem is that subprocess encodes PYTHONWARNINGS environment variable value with the filesystem encoding, whereas Py_main() decodes the variable value with the locale encoding.

History of how the variable is read in py3k:

• bpo-7301: r79880 creates this variable, use mbstowcs() and PySys_AddWarnOption()
• bpo-7301: r80066 uses setlocale(LC_ALL, ""), and replaces mbstowcs() by _Py_char2wchar() (to support surrogates)
• bpo-8589: r81358 creates PySys_AddWarnOptionUnicode() and replaces _Py_char2wchar() by PyUnicode_DecodeFSDefault()
• bpo-8589: r84694 replaces PyUnicode_DecodeFSDefault() by _Py_char2wchar() + PyUnicode_FromWideChar() "because the PyCodec machinery is not ready yet"
• bpo-8589: r84731 uses PyUnicode_FromString() (utf-8) on Mac OS X

OK, so who's messing up: subprocess or Py_main()?

OK, so who's messing up: subprocess or Py_main()?

Well, this is the real question :-)

locale encoding is used to decode command line arguments (sys.argv), filesystem encoding is used to decode environment variables and to encode subprocess arguments and environment variables.

It means that we have something funny if a Python process creates a Python subprocess: child process arguments are encoded using the filesystem encoding, whereas the arguments are decoded using the locale encoding. If both encodings are different (eg. if PYTHONFSENCODING is used by at least the parent process), we have the bug similar to bpo-4388. See also issue bpo-8775.

It sounds like you had PYTHONWARNINGS using the fs encoding before r84694, but reverted it due to bootstrapping issues.

Indeed, the fs encoding isn't initialized until later in Py_InitializeEx. Maybe the PYTHONWARNINGS code should be moved there instead?

Indeed, the fs encoding isn't initialized until later in
Py_InitializeEx. Maybe the PYTHONWARNINGS code should be moved
there instead?

sys.warnopts should be filled early because it is used to initialize the _warnings module, and the _warnings module have to be initialized before loading another non-builtin module because importing a module may emit a warning.

OK, so who's messing up: subprocess or Py_main()?

I opened issue bpo-9992 to decide which encoding should be used to encode and decode command line arguments: locale or filesystem encoding.

Maybe the PYTHONWARNINGS code should be moved there instead?

sys.warnoptions is read by the warnings module (not the _warnings module) when this module is loaded. The warnings module is loaded by Py_InitializeEx() if sys.warnoptions list is not empty.

It might be possible to read PYTHONWARNINGS env var after initfsencoding() but before loading the warnings module. But we have to ensure that Py_InitializeEx() can still be called multiple times.

Fixed by r85430 (remove PYTHONFSENCODING), see bpo-9992.