Spin-off of INADA-san's PEP 597: I propose to emit a new EncodingWarning warning when the locale encoding is used implicitly.

I propose to add a new warning to be able to control how they are handled: either make them hard error (python3.9 -W error::EncodingWarning) or ignore them (python3.9 -W ignore::EncodingWarning).

Displaying these warnings by default or not is an open question discussed in PEP 597 discussion:
https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819

I propose to only emit a warning which can be turned off easily to not bother people who don't care of encodings. There are many legit cases where a Python application works well in a control environment whereas it would break on any other environment, but the application is only used in one specific environment. For example, an application may only be run in a container which has a well defined locale and locale encoding, but will never be run on Windows, macOS, FreeBSD or HP-UX.

Writing portable code is *great*. But I'm not sure that *all* code must be portable. For example, it's also fine to use Python to write applications specific to Linux, again, for code only run in a controlled environment. Many companies have private projects which will never run outside their "walls" (outside their datacenters ;-)).

Building Python (make) with PR 13938 emits 7 EncodingWarning warnings:

$ PYTHONWARNINGS=always::EncodingWarning make

/home/vstinner/prog/python/master/Lib/sysconfig.py:228: EncodingWarning: encoding=None
  with open(filename, errors="surrogateescape") as f:
/home/vstinner/prog/python/master/Lib/sysconfig.py:371: EncodingWarning: encoding=None
  with open(config_h) as f:

./setup.py:559: EncodingWarning: encoding=None
  with open(tmpfile) as fp:
./setup.py:691: EncodingWarning: encoding=None
  with open(config_h) as file:
./setup.py:893: EncodingWarning: encoding=None
  with open(tmpfile) as fp:
./setup.py:1369: EncodingWarning: encoding=None
  with open(f) as file:
./setup.py:1496: EncodingWarning: encoding=None
  with open(zlib_h) as fp:


The test suite cannot be run if EncodingWarning are treated as error, an exception fails before running the first test:

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -j0 -r
...
  File "/home/vstinner/prog/python/master/Lib/test/libregrtest/main.py", line 427, in display_header
    print("==", platform.platform(aliased=True),
  ...
  File "/home/vstinner/prog/python/master/Lib/platform.py", line 613, in _syscmd_uname
    output = subprocess.check_output(('uname', option),
  ...
  File "/home/vstinner/prog/python/master/Lib/subprocess.py", line 821, in __init__
    self.stdout = io.TextIOWrapper(self.stdout,
EncodingWarning: encoding=None


To be able to run the test suite, I turned off the warning in subprocess.Popen using this change:

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index d34c57828b..d89911cc32 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -749,6 +749,9 @@ class Popen(object):
         self.stderr = None
         self.pid = None
         self.returncode = None
+        if not encoding:
+            import locale
+            encoding = locale.getpreferredencoding(False)
         self.encoding = encoding
         self.errors = errors


Using PR 13938 and subprocess.Popen change, at least 136 tests are failing. I have to interrupt the test suite since 7 tests hang:

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -j0 -r
...
7 tests omitted:
    test_cmd_line_script test_concurrent_futures test_httpservers
    test_multiprocessing_fork test_multiprocessing_spawn test_ssl
    test_subprocess

258 tests OK.

136 tests failed:
    test___all__ test__xxsubinterpreters test_argparse test_asyncio
    test_atexit test_base64 test_baseexception test_bdb test_bool
    test_builtin test_bz2 test_calendar test_capi test_cgi test_cgitb
    test_cmath test_cmd_line test_compile test_compileall test_complex
    test_configparser test_contextlib test_coroutines test_cprofile
    test_csv test_ctypes test_dbm test_dbm_dumb test_decimal
    test_defaultdict test_deque test_difflib test_distutils
    test_doctest test_eintr test_email test_embed test_eof
    test_exceptions test_faulthandler test_file test_file_eintr
    test_filecmp test_fileinput test_float test_gc test_gdb test_gzip
    test_hash test_http_cookiejar test_httplib test_idle test_imp
    test_import test_importlib test_inspect test_io test_iter
    test_json test_lib2to3 test_linecache test_list test_lltrace
    test_logging test_lzma test_mailbox test_mailcap test_math
    test_module test_modulefinder test_multiprocessing_forkserver
    test_multiprocessing_main_handling test_netrc test_ntpath
    test_opcodes test_os test_parser test_pathlib test_pdb test_pickle
    test_pipes test_pkg test_pkgimport test_pkgutil test_platform
    test_poll test_popen test_posix test_posixpath test_profile
    test_py_compile test_pydoc test_quopri test_random test_readline
    test_regrtest test_repl test_runpy test_sax test_script_helper
    test_select test_set test_shutil test_signal test_site test_socket
    test_source_encoding test_stat test_support test_symbol test_sys
    test_sysconfig test_tabnanny test_tarfile test_tcl test_tempfile
    test_threading test_tools test_trace test_traceback
    test_tracemalloc test_turtle test_unicodedata test_unittest
    test_univnewlines test_userlist test_utf8_mode test_venv
    test_warnings test_weakref test_webbrowser test_xml_etree
    test_xml_etree_c test_zipfile test_zipimport
    test_zipimport_support


For example, if EncodingWarning is treated as an error, test_cmd_line_script.test_repl_stderr_flush() hangs:

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -v test_cmd_line_script -m test_repl_stderr_flush
...
0:00:00 load avg: 0.83 [1/1] test_cmd_line_script
test_repl_stderr_flush (test.test_cmd_line_script.CmdLineTest) ... 


I would like to believe that Python code base is sane in general and that we handle encodings properly (I spent a significant amount of time to ensure that it's the case ;-)). So even if Python itself emits so many warnings, I don't think that it would be a good idea to introduce this warning and make it an hard error (exception) in Python 3.9.

I even don't think that such warning must be displayed by default. As DeprecationWarning, it's more a notice that should only be displayed to developers. Users don't control the code and so cannot fix the warning just by tuning their locale or locale encoding.

I don't see the point of flooding users with such warnings just to be pedantic.

As I wrote, they are many legit cases for encoding=None.

Another test: try to build a venv to get pip:


Building a venv is ok. Good, our stdlib is not so bad :-)

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m venv --without-pip env


But ensurepip fails soon:

$ PYTHONWARNINGS=error::EncodingWarning ./env/bin/python -W error::EncodingWarning -m ensurepip --verbose
/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/pyparsing.py:3068: SyntaxWarning: invalid escape sequence \w
/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/pyparsing.py:3068: SyntaxWarning: invalid escape sequence \w
/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
Exception:
Traceback (most recent call last):
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/install.py", line 255, in run
    with self._build_session(options) as session:
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/base_command.py", line 87, in _build_session
    session = PipSession(
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/download.py", line 344, in __init__
    self.headers["User-Agent"] = user_agent()
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/download.py", line 108, in user_agent
    zip(["name", "version", "id"], distro.linux_distribution()),
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py", line 120, in linux_distribution
    return _distro.linux_distribution(full_distribution_name)
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py", line 674, in linux_distribution
    self.name() if full_distribution_name else self.id(),
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py", line 712, in name
    name = self.os_release_attr('name') \
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py", line 883, in os_release_attr
    return self._os_release_info.get(attribute, '')
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py", line 550, in __get__
    ret = obj.__dict__[self._fname] = self._f(obj)
  File "/tmp/tmpyxn9yklq/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py", line 921, in _os_release_info
    with open(self.os_release_file) as release_file:
EncodingWarning: encoding=None


Note: an "up to date" pip still emits many SyntaxWarning and DeprecationWarning warnings :-(


New attempt using "always" action:

$ PYTHONWARNINGS=always::EncodingWarning ./env/bin/python -W always::EncodingWarning -m ensurepip --verbose
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/pyparsing.py:3068: SyntaxWarning: invalid escape sequence \w
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/pyparsing.py:3068: SyntaxWarning: invalid escape sequence \w
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py:921: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py:995: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py:1131: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_vendor/distro.py:1030: EncodingWarning: encoding=None
/home/vstinner/prog/python/master/Lib/subprocess.py:824: EncodingWarning: encoding=None
  self.stdout = io.TextIOWrapper(self.stdout,
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-2hizyo3q
Created temporary directory: /tmp/pip-req-tracker-kb535zvl
Created requirements tracker '/tmp/pip-req-tracker-kb535zvl'
Created temporary directory: /tmp/pip-install-_z_c4ze9
Looking in links: /tmp/tmpc309fubz
Collecting setuptools
  0 location(s) to search for versions of setuptools:
  Skipping link /tmp/tmpc309fubz (from -f); not a file
  Skipping link file:///tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl; wrong project name (not setuptools)
  Found link file:///tmp/tmpc309fubz/setuptools-40.8.0-py2.py3-none-any.whl, version: 40.8.0
  Local files found: /tmp/tmpc309fubz/setuptools-40.8.0-py2.py3-none-any.whl
  Using version 40.8.0 (newest of versions: 40.8.0)
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  Added setuptools from file:///tmp/tmpc309fubz/setuptools-40.8.0-py2.py3-none-any.whl to build tracker '/tmp/pip-req-tracker-kb535zvl'
  Removed setuptools from file:///tmp/tmpc309fubz/setuptools-40.8.0-py2.py3-none-any.whl from build tracker '/tmp/pip-req-tracker-kb535zvl'
Collecting pip
  0 location(s) to search for versions of pip:
  Found link file:///tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl, version: 19.0.3
  Skipping link file:///tmp/tmpc309fubz/setuptools-40.8.0-py2.py3-none-any.whl; wrong project name (not pip)
  Local files found: /tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl
  Using version 19.0.3 (newest of versions: 19.0.3)
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  Added pip from file:///tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl to build tracker '/tmp/pip-req-tracker-kb535zvl'
  Removed pip from file:///tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl from build tracker '/tmp/pip-req-tracker-kb535zvl'
Installing collected packages: setuptools, pip
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:140: EncodingWarning: encoding=None

/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:157: EncodingWarning: encoding=None
  changing mode of /home/vstinner/prog/python/master/env/bin/easy_install-3.9 to 775
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:140: EncodingWarning: encoding=None

/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:157: EncodingWarning: encoding=None
  changing mode of /home/vstinner/prog/python/master/env/bin/pip3 to 775
  changing mode of /home/vstinner/prog/python/master/env/bin/pip3.9 to 775
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
/tmp/tmpc309fubz/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
Successfully installed pip-19.0.3 setuptools-40.8.0
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-kb535zvl'


Hum, there are *a lot* of EncodingWarning warnings :-/


Trying to use pip fails with the same distro error:

$ PYTHONWARNINGS=error::EncodingWarning ./env/bin/python -W error::EncodingWarning -m pip install requests
...
  File "/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/distro.py", line 921, in _os_release_info
    with open(self.os_release_file) as release_file:
EncodingWarning: encoding=None


Installing requests using pip emits even more EncodingWarning warnings in many functions of pip, but also in distro and msgpack dependencies:

$ PYTHONWARNINGS=always::EncodingWarning ./env/bin/python -W always::EncodingWarning -m pip install requests
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  from collections import Mapping
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/distro.py:921: EncodingWarning: encoding=None
  with open(self.os_release_file) as release_file:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/distro.py:995: EncodingWarning: encoding=None
  with open(os.devnull, 'w') as devnull:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/distro.py:1131: EncodingWarning: encoding=None
  with open(filepath) as fp:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/distro.py:1030: EncodingWarning: encoding=None
  with open(os.devnull, 'w') as devnull:
/home/vstinner/prog/python/master/Lib/subprocess.py:824: EncodingWarning: encoding=None
  self.stdout = io.TextIOWrapper(self.stdout,
Collecting requests
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/msgpack/fallback.py:220: PendingDeprecationWarning: encoding is deprecated, Use raw=False instead.
  warnings.warn(
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  with open(entry_path, 'w') as fp:
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  with open(entry_path, 'w') as fp:
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  with open(entry_path, 'w') as fp:
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  with open(entry_path, 'w') as fp:
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py:60: EncodingWarning: encoding=None
  with open(entry_path, 'w') as fp:
Installing collected packages: certifi, chardet, idna, urllib3, requests
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:140: EncodingWarning: encoding=None
  with open(os.path.join(wheeldir, item, 'WHEEL')) as wheel:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:140: EncodingWarning: encoding=None
  with open(os.path.join(wheeldir, item, 'WHEEL')) as wheel:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:157: EncodingWarning: encoding=None
  with open(filename) as fp:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:140: EncodingWarning: encoding=None
  with open(os.path.join(wheeldir, item, 'WHEEL')) as wheel:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:140: EncodingWarning: encoding=None
  with open(os.path.join(wheeldir, item, 'WHEEL')) as wheel:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:140: EncodingWarning: encoding=None
  with open(os.path.join(wheeldir, item, 'WHEEL')) as wheel:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/wheel.py:95: EncodingWarning: encoding=None
  return open(name, mode + bin, **nl)
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/utils/outdated.py:40: EncodingWarning: encoding=None
  with open(self.statefile_path) as statefile:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_vendor/msgpack/fallback.py:220: PendingDeprecationWarning: encoding is deprecated, Use raw=False instead.
  warnings.warn(
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/utils/outdated.py:64: EncodingWarning: encoding=None
  with open(self.statefile_path) as statefile:
/home/vstinner/prog/python/master/env/lib/python3.9/site-packages/pip/_internal/utils/outdated.py:74: EncodingWarning: encoding=None
  with open(self.statefile_path, "w") as statefile:
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

My subprocess patch was wrong: you must use attached subprocess.patch.

I wrote this issue to discuss https://www.python.org/dev/peps/pep-0597/ but it's unrelated to the implementation of this PEP :-)

I reject this issue: see PEP 597 for the follow-up.