This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients vstinner
Date 2019-06-10.11:12:24
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1560165145.61.0.401899983978.issue37214@roundup.psfhosted.org>
In-reply-to
Content
Building Python (make) with PR 13938 emits 7 EncodingWarning warnings:

$ PYTHONWARNINGS=always::EncodingWarning make

/home/vstinner/prog/python/master/Lib/sysconfig.py:228: EncodingWarning: encoding=None
  with open(filename, errors="surrogateescape") as f:
/home/vstinner/prog/python/master/Lib/sysconfig.py:371: EncodingWarning: encoding=None
  with open(config_h) as f:

./setup.py:559: EncodingWarning: encoding=None
  with open(tmpfile) as fp:
./setup.py:691: EncodingWarning: encoding=None
  with open(config_h) as file:
./setup.py:893: EncodingWarning: encoding=None
  with open(tmpfile) as fp:
./setup.py:1369: EncodingWarning: encoding=None
  with open(f) as file:
./setup.py:1496: EncodingWarning: encoding=None
  with open(zlib_h) as fp:


The test suite cannot be run if EncodingWarning are treated as error, an exception fails before running the first test:

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -j0 -r
...
  File "/home/vstinner/prog/python/master/Lib/test/libregrtest/main.py", line 427, in display_header
    print("==", platform.platform(aliased=True),
  ...
  File "/home/vstinner/prog/python/master/Lib/platform.py", line 613, in _syscmd_uname
    output = subprocess.check_output(('uname', option),
  ...
  File "/home/vstinner/prog/python/master/Lib/subprocess.py", line 821, in __init__
    self.stdout = io.TextIOWrapper(self.stdout,
EncodingWarning: encoding=None


To be able to run the test suite, I turned off the warning in subprocess.Popen using this change:

diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index d34c57828b..d89911cc32 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -749,6 +749,9 @@ class Popen(object):
         self.stderr = None
         self.pid = None
         self.returncode = None
+        if not encoding:
+            import locale
+            encoding = locale.getpreferredencoding(False)
         self.encoding = encoding
         self.errors = errors


Using PR 13938 and subprocess.Popen change, at least 136 tests are failing. I have to interrupt the test suite since 7 tests hang:

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -j0 -r
...
7 tests omitted:
    test_cmd_line_script test_concurrent_futures test_httpservers
    test_multiprocessing_fork test_multiprocessing_spawn test_ssl
    test_subprocess

258 tests OK.

136 tests failed:
    test___all__ test__xxsubinterpreters test_argparse test_asyncio
    test_atexit test_base64 test_baseexception test_bdb test_bool
    test_builtin test_bz2 test_calendar test_capi test_cgi test_cgitb
    test_cmath test_cmd_line test_compile test_compileall test_complex
    test_configparser test_contextlib test_coroutines test_cprofile
    test_csv test_ctypes test_dbm test_dbm_dumb test_decimal
    test_defaultdict test_deque test_difflib test_distutils
    test_doctest test_eintr test_email test_embed test_eof
    test_exceptions test_faulthandler test_file test_file_eintr
    test_filecmp test_fileinput test_float test_gc test_gdb test_gzip
    test_hash test_http_cookiejar test_httplib test_idle test_imp
    test_import test_importlib test_inspect test_io test_iter
    test_json test_lib2to3 test_linecache test_list test_lltrace
    test_logging test_lzma test_mailbox test_mailcap test_math
    test_module test_modulefinder test_multiprocessing_forkserver
    test_multiprocessing_main_handling test_netrc test_ntpath
    test_opcodes test_os test_parser test_pathlib test_pdb test_pickle
    test_pipes test_pkg test_pkgimport test_pkgutil test_platform
    test_poll test_popen test_posix test_posixpath test_profile
    test_py_compile test_pydoc test_quopri test_random test_readline
    test_regrtest test_repl test_runpy test_sax test_script_helper
    test_select test_set test_shutil test_signal test_site test_socket
    test_source_encoding test_stat test_support test_symbol test_sys
    test_sysconfig test_tabnanny test_tarfile test_tcl test_tempfile
    test_threading test_tools test_trace test_traceback
    test_tracemalloc test_turtle test_unicodedata test_unittest
    test_univnewlines test_userlist test_utf8_mode test_venv
    test_warnings test_weakref test_webbrowser test_xml_etree
    test_xml_etree_c test_zipfile test_zipimport
    test_zipimport_support


For example, if EncodingWarning is treated as an error, test_cmd_line_script.test_repl_stderr_flush() hangs:

$ PYTHONWARNINGS=error::EncodingWarning ./python -W error::EncodingWarning -m test -v test_cmd_line_script -m test_repl_stderr_flush
...
0:00:00 load avg: 0.83 [1/1] test_cmd_line_script
test_repl_stderr_flush (test.test_cmd_line_script.CmdLineTest) ... 


I would like to believe that Python code base is sane in general and that we handle encodings properly (I spent a significant amount of time to ensure that it's the case ;-)). So even if Python itself emits so many warnings, I don't think that it would be a good idea to introduce this warning and make it an hard error (exception) in Python 3.9.

I even don't think that such warning must be displayed by default. As DeprecationWarning, it's more a notice that should only be displayed to developers. Users don't control the code and so cannot fix the warning just by tuning their locale or locale encoding.

I don't see the point of flooding users with such warnings just to be pedantic.

As I wrote, they are many legit cases for encoding=None.
History
Date User Action Args
2019-06-10 11:12:25vstinnersetrecipients: + vstinner
2019-06-10 11:12:25vstinnersetmessageid: <1560165145.61.0.401899983978.issue37214@roundup.psfhosted.org>
2019-06-10 11:12:25vstinnerlinkissue37214 messages
2019-06-10 11:12:24vstinnercreate