classification
Title: Wrong ANSI encoding used by subprocess for some locales
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Segev Finer, eryksun, miss-islington, paul.moore, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2017-12-19 00:24 by Segev Finer, last changed 2018-02-13 09:18 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5608 merged Segev Finer, 2018-02-10 13:40
PR 5652 merged miss-islington, 2018-02-13 06:31
PR 5654 merged serhiy.storchaka, 2018-02-13 07:24
Messages (6)
msg308597 - (view) Author: Segev Finer (Segev Finer) * Date: 2017-12-19 00:24
The following test is failing randomly for me (python -m test test_uuid -v):
======================================================================
ERROR: test_ipconfig_getnode (test.test_uuid.TestInternalsWithoutExtModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "cpython\lib\test\test_uuid.py", line 551, in test_ipconfig_getnode
    node = self.uuid._ipconfig_getnode()
  File "cpython\lib\uuid.py", line 487, in _ipconfig_getnode
    for line in pipe:
  File "cpython\lib\encodings\cp1255.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 682: character maps to <undefined>

This is caused by trying to decode the output of "ipconfig" using cp1255, while the output really uses cp862 and annoyingly it started to print times using the current locale (Which displays broken in the console anyhow, question mark boxes... *sigh*) in some Windows version (Using Windows 10.0.16299 ATM).
msg308614 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-12-19 05:44
ipconfig uses (or defaults to) OEM encoded output when writing to a pipe or file. On the other hand, Python's TextIOWrapper defaults to ANSI (i.e. 'mbcs'). In 3.6+, uuid._ipconfig_getnode could be rewritten to call subprocess.Popen with the new 'oem' encoding. 

In Windows 10, the networking command-line utilities (e.g. ipconfig, netstat, nbtstat, ping, tracert, hostname, finger, and ftp) support an "OutputEncoding" environment variable. Its value can be set to "Unicode" (UTF-16), "UTF-8", or "Ansi". I don't think this is supported in Windows 7, however.
msg311952 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2018-02-10 16:01
The patch looks good, but the rest of the code for this looks horrible(and potentially a security risk)... maybe we ought to take this fix for now but prioritise replacing this whole function with a C implementation or at least a better search for the ipconfig tool.
msg312104 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-13 06:29
New changeset da6c3da6c33c6bf794f741e348b9c6d86cc43ec5 by Serhiy Storchaka (Segev Finer) in branch 'master':
bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
https://github.com/python/cpython/commit/da6c3da6c33c6bf794f741e348b9c6d86cc43ec5
msg312110 - (view) Author: miss-islington (miss-islington) Date: 2018-02-13 08:58
New changeset 9b5a90b975ef32b261d60b8ec06504f4ffd00d63 by Miss Islington (bot) in branch '3.7':
bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
https://github.com/python/cpython/commit/9b5a90b975ef32b261d60b8ec06504f4ffd00d63
msg312111 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-13 09:15
New changeset c3f9d7e0ea30e94c901d13e1d43ff0be1e5dbcb7 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608). (#5654)
https://github.com/python/cpython/commit/c3f9d7e0ea30e94c901d13e1d43ff0be1e5dbcb7
History
Date User Action Args
2018-02-13 09:18:06serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-02-13 09:15:26serhiy.storchakasetmessages: + msg312111
2018-02-13 08:58:30miss-islingtonsetnosy: + miss-islington
messages: + msg312110
2018-02-13 07:24:44serhiy.storchakasetpull_requests: + pull_request5456
2018-02-13 06:31:02miss-islingtonsetpull_requests: + pull_request5454
2018-02-13 06:29:57serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg312104
2018-02-11 10:48:12serhiy.storchakasetversions: + Python 3.6
2018-02-10 16:01:36steve.dowersetmessages: + msg311952
2018-02-10 13:43:45Segev Finersetversions: + Python 3.8
2018-02-10 13:40:58Segev Finersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5418
2017-12-19 05:44:09eryksunsetnosy: + eryksun
messages: + msg308614
2017-12-19 00:24:51Segev Finercreate