Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wrong ANSI encoding used by subprocess for some locales #76551

Closed
segevfiner mannequin opened this issue Dec 19, 2017 · 6 comments
Closed

Wrong ANSI encoding used by subprocess for some locales #76551

segevfiner mannequin opened this issue Dec 19, 2017 · 6 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes OS-windows type-bug An unexpected behavior, bug, or error

Comments

@segevfiner
Copy link
Mannequin

segevfiner mannequin commented Dec 19, 2017

BPO 32370
Nosy @pfmoore, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @segevfiner, @miss-islington
PRs
  • bpo-32370: Use the correct encoding for ipconfig output in the uuid module #5608
  • [3.7] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608) #5652
  • [3.6] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608). #5654
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-02-13.09:18:06.475>
created_at = <Date 2017-12-19.00:24:51.164>
labels = ['3.8', 'type-bug', '3.7', 'OS-windows']
title = 'Wrong ANSI encoding used by subprocess for some locales'
updated_at = <Date 2018-02-13.09:18:06.474>
user = 'https://github.com/segevfiner'

    bugs.python.org fields:

    activity = <Date 2018-02-13.09:18:06.474>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-02-13.09:18:06.475>
closer = 'serhiy.storchaka'
components = ['Windows']
creation = <Date 2017-12-19.00:24:51.164>
creator = 'Segev Finer'
dependencies = []
files = []
hgrepos = []
issue_num = 32370
keywords = ['patch']
message_count = 6.0
messages = ['308597', '308614', '311952', '312104', '312110', '312111']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'Segev Finer', 'miss-islington']
pr_nums = ['5608', '5652', '5654']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32370'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
    @segevfiner
    Copy link
    Mannequin Author

    segevfiner mannequin commented Dec 19, 2017

    The following test is failing randomly for me (python -m test test_uuid -v):
    ======================================================================
    ERROR: test_ipconfig_getnode (test.test_uuid.TestInternalsWithoutExtModule)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
  File "cpython\lib\test\test_uuid.py", line 551, in test_ipconfig_getnode
    node = self.uuid._ipconfig_getnode()
  File "cpython\lib\uuid.py", line 487, in _ipconfig_getnode
    for line in pipe:
  File "cpython\lib\encodings\cp1255.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 682: character maps to <undefined>

    This is caused by trying to decode the output of "ipconfig" using cp1255, while the output really uses cp862 and annoyingly it started to print times using the current locale (Which displays broken in the console anyhow, question mark boxes... *sigh*) in some Windows version (Using Windows 10.0.16299 ATM).

    @segevfiner segevfiner mannequin added 3.7 (EOL) end of life OS-windows type-bug An unexpected behavior, bug, or error labels Dec 19, 2017
    @eryksun
    Copy link
    Contributor

    eryksun commented Dec 19, 2017

    ipconfig uses (or defaults to) OEM encoded output when writing to a pipe or file. On the other hand, Python's TextIOWrapper defaults to ANSI (i.e. 'mbcs'). In 3.6+, uuid._ipconfig_getnode could be rewritten to call subprocess.Popen with the new 'oem' encoding.

    In Windows 10, the networking command-line utilities (e.g. ipconfig, netstat, nbtstat, ping, tracert, hostname, finger, and ftp) support an "OutputEncoding" environment variable. Its value can be set to "Unicode" (UTF-16), "UTF-8", or "Ansi". I don't think this is supported in Windows 7, however.

    @segevfiner segevfiner mannequin added the 3.8 only security fixes label Feb 10, 2018
    @zooba
    Copy link
    Member

    zooba commented Feb 10, 2018

    The patch looks good, but the rest of the code for this looks horrible(and potentially a security risk)... maybe we ought to take this fix for now but prioritise replacing this whole function with a C implementation or at least a better search for the ipconfig tool.

    @serhiy-storchaka
    Copy link
    Member

    New changeset da6c3da by Serhiy Storchaka (Segev Finer) in branch 'master':
    bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
    da6c3da

    @miss-islington
    Copy link
    Contributor

    New changeset 9b5a90b by Miss Islington (bot) in branch '3.7':
    bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
    9b5a90b

    @serhiy-storchaka
    Copy link
    Member

    New changeset c3f9d7e by Serhiy Storchaka in branch '3.6':
    [3.6] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608). (bpo-5654)
    c3f9d7e

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life 3.8 only security fixes OS-windows type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @eryksun @zooba @serhiy-storchaka @miss-islington