This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pip can't handle MSVC warnings containing special characters
Type: crash Stage: resolved
Components: Windows Versions: Python 3.6
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: paul.moore Nosy List: Cutter, Marcus.Smith, dstufft, ncoghlan, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-12-17 11:59 by Cutter, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
console.txt Cutter, 2017-12-17 11:59 Console log of pylint installation
Messages (2)
msg308485 - (view) Author: (Cutter) Date: 2017-12-17 11:59
When trying to install pylint using pip on Windows 10, the installation of wrapt (a dependency of pylint) fails because a special character in an MSVC warning can't be decoded to utf-8.

Below is the relevant part of the console output:
---
Exception:
Traceback (most recent call last):
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte
---

I changed line 73 in \pip\compat\__init__.py to: print("!! TEST !! : ", s). The full console output after that change is in the attachment. The relevant part is:

 !! TEST !! :  b'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.11.25503\\bin\\HostX86\\x64\\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\\Users\\(...)\\AppData\\Local\\Programs\\Python\\Python36\\include -IC:\\Users\\(...)\\AppData\\Local\\Programs\\Python\\Python36\\include "-IC:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.11.25503\\ATLMFC\\include" "-IC:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.11.25503\\include" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\ucrt" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\shared" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\um" "-IC:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.16299.0\\winrt" /Tcsrc/wrapt/_wrappers.c /Fobuild\\temp.win-amd64-3.6\\Release\\src/wrapt/_wrappers.obj\r\n'
 !! TEST !! :  b'_wrappers.c\r\n'
 !! TEST !! :  b"src/wrapt/_wrappers.c(195): warning C4244: 'return'\xff: conversion de 'Py_hash_t' en 'long', perte possible de donn\x82es\r\n"
 error
 Exception:
 Traceback (most recent call last):
   File "C:\Users\(...)\AppData\Local\Programs\Python\Python36\Lib\site-packages\pip\compat\__init__.py", line 74, in console_to_str
     return s.decode(sys.__stdout__.encoding)
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 51: invalid start byte

As a workaround I've changed the original line 73 in \pip\compat\__init__.py :

return s.decode(sys.__stdout__.encoding)

to:

return s.decode(sys.__stdout__.encoding, "replace")

(thanks to dieter on comp.lang.python for his help). I don't have the knowledge to propose a patch.
msg308486 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2017-12-17 12:17
This is a pip issue, not a Python issue. It's a known issue in the current version of pip, and should be fixed in the development version (via pull request https://github.com/pypa/pip/pull/4486).

If you like, you can try the development version of pip. At the moment there is no planned date for the next release of pip, but we're hoping to put together a release in the not too distant future.
History
Date User Action Args
2022-04-11 14:58:55adminsetgithub: 76531
2017-12-17 12:17:26paul.mooresetstatus: open -> closed
messages: + msg308486

assignee: paul.moore
resolution: third party
stage: resolved
2017-12-17 11:59:07Cuttercreate