This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: UnicodeDecodeError in grep_headers_for
Type: compile error Stage: resolved
Components: Build Versions: Python 3.10, Python 3.9, Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: epaine, ronaldoussoren, yan12125
Priority: normal Keywords: patch

Created on 2020-11-13 17:59 by epaine, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23279 merged ronaldoussoren, 2020-11-14 11:20
Messages (6)
msg380913 - (view) Author: E. Paine (epaine) * Date: 2020-11-13 17:59
When compiling the master branch (i.e. running 'make'), I get a UnicodeDecodeError as follows:
Traceback (most recent call last):
  File "/home/elisha/Documents/Python/cp0/cpython/./", line 2619, in <module>
  File "/home/elisha/Documents/Python/cp0/cpython/./", line 2589, in main
    setup(# PyPI Metadata (PEP 301)
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/", line 148, in setup
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/", line 966, in run_commands
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/", line 985, in run_command
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/command/", line 135, in run
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/", line 313, in run_command
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/", line 985, in run_command
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/distutils/command/", line 340, in run
  File "/home/elisha/Documents/Python/cp0/cpython/./", line 471, in build_extensions
  File "/home/elisha/Documents/Python/cp0/cpython/./", line 1825, in detect_modules
  File "/home/elisha/Documents/Python/cp0/cpython/./", line 2205, in detect_ctypes
    if grep_headers_for('ffi_prep_cif_var', ffi_headers):
  File "/home/elisha/Documents/Python/cp0/cpython/./", line 246, in grep_headers_for
    if function in
  File "/home/elisha/Documents/Python/cp0/cpython/Lib/", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 4210: invalid start byte

The problematic file it is trying to read is /usr/include/OMX_Other.h which is part of the libomxil-bellagio package (a copy of this package can be downloaded from More specifically, there are several characters in the comments which cannot be decoded correctly (the first of these is on line 93).

The fix is a very simple one and is just to add errors='replace' to line 244 of (I cannot see this having any ill-effects).

I couldn't find who to nosy for this so apologies about that.
msg380955 - (view) Author: (yan12125) * Date: 2020-11-14 04:59
I can also confirm the issue on our Arch Linux server [1]. The problematic file is also /usr/include/OMX_Other.h.

Looks like it is a regression from ( Ronald Oussoren, mind to have a look?

msg380965 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-14 09:34
That's annoying. A quick workaround is to patch and add "encoding='latin1'" to the arguments of open.

I'll look into a better fix later this weekend.
msg380971 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-14 11:20
I've created PR. Could you please check if that fixes the problem?
msg380978 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-14 15:07
New changeset 7a27c7ed4b2b45bb9ea27d3f5c4f423495d6e939 by Ronald Oussoren in branch 'master':
bpo-42351: Avoid error when opening header with non-UTF8 encoding (GH-23279)
msg380979 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-11-14 15:08
Thanks for testing!
Date User Action Args
2022-04-11 14:59:38adminsetgithub: 86517
2020-11-14 15:09:22epainesetstatus: open -> closed
2020-11-14 15:08:32ronaldoussorensetresolution: fixed
messages: + msg380979
stage: patch review -> resolved
2020-11-14 15:07:51ronaldoussorensetmessages: + msg380978
2020-11-14 11:20:51ronaldoussorensetmessages: + msg380971
2020-11-14 11:20:07ronaldoussorensetkeywords: + patch
stage: patch review
pull_requests: + pull_request22173
2020-11-14 09:34:25ronaldoussorensetmessages: + msg380965
2020-11-14 04:59:24yan12125setnosy: + ronaldoussoren, yan12125
messages: + msg380955
2020-11-13 17:59:24epainecreate