This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: h2py.py can fail with UnicodeDecodeError
Type: Stage: resolved
Components: Demos and Tools Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, floppymaster, georg.brandl, vstinner
Priority: normal Keywords: needs review, patch

Created on 2011-09-23 08:37 by Arfrever, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
h2py.py.patch Arfrever, 2011-09-23 08:37
Messages (5)
msg144438 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2011-09-23 08:37
Tools/scripts/h2py.py fails with UnicodeDecodeError when a header file contains characters undecodable in current locale. I suggest to use binary mode. I'm attaching a patch.
msg145651 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-10-16 23:29
Using your patch, h2py.py skips all statements that cannot be decoded from UTF-8, whereas unpatched h2py.py accepts all statements that can be decoded from the locale encoding. I don't know if it is intentional to accept non-ASCII statements. It is maybe safer to ensure that a statement is encodable to ASCII using:

try:
    stmt.encode('ASCII')
    exec(stmt, env)
except:
    ...

Anyway, I would prefer to just drop this script with all Lib/plat-*/ directoes. I reopened this topic on python-dev (I already asked when I was working on the sys.platform=="linux3" issue: #12326).
msg159814 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * (Python triager) Date: 2012-05-02 18:46
UTF-8 is default encoding in Python 3, so statements with UTF-8 characters could be accepted.

Any strings are very rare in these statements. On my system, only generated TYPES.py contains 2 strings:
# Included from bits/select.h
__FD_ZERO_STOS = "stosq"
__FD_ZERO_STOS = "stosl"

/usr/include/bits/select.h contains:
# if __WORDSIZE == 64
#  define __FD_ZERO_STOS "stosq"
# else
#  define __FD_ZERO_STOS "stosl"
# endif
msg348644 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-29 12:00
I created bpo-37704 to propose to remove Tools/scripts/h2py.py.
msg348755 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-30 15:45
I just removed Tools/scripts/h2py.py in bpo-37704.
History
Date User Action Args
2022-04-11 14:57:21adminsetgithub: 57241
2019-07-30 15:45:43vstinnersetstatus: open -> closed
resolution: wont fix
messages: + msg348755

stage: resolved
2019-07-29 12:00:39vstinnersetkeywords: - easy

messages: + msg348644
2012-05-02 18:46:02Arfreversetmessages: + msg159814
2012-05-02 15:30:27floppymastersetnosy: + floppymaster
2011-10-16 23:29:45vstinnersetmessages: + msg145651
2011-09-23 08:38:29Arfreversetkeywords: + easy
2011-09-23 08:37:40Arfrevercreate