classification
Title: h2py.py can fail with UnicodeDecodeError
Type: Stage:
Components: Demos and Tools Versions: Python 3.3, Python 3.2
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, floppymaster, georg.brandl, haypo
Priority: normal Keywords: easy, needs review, patch

Created on 2011-09-23 08:37 by Arfrever, last changed 2012-05-02 18:46 by Arfrever.

Files
File name Uploaded Description Edit
h2py.py.patch Arfrever, 2011-09-23 08:37
Messages (3)
msg144438 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2011-09-23 08:37
Tools/scripts/h2py.py fails with UnicodeDecodeError when a header file contains characters undecodable in current locale. I suggest to use binary mode. I'm attaching a patch.
msg145651 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-10-16 23:29
Using your patch, h2py.py skips all statements that cannot be decoded from UTF-8, whereas unpatched h2py.py accepts all statements that can be decoded from the locale encoding. I don't know if it is intentional to accept non-ASCII statements. It is maybe safer to ensure that a statement is encodable to ASCII using:

try:
    stmt.encode('ASCII')
    exec(stmt, env)
except:
    ...

Anyway, I would prefer to just drop this script with all Lib/plat-*/ directoes. I reopened this topic on python-dev (I already asked when I was working on the sys.platform=="linux3" issue: #12326).
msg159814 - (view) Author: Arfrever Frehtes Taifersar Arahesis (Arfrever) * Date: 2012-05-02 18:46
UTF-8 is default encoding in Python 3, so statements with UTF-8 characters could be accepted.

Any strings are very rare in these statements. On my system, only generated TYPES.py contains 2 strings:
# Included from bits/select.h
__FD_ZERO_STOS = "stosq"
__FD_ZERO_STOS = "stosl"

/usr/include/bits/select.h contains:
# if __WORDSIZE == 64
#  define __FD_ZERO_STOS "stosq"
# else
#  define __FD_ZERO_STOS "stosl"
# endif
History
Date User Action Args
2012-05-02 18:46:02Arfreversetmessages: + msg159814
2012-05-02 15:30:27floppymastersetnosy: + floppymaster
2011-10-16 23:29:45hayposetmessages: + msg145651
2011-09-23 08:38:29Arfreversetkeywords: + easy
2011-09-23 08:37:40Arfrevercreate