Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

h2py.py can fail with UnicodeDecodeError #57241

Closed
Arfrever mannequin opened this issue Sep 23, 2011 · 5 comments
Closed

h2py.py can fail with UnicodeDecodeError #57241

Arfrever mannequin opened this issue Sep 23, 2011 · 5 comments

Comments

@Arfrever
Copy link
Mannequin

Arfrever mannequin commented Sep 23, 2011

BPO 13032
Nosy @birkenfeld, @vstinner, @floppym
Files
  • h2py.py.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-07-30.15:45:43.888>
created_at = <Date 2011-09-23.08:37:40.682>
labels = []
title = 'h2py.py can fail with UnicodeDecodeError'
updated_at = <Date 2019-07-30.15:45:43.887>
user = 'https://bugs.python.org/Arfrever'

    bugs.python.org fields:

    activity = <Date 2019-07-30.15:45:43.887>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-07-30.15:45:43.888>
closer = 'vstinner'
components = ['Demos and Tools']
creation = <Date 2011-09-23.08:37:40.682>
creator = 'Arfrever'
dependencies = []
files = ['23230']
hgrepos = []
issue_num = 13032
keywords = ['patch', 'needs review']
message_count = 5.0
messages = ['144438', '145651', '159814', '348644', '348755']
nosy_count = 4.0
nosy_names = ['georg.brandl', 'vstinner', 'Arfrever', 'floppymaster']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue13032'
versions = ['Python 3.2', 'Python 3.3']
    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented Sep 23, 2011

    Tools/scripts/h2py.py fails with UnicodeDecodeError when a header file contains characters undecodable in current locale. I suggest to use binary mode. I'm attaching a patch.

    @Arfrever Arfrever mannequin added the easy label Sep 23, 2011
    @vstinner
    Copy link
    Member

    Using your patch, h2py.py skips all statements that cannot be decoded from UTF-8, whereas unpatched h2py.py accepts all statements that can be decoded from the locale encoding. I don't know if it is intentional to accept non-ASCII statements. It is maybe safer to ensure that a statement is encodable to ASCII using:

    try:
    stmt.encode('ASCII')
    exec(stmt, env)
    except:
    ...

    Anyway, I would prefer to just drop this script with all Lib/plat-*/ directoes. I reopened this topic on python-dev (I already asked when I was working on the sys.platform=="linux3" issue: bpo-12326).

    @Arfrever
    Copy link
    Mannequin Author

    Arfrever mannequin commented May 2, 2012

    UTF-8 is default encoding in Python 3, so statements with UTF-8 characters could be accepted.

    Any strings are very rare in these statements. On my system, only generated TYPES.py contains 2 strings:
    # Included from bits/select.h
    __FD_ZERO_STOS = "stosq"
    __FD_ZERO_STOS = "stosl"

    /usr/include/bits/select.h contains:
    # if __WORDSIZE == 64
    # define __FD_ZERO_STOS "stosq"
    # else
    # define __FD_ZERO_STOS "stosl"
    # endif

    @vstinner
    Copy link
    Member

    I created bpo-37704 to propose to remove Tools/scripts/h2py.py.

    @vstinner vstinner removed the easy label Jul 29, 2019
    @vstinner
    Copy link
    Member

    I just removed Tools/scripts/h2py.py in bpo-37704.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    None yet
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner