Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 461: Inconsistency between str and bytes formatting of integers #67654

Closed
serhiy-storchaka opened this issue Feb 15, 2015 · 18 comments
Closed
Assignees
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@serhiy-storchaka
Copy link
Member

BPO 23466
Nosy @vstinner, @ethanfurman, @serhiy-storchaka, @wm75
Files
  • bytes_int_format.patch
  • bytes_format_overflow.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2015-04-03.17:56:55.526>
    created_at = <Date 2015-02-15.18:21:46.822>
    labels = ['interpreter-core', 'type-bug']
    title = 'PEP 461: Inconsistency between str and bytes formatting of integers'
    updated_at = <Date 2015-04-04.14:31:51.010>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2015-04-04.14:31:51.010>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2015-04-03.17:56:55.526>
    closer = 'serhiy.storchaka'
    components = ['Interpreter Core']
    creation = <Date 2015-02-15.18:21:46.822>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = ['38156', '38812']
    hgrepos = []
    issue_num = 23466
    keywords = ['patch']
    message_count = 18.0
    messages = ['236056', '236113', '239157', '239191', '239553', '239561', '239569', '239922', '239933', '239941', '239955', '239958', '239962', '240011', '240018', '240070', '240077', '240078']
    nosy_count = 5.0
    nosy_names = ['vstinner', 'ethan.furman', 'python-dev', 'serhiy.storchaka', 'wolma']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue23466'
    versions = ['Python 3.5']

    @serhiy-storchaka
    Copy link
    Member Author

    PEP-461 says that all numeric bytes formatting codes will work as they do for str. In particular b"%x" % val is equivalent to ("%x" % val).encode("ascii"). But this is wrong with current implementation:

    >>> '%x' % 3.14
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: %x format: an integer is required, not float
    >>> b'%x' % 3.14
    b'3'

    The same is for %X, %o and %c.

    Raising TypeError on non-integer input to %c, %o, %x, and %X was added in bpo-19995.

    @serhiy-storchaka serhiy-storchaka added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Feb 15, 2015
    @vstinner vstinner changed the title Inconsistency between str and bytes formatting of integers PEP 461: Inconsistency between str and bytes formatting of integers Feb 15, 2015
    @serhiy-storchaka
    Copy link
    Member Author

    Here is a patch.

    The difference between string and bytes formatting is that %c with out of the range integer raises OverflowError for str and TypeError for bytes. May be ValueError is more suitable in both cases.

    @serhiy-storchaka
    Copy link
    Member Author

    Ethan?

    @vstinner
    Copy link
    Member

    The patch looks good to me, except of a question added on the review.

    @ethanfurman
    Copy link
    Member

    Patch looks good.

    Changing the raised exceptions to ValueError would require deprecation periods.

    @vstinner
    Copy link
    Member

    Changing the raised exceptions to ValueError would require deprecation
    periods.

    bytes%args is not a new feature of python 3.5? It sounds strange to
    deprecate a part of a new feature.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Mar 30, 2015

    New changeset cb96fd376baa by Serhiy Storchaka in branch 'default':
    Issue bpo-23466: %c, %o, %x, and %X in bytes formatting now raise TypeError on
    https://hg.python.org/cpython/rev/cb96fd376baa

    @ethanfurman
    Copy link
    Member

    It's a new feature for 3.5 that is partly responsible for compatibility with 2.7 code.

    2.7 raises Overflow error, so 3.5 should also (for out of range values -- wrong value types raise TypeError).

    @ethanfurman
    Copy link
    Member

    b'%c' is still raising a TypeError. The error message is fine ("%c requires an integer in range(256) or a single byte") but it should be an OverflowError for backwards compatibility.

    @ethanfurman ethanfurman reopened this Apr 2, 2015
    @serhiy-storchaka
    Copy link
    Member Author

    OverflowError is for platform limitations (such as the size of machine word or addressed space). When limits are well defined and platform-independent, ValueError or may be TypeError are considered as better types. It would be better to change OverflowError to ValueError or TypeError in formatting.

    @serhiy-storchaka
    Copy link
    Member Author

    Here is a patch that makes bytes formatting raise an OverflowError if integer argument of %c is out of range.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 3, 2015

    b'%c' is still raising a TypeError. The error message is fine ("%c requires an integer in range(256) or a single byte") but it should be an OverflowError for backwards compatibility.

    I don't understand why you care so much on the exact exception. It doesn't look right to me to rely on the *exact* exception raised by "%c" % arg. It's an obvious bug in the application.

    Sometimes, you may want to be extra safe and catch exception while formating a message. The logging module does this. But the logging doesn't care of the exact exception, it uses a generic "except Except:" in StreamHandler.emit():

        def emit(self, record):
            try:
                msg = self.format(record)
                stream = self.stream
                stream.write(msg)
                stream.write(self.terminator)
                self.flush()
            except Exception:
                self.handleError(record)

    IMO b"%c" % int must raise ValueError, not OverflowError, if the value is not in the range 0..255.

    @serhiy-storchaka
    Copy link
    Member Author

    See also bpo-18184.

    @ethanfurman
    Copy link
    Member

    Looks good, thanks Serhiy.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 3, 2015

    New changeset 313fd1c819c5 by Serhiy Storchaka in branch 'default':
    Issue bpo-23466: Raised OverflowError if %c argument is out of range.
    https://hg.python.org/cpython/rev/313fd1c819c5

    @wm75
    Copy link
    Mannequin

    wm75 mannequin commented Apr 4, 2015

    the new test:

    test_exc('%x', '1', TypeError, "%x format: a number is required, not str")

    expects the wrong error message.

    python -m unittest -v test.test_format

    ...
    '%x' % '1' works? ... no
    Unexpected <class 'TypeError'> : '%x format: an integer is required, not str'
    ...

    -> it's "an integer", not "a number"

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 4, 2015

    New changeset 11e6986c794d by Serhiy Storchaka in branch 'default':
    Issue bpo-23466: Fixed expected error message in test_format.
    https://hg.python.org/cpython/rev/11e6986c794d

    @serhiy-storchaka
    Copy link
    Member Author

    Good catch, Wolfgang!

    Definitely we should make test_format more unittest compatible.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants