Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

can't interpolate byte string with \x00 before replacement identifier #73900

Closed
NickHuber mannequin opened this issue Mar 3, 2017 · 10 comments
Closed

can't interpolate byte string with \x00 before replacement identifier #73900

NickHuber mannequin opened this issue Mar 3, 2017 · 10 comments
Assignees
@zhangyangyu
Labels
3.7 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@NickHuber
Copy link
Mannequin

NickHuber mannequin commented Mar 3, 2017

BPO 29714
Nosy @vstinner, @serhiy-storchaka, @zhangyangyu
PRs
  • bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. #499
  • [3.6] bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. #504
  • [Do Not Merge] Sample of CPython life with blurb. #703
  • [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/zhangyangyu'
closed_at = <Date 2017-03-06.10:18:22.945>
created_at = <Date 2017-03-03.19:05:28.034>
labels = ['interpreter-core', 'type-bug', '3.7']
title = "can't interpolate byte string with \\x00 before replacement identifier"
updated_at = <Date 2017-03-31.16:36:26.220>
user = 'https://bugs.python.org/NickHuber'

    bugs.python.org fields:

    activity = <Date 2017-03-31.16:36:26.220>
actor = 'dstufft'
assignee = 'xiang.zhang'
closed = True
closed_date = <Date 2017-03-06.10:18:22.945>
closer = 'xiang.zhang'
components = ['Interpreter Core']
creation = <Date 2017-03-03.19:05:28.034>
creator = 'Nick Huber'
dependencies = []
files = []
hgrepos = []
issue_num = 29714
keywords = ['3.6regression']
message_count = 10.0
messages = ['288912', '288916', '289065', '289085', '289086', '289087', '289092', '289106', '290290', '290291']
nosy_count = 4.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'xiang.zhang', 'Nick Huber']
pr_nums = ['499', '504', '703', '552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29714'
versions = ['Python 3.6', 'Python 3.7']
    @NickHuber
    Copy link
    Mannequin Author

    NickHuber mannequin commented Mar 3, 2017 

    Python 3.6.0 (default, Mar  3 2017, 00:15:36) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'a\x00%i' % 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: not all arguments converted during bytes formatting
>>> b'a%i' % 1
b'a1'
>>> b'a%i\x00' % 1
b'a1\x00'

    On python3.5, this works in all the scenarios

    Python 3.5.1 (default, Jan 14 2017, 03:58:20) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'a\x00%i' % 1
b'a\x001'
>>> b'a%i' % 1
b'a1'
>>> b'a%i\x00' % 1
b'a1\x00'

    @NickHuber NickHuber mannequin added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Mar 3, 2017
    @serhiy-storchaka
    Copy link
    Member

    Could this be due to using _PyBytesWriter?

    @serhiy-storchaka serhiy-storchaka added 3.7 (EOL) end of life type-bug An unexpected behavior, bug, or error labels Mar 3, 2017
    @zhangyangyu
    Copy link
    Member

    The problem is now _PyBytes_FormatEx uses strchr to sniff %. It should use memchr instead.

    @zhangyangyu zhangyangyu self-assigned this Mar 6, 2017
    @zhangyangyu
    Copy link
    Member

    Thanks for the report, Nick!

    @vstinner
    Copy link
    Member

    vstinner commented Mar 6, 2017

    I confirm that I introduced the regression with the commit fa7762e: "Issue bpo-25349: Optimize bytes % args using the new private _PyBytesWriter API". Sorry about that, I always forget that str*() functions should not be used in Python since we support embedded null bytes :-/ It seems like only Python 3.6.0 release is affected.

    "git tag --contains fa7762e" also returns v3.5.2, but I don't understand since I only pushed my change to the 3.6 branch...

    @vstinner
    Copy link
    Member

    vstinner commented Mar 6, 2017

    Oh, and thank you for adding new unit tests ;-) Do we need similar unit tests for str%args and str.format()?

    @zhangyangyu
    Copy link
    Member

    I think no. String is not affected now and its code uses related macros so I don't think it could suffer possible regression.

    @vstinner
    Copy link
    Member

    vstinner commented Mar 6, 2017

    Xiang Zhang added the comment:

    I think no. String is not affected now and its code uses related macros so I don't think it could suffer possible regression.

    Regressions are not something "expected", shit happens, all the time.
    Unexpected corner cases are common :-)

    @zhangyangyu
    Copy link
    Member

    New changeset df6d7b4 by Xiang Zhang in branch '3.6':
    [3.6] bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. (GH-504)
    df6d7b4

    @zhangyangyu
    Copy link
    Member

    New changeset b76ad51 by Xiang Zhang in branch 'master':
    bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. (GH-499)
    b76ad51

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @zhangyangyu zhangyangyu

    Labels
    3.7 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @vstinner @serhiy-storchaka @zhangyangyu