classification
Title: can't interpolate byte string with \x00 before replacement identifier
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: xiang.zhang Nosy List: Nick Huber, serhiy.storchaka, vstinner, xiang.zhang
Priority: normal Keywords: 3.6regression

Created on 2017-03-03 19:05 by Nick Huber, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 499 merged xiang.zhang, 2017-03-06 07:57
PR 504 merged xiang.zhang, 2017-03-06 09:28
PR 703 larry, 2017-03-17 21:00
PR 552 closed dstufft, 2017-03-31 16:36
Messages (10)
msg288912 - (view) Author: Nick Huber (Nick Huber) Date: 2017-03-03 19:05
Python 3.6.0 (default, Mar  3 2017, 00:15:36) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'a\x00%i' % 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: not all arguments converted during bytes formatting
>>> b'a%i' % 1
b'a1'
>>> b'a%i\x00' % 1
b'a1\x00'


On python3.5, this works in all the scenarios

Python 3.5.1 (default, Jan 14 2017, 03:58:20) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'a\x00%i' % 1
b'a\x001'
>>> b'a%i' % 1
b'a1'
>>> b'a%i\x00' % 1
b'a1\x00'
msg288916 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-03 19:15
Could this be due to using _PyBytesWriter?
msg289065 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-06 05:56
The problem is now _PyBytes_FormatEx uses strchr to sniff %. It should use memchr instead.
msg289085 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-06 10:18
Thanks for the report, Nick!
msg289086 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-06 10:27
I confirm that I introduced the regression with the commit fa7762ec066aa3632a25b6a52bb7597b8f17c2f3: "Issue #25349: Optimize bytes % args using the new private _PyBytesWriter API". Sorry about that, I always forget that str*() functions should not be used in Python since we support embedded null bytes :-/ It seems like only Python 3.6.0 release is affected.

"git tag --contains fa7762ec066aa3632a25b6a52bb7597b8f17c2f3" also returns v3.5.2, but I don't understand since I only pushed my change to the 3.6 branch...
msg289087 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-06 10:27
Oh, and thank you for adding new unit tests ;-) Do we need similar unit tests for str%args and str.format()?
msg289092 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-06 11:08
I think no. String is not affected now and its code uses related macros so I don't think it could suffer possible regression.
msg289106 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-06 14:31
Xiang Zhang added the comment:
> I think no. String is not affected now and its code uses related macros so I don't think it could suffer possible regression.

Regressions are not something "expected", shit happens, all the time.
Unexpected corner cases are common :-)
msg290290 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-24 22:50
New changeset df6d7b406f3d1b2e4e2014751bfa25574c4df222 by Xiang Zhang in branch '3.6':
[3.6] bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. (GH-504)
https://github.com/python/cpython/commit/df6d7b406f3d1b2e4e2014751bfa25574c4df222
msg290291 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-24 22:50
New changeset b76ad5121e2cfa89d6476d700cbcb65b7ffc39ac by Xiang Zhang in branch 'master':
bpo-29714:  Fix a regression that bytes format may fail when containing zero bytes inside. (GH-499)
https://github.com/python/cpython/commit/b76ad5121e2cfa89d6476d700cbcb65b7ffc39ac
History
Date User Action Args
2017-07-17 12:55:04r.david.murraylinkissue30943 superseder
2017-03-31 16:36:26dstufftsetpull_requests: + pull_request998
2017-03-28 14:43:07xiang.zhanglinkissue29934 superseder
2017-03-24 22:50:43xiang.zhangsetmessages: + msg290291
2017-03-24 22:50:32xiang.zhangsetmessages: + msg290290
2017-03-17 21:00:35larrysetpull_requests: + pull_request616
2017-03-06 14:31:38vstinnersetmessages: + msg289106
2017-03-06 11:08:22xiang.zhangsetmessages: + msg289092
2017-03-06 10:27:59vstinnersetmessages: + msg289087
2017-03-06 10:27:23vstinnersetmessages: + msg289086
2017-03-06 10:18:22xiang.zhangsetstatus: open -> closed
resolution: fixed
messages: + msg289085

stage: needs patch -> resolved
2017-03-06 09:28:47xiang.zhangsetpull_requests: + pull_request412
2017-03-06 07:57:46xiang.zhangsetpull_requests: + pull_request407
2017-03-06 05:56:33xiang.zhangsetassignee: xiang.zhang

messages: + msg289065
nosy: + xiang.zhang
2017-03-03 19:15:41serhiy.storchakasettype: behavior
versions: + Python 3.7
keywords: + 3.6regression
nosy: + vstinner, serhiy.storchaka

messages: + msg288916
stage: needs patch
2017-03-03 19:05:28Nick Hubercreate