classification
Title: %b format for bytes does not support objects that follow the buffer protocol
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, ethan.furman, serhiy.storchaka, skrah, xiang.zhang
Priority: normal Keywords:

Created on 2016-12-02 02:30 by belopolsky, last changed 2017-03-24 22:20 by xiang.zhang. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 546 merged xiang.zhang, 2017-03-07 10:34
PR 664 merged xiang.zhang, 2017-03-14 07:12
PR 703 larry, 2017-03-17 21:00
Messages (12)
msg282215 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-12-02 02:30
Python 3.7.0a0 (default:be70d64bbf88, Dec  1 2016, 21:21:25)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from array import array
>>> a = array('B', [1, 2])
>>> b'%b' % a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %b requires bytes, or an object that implements __bytes__, not 'array.array'
>>> m = memoryview(a)
>>> b'%b' % m
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: %b requires bytes, or an object that implements __bytes__, not 'memoryview'

Accorfing to documentation [1] objects that follow the buffer protocol should be supported.  Both array.array and memoryview follow the buffer protocol.

[1]: https://docs.python.org/3/library/stdtypes.html#printf-style-bytes-formatting


See also issue 20284 and PEP 461.
msg289159 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-07 11:16
printf-style bytes formatting was added mainly for increasing compatibility with Python 2. It was restricted to support mostly features existing in Python 2.

'%s' formatting in Python 3 supports bytes-like objects partially:

>>> b'%s' % array('B', [1, 2])
"array('B', [1, 2])"
>>> b'%s' % buffer(array('B', [1, 2]))
'\x01\x02'
>>> b'%s' % memoryview(array('B', [1, 2]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot make memory view because object does not have the buffer interface
>>> b'%s' % bytearray(b'abc')
'abc'
>>> b'%s' % buffer(bytearray(b'abc'))
'abc'
>>> b'%s' % memoryview(bytearray(b'abc'))
'<memory at 0xb70902ac>'

I don't know whether there is a need of supporting the buffer protocol in printf-style bytes formatting. bytearray is already supported, buffer() doesn't exist in Python 3, memoryview() is not supported in Python 2. Seems this doesn't add anything for increasing the compatibility.
msg289172 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-07 16:38
Isn't this a discussed behaviour that is explicitly documented in PEP 461?
msg289173 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-03-07 16:43
For '%b', it looks like the PEP supports it. I didn't follow the PEP discussions, I think Ethan will know more.
msg289520 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-13 02:08
What's your opinions Alexander and Ethan?
msg289533 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-13 06:22
Sometimes the implementation can expose drawbacks of initial design. I don't know whether there was good reason for omitting the support of the buffer protocol (in that case the PEP should be updated) or this is just an oversign. We should ask Ethan about this.

The change proposed by Xiang looks correct, but not very efficient. It makes one redundant copy of the data. More efficient implementation will complicate the code, and that can hit the performance of other cases.
msg289540 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2017-03-13 15:09
I suspect it was a simple oversight, and should be added now.  Since it's been missing for so long I think we should put it in 3.7, maybe put it in 3.6 (maybe not, since it has two point releases out now), but definitely not in 3.5.
msg289542 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2017-03-13 17:16
@xiang.zhang - I am the OP for this issue, so naturally I expect this to be fixed.  I have a work-around in place for my own code, so I have no opinion on the particular versions.  I guess the normal policy on bug fixes should apply.
msg289570 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-14 07:22
Following example copies the entire buffer object while copying only smart part is needed:

    m = memoryview(b'x'*10**6)
    b'%.100b' % m

I don't know whether this is important use case that is worth an optimization. The workaround is using slicing rather than truncating in format:

    b'%b' % m[:100]

Or in the case of general buffer object:

    b'%b' % memoryview(m).cast('B')[:100]

But in that case it is not hard to add an explicit conversion to bytes.

    b'%b' % bytes(memoryview(m).cast('B')[:100])
msg289571 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-14 07:36
I committed the suboptimal patch. I close this issue now and if there is any enhancement solution, let's make it another issue. Thank you all.
msg290187 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-24 22:20
New changeset faa2cc63e45bc7d7ffab84bebe5a9f4fe065bd96 by Xiang Zhang in branch '3.6':
bpo-28856: Let %b format for bytes support objects that follow the buffer protocol  (GH-664)
https://github.com/python/cpython/commit/faa2cc63e45bc7d7ffab84bebe5a9f4fe065bd96
msg290188 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-24 22:20
New changeset 7e2a54cdd977078b40b82182e46b201f8163f659 by Xiang Zhang in branch 'master':
bpo-28856: Let %b format for bytes support objects that follow the buffer protocol (GH-546)
https://github.com/python/cpython/commit/7e2a54cdd977078b40b82182e46b201f8163f659
History
Date User Action Args
2017-03-24 22:20:23xiang.zhangsetmessages: + msg290188
2017-03-24 22:20:14xiang.zhangsetmessages: + msg290187
2017-03-17 21:00:32larrysetpull_requests: + pull_request582
2017-03-14 07:36:34xiang.zhangsetstatus: open -> closed
versions: - Python 3.5
messages: + msg289571

resolution: fixed
stage: patch review -> resolved
2017-03-14 07:22:02serhiy.storchakasetmessages: + msg289570
2017-03-14 07:12:12xiang.zhangsetpull_requests: + pull_request546
2017-03-13 17:16:13belopolskysetmessages: + msg289542
2017-03-13 15:09:05ethan.furmansetmessages: + msg289540
2017-03-13 06:22:05serhiy.storchakasetmessages: + msg289533
2017-03-13 02:08:24xiang.zhangsetmessages: + msg289520
2017-03-07 16:43:30skrahsetmessages: + msg289173
2017-03-07 16:38:52xiang.zhangsetmessages: + msg289172
2017-03-07 11:16:23serhiy.storchakasetnosy: + skrah
messages: + msg289159
2017-03-07 10:34:16xiang.zhangsetpull_requests: + pull_request449
2017-03-07 10:32:04xiang.zhangsetstage: test needed -> patch review
components: + Interpreter Core
versions: + Python 3.5, Python 3.6, Python 3.7
2016-12-02 11:20:51xiang.zhangsetnosy: + xiang.zhang
2016-12-02 07:04:08serhiy.storchakasetnosy: + ethan.furman, serhiy.storchaka
2016-12-02 02:30:44belopolskycreate