classification
Title: Documentation is unclear how "y*" and "y#" format units vary
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, indygreg, martin.panter, skrah
Priority: normal Keywords:

Created on 2017-06-10 18:16 by indygreg, last changed 2017-06-17 03:18 by martin.panter.

Messages (5)
msg295649 - (view) Author: Gregory Szorc (indygreg) * Date: 2017-06-10 18:16
https://github.com/indygreg/python-zstandard/issues/26 is a bug report against python-zstandard that a C API using the "y#" format unit is unable to accept a memoryview instance constructed from a bytes literal. It fails with "TypeError: argument 1 must be read-only bytes-like object, not memoryview".

I understand why the "y*" format unit and the buffer protocol are a superior API. In hindsight, I should have used the buffer protocol from the beginning in python-zstandard. However, this decision was primarily influenced because the docs aren't clear about the apparent limitations of "y#" compared to "y*" and I believed "y#" would get me what I wanted (pointer to read-only memory) without having to be burdened by the complexity of the buffer protocol in my C code.

So, docs issue #1 is that the limitations of "y#" compared to "y*" aren't immediately obvious when reading their respective sections in https://docs.python.org/3/c-api/arg.html#strings-and-buffers. To their credit, the docs for "y*" do have a bold recommendation to use it. But what's missing is the critical "why." There is also a paragraph above the format unit list explaining "bytes-like object" but this is detached from the definitions for "y#" and "y*" so it is easy to pass over.

Issue #2 (which may be an implementation bug) is why "y#" isn't accepting a memoryview constructed from bytes. The docs for "y#" say it accepts a "read-only bytes-like object," which is defined at https://docs.python.org/3/glossary.html#term-bytes-like-object. And the paragraphs there explicitly state that a memoryview of an immutable bytes is in fact a "read-only bytes-like object." So I have no clue why "y#" is refusing such an object.

I'll gladly author a documentation fix. However, I'm not sure what to say because I don't understand why "y#" isn't working in this memoryview(bytes) case and whether that issue applies to other types.
msg296010 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-06-14 15:09
Out of curiosity:

Is the 3.2 documentation clearer?

https://docs.python.org/3.2/c-api/arg.html#strings-and-buffers


Lately we have a lot of churn in the docs, not necessarily written by subject experts.
msg296116 - (view) Author: Gregory Szorc (indygreg) * Date: 2017-06-15 17:21
IMO I don't find the 3.2 docs more useful. Specifically, the behavior for memoryview is still unclear.
msg296182 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2017-06-16 12:23
Okay thanks, it's good to hear what others think about the docs.

> So I have no clue why "y#" is refusing such an object.

"y#" is refusing memoryview(bytes) because "y#" only allows objects without a releasebufferproc and memoryview itself always has one.


I wonder if we could simply use the cleanup() solution also for "y#", but I have to look closer at this.
msg296229 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2017-06-17 03:18
Issue 24009 proposes deprecating y# (among other units).

IMO the documentation and error message aren’t specific enough regarding the reference to “bytes-like”.
History
Date User Action Args
2017-06-17 03:18:44martin.pantersetnosy: + martin.panter
messages: + msg296229
2017-06-16 12:23:48skrahsetmessages: + msg296182
2017-06-15 17:21:48indygregsetmessages: + msg296116
2017-06-14 15:09:15skrahsetmessages: + msg296010
2017-06-14 14:17:34pitrousetnosy: + skrah
2017-06-11 04:06:17serhiy.storchakasettitle: Documentation is unclear how "y*" and "y#" formit units vary -> Documentation is unclear how "y*" and "y#" format units vary
2017-06-10 18:39:57pitrousetstage: needs patch
type: enhancement -> behavior
versions: + Python 3.6, Python 3.7
2017-06-10 18:16:12indygregcreate