classification
Title: Accept mutable bytes-like objects
Type: enhancement Stage: resolved
Components: Extension Modules Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Arfrever, ethan.furman, josh.r, martin.panter, pitrou, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: needs review, patch

Created on 2014-12-06 14:25 by serhiy.storchaka, last changed 2015-03-20 10:01 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
accept_mutable_buffers.patch serhiy.storchaka, 2014-12-06 14:25 review
accept_mutable_buffers_2.patch serhiy.storchaka, 2015-03-19 11:17 review
Messages (14)
msg232244 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-12-06 14:25
Some builtins accept only read-only bytes-like objects (PyArg_Parse format codes "s#", "z#", "y", and "y#"). Proposed patch makes them accepting also mutable bytes-like objects such as bytearray.

I'm not sure that all these changes are useful, but in some cases this can get rid of copying binary data created in bytearray (e.g. read by readinto()).
msg232246 - (view) Author: Josh Rosenberg (josh.r) * Date: 2014-12-06 15:55
In the event of calls back into Python code in multithreaded execution (or GIL release), this would mean you no longer have guarantees as to the contents (or even the validity) of the pointer you get back. I'd think the only safe way to accept mutable buffers would be to use the s*, z*, y* codes, which lock the buffer to prevent resize/destruction. Do we want to open segfault vulnerabilities in arbitrary functions?
msg232252 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-12-06 20:51
That is was the patch does. Convert from s# to s* etc.

See also issue22896 about potential bugs with the use of pointers to unlocked 
buffers.
msg232255 - (view) Author: Josh Rosenberg (josh.r) * Date: 2014-12-06 22:22
Ah, sorry. Should have examined patch. I thought you were making a change to the behavior of s#, z#, y and y#, not converting actual uses of them. Again, sorry.
msg232264 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-12-07 01:40
+1 on the principle. I haven't looked at the patch.
msg233438 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-01-05 05:37
If this goes ahead, it would be nice adding notes to the documentation saying that bytearray() or whatever was previously not supported. There are APIs in Python 2.6 that had similar treatment with no documentation updates, and I keep being bitten by it.
msg238502 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-19 11:17
Added versionchanged directives and fixed other documentation.
msg238598 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-20 01:41
I had a closer look at the changes, and most of them seem good. However I think I found one leak in fcntl(); see Reitveld.

I noticed there is no test for “ossaudiodev”. Would that be too hard, or is it just an oversight?
msg238607 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-20 06:53
Thank you for your review Martin. Yes, the problem with fcntl() is not so easy and I'll left it for the separate issue.

> I noticed there is no test for “ossaudiodev”. Would that be too hard, or is it just an oversight?

It is hard because test_ossaudiodev is not designed to just apply the test with different type of data, and this test doesn't work on my computer at all (no /dev/dsp).
msg238608 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-20 07:04
New changeset 17c77938c4e2 by Serhiy Storchaka in branch 'default':
Issue #23001: Few functions in modules mmap, ossaudiodev, socket, ssl, and
https://hg.python.org/cpython/rev/17c77938c4e2
msg238628 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-20 09:24
New changeset 11548e1ac920 by Victor Stinner in branch 'default':
Issue #23001: Fix typo
https://hg.python.org/cpython/rev/11548e1ac920
msg238630 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-20 09:39
New changeset d478a2a5738a by Victor Stinner in branch 'default':
Issue #23709, #23001: ossaudiodev now uses Py_ssize_t for sizes instead of int
https://hg.python.org/cpython/rev/d478a2a5738a
msg238633 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-20 09:55
Thanks Victor.
msg238635 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-20 10:01
> Thanks Victor.

You're welcome, I just worked recently on ossaudiodev for _Py_read/write functions functions. But I'm not able to run ossaudiodev neither :-( I hope that the test runs on some buildbots (FreeBSD?).
History
Date User Action Args
2015-03-20 10:01:21vstinnersetnosy: + vstinner
messages: + msg238635
2015-03-20 09:55:54serhiy.storchakasetmessages: + msg238633
2015-03-20 09:39:49python-devsetmessages: + msg238630
2015-03-20 09:24:35python-devsetmessages: + msg238628
2015-03-20 07:06:26serhiy.storchakasetstatus: open -> closed
assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2015-03-20 07:04:51python-devsetnosy: + python-dev
messages: + msg238608
2015-03-20 06:53:28serhiy.storchakasetmessages: + msg238607
2015-03-20 01:41:30martin.pantersetmessages: + msg238598
2015-03-19 11:17:55serhiy.storchakasetfiles: + accept_mutable_buffers_2.patch

messages: + msg238502
2015-01-05 05:37:52martin.pantersetmessages: + msg233438
2014-12-18 23:49:04martin.pantersetnosy: + martin.panter
2014-12-18 15:40:46serhiy.storchakasetkeywords: + needs review
2014-12-07 01:40:28pitrousetnosy: + pitrou
messages: + msg232264
2014-12-06 23:52:56Arfreversetnosy: + Arfrever
2014-12-06 22:22:28josh.rsetmessages: + msg232255
2014-12-06 20:51:26serhiy.storchakasetmessages: + msg232252
2014-12-06 18:49:37ethan.furmansetnosy: + ethan.furman
2014-12-06 15:55:28josh.rsetnosy: + josh.r
messages: + msg232246
2014-12-06 14:25:06serhiy.storchakacreate