classification
Title: Custom iterator to memoryview - performance improvement
Type: Stage: resolved
Components: Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: corona10, dxflores, gvanrossum, vstinner
Priority: normal Keywords: patch

Created on 2020-09-06 09:06 by dxflores, last changed 2020-09-08 10:11 by dxflores. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22119 merged dxflores, 2020-09-06 09:20
Messages (6)
msg376460 - (view) Author: Diogo Flores (dxflores) * Date: 2020-09-06 09:06
Hi,

Here's a patch that adds an iterator to memoryview, which makes iterating over memoryview objs considerably faster.

The following values result from a compilation with debug ON.

------------------------------------------------------------------------

Without patch:

cpython: ./python.exe -m timeit -s "a=memoryview(b'x'*1000000)" "for x in a: pass"
5 loops, best of 5: 98 msec per loop

cpython: ./python.exe -m timeit -s "a=b'x'*1000000" "for x in a: pass"
5 loops, best of 5: 68.6 msec per loop


With patch:

cpython: ./python.exe -m timeit -s "a=memoryview(b'x'*1000000)" "for x in a: pass"
5 loops, best of 5: 68.1 msec per loop

cpython: ./python.exe -m timeit -s "a=b'x'*1000000" "for x in a: pass"
5 loops, best of 5: 70 msec per loop

------------------------------------------------------------------------

Please let me know your thoughts regarding it.

Diogo
msg376549 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-09-08 07:28
New changeset b0ac5d75a59c356c44cfc9b25077da3049dd16e9 by dxflores in branch 'master':
bpo-41732: add iterator to memoryview (GH-22119)
https://github.com/python/cpython/commit/b0ac5d75a59c356c44cfc9b25077da3049dd16e9
msg376550 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2020-09-08 08:21
It would have been nice to have had a chance to look at this, but
apparently those days are over.
msg376558 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-09-08 09:19
Thanks, Diogo for working on this issue.
And thank you for the mentoring guido!
msg376559 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-08 09:22
Thanks Diogo Flores, it's a nice optimization!


Stefan Krah: "It would have been nice to have had a chance to look at this, but apparently those days are over."

The PR has been approved by two core developers. It's a nice optimization, I don't see what is the problem here.
msg376561 - (view) Author: Diogo Flores (dxflores) * Date: 2020-09-08 10:11
Thank you Dong-hee Na and Victor.

@Stefan Krah: I am sorry that you didn't had the chace to try out the patch before it got merged, however iff you are curious about the any aspect of the implementation (or the motivation behind it) I am happy to discuss it in depth.
History
Date User Action Args
2020-09-08 10:11:31dxfloressetmessages: + msg376561
2020-09-08 09:22:25vstinnersetnosy: + vstinner
messages: + msg376559
2020-09-08 09:19:26corona10setstatus: open -> closed

nosy: + gvanrossum
messages: + msg376558

resolution: fixed
stage: patch review -> resolved
2020-09-08 08:21:48skrahsetnosy: - skrah
2020-09-08 08:21:34skrahsetmessages: + msg376550
2020-09-08 07:28:59corona10setnosy: + corona10
messages: + msg376549
2020-09-06 09:20:15dxfloressetkeywords: + patch
stage: patch review
pull_requests: + pull_request21203
2020-09-06 09:19:12xtreaksetnosy: + skrah
2020-09-06 09:06:52dxflorescreate