This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Optimize bytearray(int) constructor to use calloc()
Type: performance Stage:
Components: Versions: Python 3.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: bmerry, pitrou, skrah, vstinner
Priority: normal Keywords:

Created on 2014-06-02 20:25 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (9)
msg219632 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-02 20:25
Python 3.5 has a new PyObject_Calloc() function which can be used for fast memory allocation of memory block initialized with zeros.

I already implemented an optimization, but Stefan Krah found issues in my change:
http://bugs.python.org/issue21233#msg217826

I reverted the optimization in the changeset dff6b4b61cac.
msg219633 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-02 20:26
Stefan also wrote:

"3) Somewhat similarly, I wonder if it was necessary to refactor
   PyBytes_FromStringAndSize(). I find the new version more difficult
   to understand."

This issue can also be addressed here.
msg219636 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-02 20:30
Stefan wrote:
"3) Somewhat similarly, I wonder if it was necessary to refactor
   PyBytes_FromStringAndSize(). I find the new version more difficult
   to understand."

Do you mean that the optimization is useless or that the implementation should be changed?
msg219672 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-06-03 09:08
Responding to a comment in issue21233:

>    Before:
>    =======
>        >>> x = bytearray(0)
>        >>> m = memoryview(x)
>        >>> x.__init__(10)
>        Traceback (most recent call last):
>          File "<stdin>", line 1, in <module>
>        BufferError: Existing exports of data: object cannot be re-sized

I don't think such use cases are supported. Generally, reinitializing an 
object by calling __init__() explicitly is not well-defined, except when 
advertised explicitly in the documentation. The only property we should 
guarantee here is that it doesn't lead to an inconsistent object state, or to 
hard crashes. Raising an exception is fine, and changing the raised
exception to another one should be fine as well.
msg223642 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-07-22 07:49
See also issue #22030 (set).
msg238434 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 13:28
I'm not interested to work on this optimization, so I just close the issue.
msg376942 - (view) Author: Bruce Merry (bmerry) * Date: 2020-09-15 15:27
Was this abandoned just because nobody had the time, or was there a problem with the approach? I independently wanted this optimisation, and have ended up implementing something very similar to what was reverted in https://hg.python.org/lookup/dff6b4b61cac.

In a benchmark that creates a large bytearray, then fills it with socket.readinto, I'm seeing a 2x performance improvement on Linux, and from some quick benchmarking it seems to be just as fast as the old code for small arrays that are allocated from the pool.
msg405945 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-11-08 12:47
I abandonned the issue because I didn't have time to work on it. If you want, you can open a new issue for that.
msg406321 - (view) Author: Bruce Merry (bmerry) * Date: 2021-11-14 13:34
> I abandonned the issue because I didn't have time to work on it. If you want, you can open a new issue for that.

If I make a pull request and run some microbenchmarks, will you (or some other core dev) have time to review it? I've had a bad experience before with a PR that I'm still unable to get reviewed after several years, so I'd like to get at least a tentative agreement before I invest time in it.
History
Date User Action Args
2022-04-11 14:58:04adminsetgithub: 65843
2021-11-14 13:34:15bmerrysetmessages: + msg406321
2021-11-08 12:47:36vstinnersetmessages: + msg405945
2020-09-15 15:27:19bmerrysetnosy: + bmerry
messages: + msg376942
2015-03-18 13:28:53vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg238434
2014-07-22 07:49:22vstinnersetmessages: + msg223642
2014-06-03 09:08:10pitrousetnosy: + pitrou
messages: + msg219672
2014-06-02 20:30:27vstinnersetmessages: + msg219636
2014-06-02 20:26:51vstinnersetmessages: + msg219633
2014-06-02 20:25:58vstinnercreate