Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memoryview.__len__ should raise an exception for 0d buffers #83791

Closed
EricWieser mannequin opened this issue Feb 11, 2020 · 8 comments
Closed

memoryview.__len__ should raise an exception for 0d buffers #83791

EricWieser mannequin opened this issue Feb 11, 2020 · 8 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@EricWieser
Copy link
Mannequin

EricWieser mannequin commented Feb 11, 2020

BPO 39610
Nosy @skrah, @serhiy-storchaka
PRs
  • gh-83791: Raise TypeError for len(memoryview_0d) #18463
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/skrah'
    closed_at = None
    created_at = <Date 2020-02-11.15:42:52.359>
    labels = ['interpreter-core', '3.8', 'type-bug', '3.7', '3.9']
    title = 'memoryview.__len__ should raise an exception for 0d buffers'
    updated_at = <Date 2020-09-07.17:25:22.315>
    user = 'https://bugs.python.org/EricWieser'

    bugs.python.org fields:

    activity = <Date 2020-09-07.17:25:22.315>
    actor = 'skrah'
    assignee = 'skrah'
    closed = False
    closed_date = None
    closer = None
    components = ['Interpreter Core']
    creation = <Date 2020-02-11.15:42:52.359>
    creator = 'Eric Wieser'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 39610
    keywords = ['patch']
    message_count = 7.0
    messages = ['361821', '361840', '363159', '376512', '376518', '376519', '376520']
    nosy_count = 3.0
    nosy_names = ['skrah', 'serhiy.storchaka', 'Eric Wieser']
    pr_nums = ['18463']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue39610'
    versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

    @EricWieser
    Copy link
    Mannequin Author

    EricWieser mannequin commented Feb 11, 2020

    Right now, the behavior is:

    >>> import numpy as np
    >>> arr_0d = np.array(42)
    >>> mem_0d = memoryview(arr_0d)
    >>> len(mem_0d)
    1
    >>> mem_0d[0]
    TypeError: invalid indexing of 0-dim memory

    It seems bizarre to have this object pretend to be a sequence when you ask for its length, yet not behave like one when you actually try to use this length. I'd suggest cpython should behave like numpy here, and fail:

    >>> len(arr_0d)
    TypeError: len() of unsized object

    Perhaps TypeError: cannot get length of 0-dim memory would be more appropriate as a message.

    ---

    Wasn't sure how to classify this, feel free to reclassify

    @EricWieser EricWieser mannequin added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Feb 11, 2020
    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Feb 12, 2020

    The change looks reasonable, but unfortunately this is a long-standing
    behavior that originates from before the Python-3.3 memoryview rewrite.

    It is also present in 2.7 (the previous implementation) and documented
    in 3.3:

    https://docs.python.org/3/library/stdtypes.html#memoryview

    I think I documented it because it looked strange back then, too.
    So I've to think a bit what will be affected and how this behavior
    came into place.

    @EricWieser
    Copy link
    Mannequin Author

    EricWieser mannequin commented Mar 2, 2020

    Thanks for pointing out the docs reference, I updated the patch to reword that section.

    There's a sentence right before the one you draw attention to which to me reads as another argument to change this:

    ``len(view)`` is equal to the length of :class:`~memoryview.tolist`

    On Python 2.7, this gives

    >>> len(view_0d)
    1
    >>> len(view_0d.tolist())
    NotImplementedError: tolist() only supports one-dimensional objects

    On Python 3.8 before my patch, this gives:

    >>> len(view_0d)
    1
    >>> len(view_0d.tolist())
    TypeError: object of type 'int' has no len()

    On Python 3.8, with my patch, this gives:

    >>> len(view_0d)
    TypeError: 0-dim memory has no length
    >>> len(view_0d.tolist())
    TypeError: object of type 'int' has no len()

    As I read it, only with my patch is this sentence satisfied by the implementation.

    @serhiy-storchaka
    Copy link
    Member

    But iterating it raises different type of exception, wish obscure message.

    >>> import ctypes
    >>> mem_0d = memoryview(ctypes.c_uint8(42))
    >>> list(mem_0d)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NotImplementedError: memoryview: unsupported format <B

    Should not '<B' and be equivalent to '>B' and just 'B'? Would not adding support of formats with specified endianess make 0-dimensional memoryview objects iterable?

    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Sep 7, 2020

    memoryview can store anything, but only access and display
    the native format. The reason is that conversions in the manner
    of the struct module are very expensive.

    You can construct all sorts of formats that memoryview cannot
    handle:

    >>> x = array(1, "c")
    >>> x
    array(b'1', dtype='|S1')
    >>> y = memoryview(x)
    >>> 
    >>> list(y)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NotImplementedError: memoryview: unsupported format 1s

    I don't think that the message is obscure, since all these
    facts are documented.

    @skrah skrah mannequin assigned skrah Sep 7, 2020
    @serhiy-storchaka
    Copy link
    Member

    It is surprising the endianess of byte matters.

    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Sep 7, 2020

    list() has to *interpret* the endianness to display it. The endianness
    of the *native* format does not matter. The endianness of an explicit
    format matters, since list has to be able to convert non-native formats.

    I suggest to look at XND, which is a memoryview on steroids and
    has a lot of those features.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    mdickinson pushed a commit that referenced this issue Apr 22, 2023
    Changes the behaviour of `len` on a zero-dimensional `memoryview` to raise `TypeError`. Previously, `len` would return `1`.
    @mdickinson
    Copy link
    Member

    mdickinson commented Apr 22, 2023

    Closed by #18463

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    2 participants