Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose dict_proxy internal type as types.MappingProxyType #58594

Closed
vstinner opened this issue Mar 22, 2012 · 40 comments
Closed

Expose dict_proxy internal type as types.MappingProxyType #58594

vstinner opened this issue Mar 22, 2012 · 40 comments
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

BPO 14386
Nosy @gvanrossum, @birkenfeld, @rhettinger, @pitrou, @vstinner, @giampaolo, @merwok, @bitdancer, @voidspace, @skrah, @ericsnowcurrently, @JimJJewett
Files
  • dictproxy.patch
  • dictproxy-2.patch
  • mappingview-3.patch
  • mappingview-4.patch
  • mappingproxy-5.patch
  • mappingproxy_abc.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2012-04-19.23:42:55.098>
    created_at = <Date 2012-03-22.01:50:03.533>
    labels = ['interpreter-core', 'type-feature']
    title = 'Expose dict_proxy internal type as types.MappingProxyType'
    updated_at = <Date 2012-04-19.23:42:55.097>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2012-04-19.23:42:55.097>
    actor = 'python-dev'
    assignee = 'none'
    closed = True
    closed_date = <Date 2012-04-19.23:42:55.098>
    closer = 'python-dev'
    components = ['Interpreter Core']
    creation = <Date 2012-03-22.01:50:03.533>
    creator = 'vstinner'
    dependencies = []
    files = ['24990', '24991', '25028', '25063', '25070', '25229']
    hgrepos = []
    issue_num = 14386
    keywords = ['patch']
    message_count = 40.0
    messages = ['156536', '156538', '156550', '156551', '156634', '156780', '156781', '156783', '156785', '156786', '156788', '156789', '156792', '156796', '156815', '156860', '156864', '156867', '156868', '156871', '156934', '156936', '156971', '156973', '156980', '157008', '157012', '157030', '157036', '157083', '157111', '157112', '157113', '157517', '157618', '157619', '157630', '158374', '158375', '158776']
    nosy_count = 15.0
    nosy_names = ['gvanrossum', 'georg.brandl', 'rhettinger', 'pitrou', 'vstinner', 'giampaolo.rodola', 'eric.araujo', 'Arfrever', 'r.david.murray', 'michael.foord', 'skrah', 'python-dev', 'eric.snow', 'poq', 'Jim.Jewett']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue14386'
    versions = ['Python 3.3']

    @vstinner
    Copy link
    Member Author

    Attached patch makes the dictproxy type public. Example:

    $ ./python 
    Python 3.3.0a1+ (default:059489cec7b9+, Mar 22 2012, 02:45:36) 
    >>> d=dictproxy({1:2})
    >>> d
    dict_proxy({1: 2})
    >>> d[1]
    2
    >>> d[1]=3
    TypeError: 'dictproxy' object does not support item assignment
    >>> del d[1]
    TypeError: 'dictproxy' object doesn't support item deletion
    >>> d.copy()
    {1: 2}
    >>> dir(d)
    [..., '__getitem__', 'copy', 'get', 'items', 'keys', 'values']

    The patch doesn't have any test or documentation yet.

    See also the (now rejected) PEP-416 (frozendict).

    @vstinner vstinner added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Mar 22, 2012
    @vstinner
    Copy link
    Member Author

    dictproxy.patch: proxy_new() should check that dict is a dict (PyDict_Check).

    @vstinner
    Copy link
    Member Author

    Patch version 2:

    • add tests and documentation
    • dictproxy code moved to dictproxyobject.c
    • dict_proxy replaced with dictproxy in repr(dictproxy)
    • "key in dictproxy" now handles correctly dict subclasses
    • dictproxy constructor raises a TypeError if the argument is not a dict

    I added a section "Immutable Mapping" in the doc, this title is maybe wrong because a dictproxy is not truly immutable. If the dict referenced by the dictproxy is modified, the dictproxy is also modified.

    dictproxy implementation supports any mapping, it is not specific to dict. It would be interesting any mapping, collections.ChainMap for example. The problem is to reject sequence in dictproxy constructor.

    The "PyMapping_Check(dict) && !PyMapping_Check(dict)" fails on ChainMap. type.__new__(ChainMap) fills tp_as_sequence->sq_item slot is defined because ChainMap has a __getitem__ method.

    Accepting sequences would be a bad idea because dictproxy(sequence)[missing_key] raises IndexError instead of KeyError.

    Note: issubclass(collections.ChainMap, collections.abc.Sequence) is False.

    @vstinner
    Copy link
    Member Author

    The dictproxy is useful to implement a Python sandbox: see the related issue bpo-14385.

    @jimjjewett
    Copy link
    Mannequin

    jimjjewett mannequin commented Mar 23, 2012

    The problem is to reject sequence in dictproxy constructor.

    Why? Just because you can't delegate in quite the same way? A sequence *does* meet the (immutable) Mapping interface; it just won't happen to have any non-integer keys.

    Or are you worried about IndexError vs KeyError? Personally, I would just document it as returning LookupError, but if you're really worried, you could always catch the IndexError and raise a KeyError from it.

    The "PyMapping_Check(dict) && !PyMapping_Check(dict)"

    I'm not seeing how anything could sanely pass that ... was there a typo when pasting?

    @vstinner
    Copy link
    Member Author

    A sequence *does* meet the (immutable) Mapping interface

    Ah? Let's try:

    >>> x=[1, 2, 3]
    >>> x.get(2)
    AttributeError: 'list' object has no attribute 'get'
    >>> x.keys()
    AttributeError: 'list' object has no attribute 'keys'

    list doesn't implement the collections.abc.Mapping ABC.

    @vstinner
    Copy link
    Member Author

    The "PyMapping_Check(dict) && !PyMapping_Check(dict)"

    Oh, I mean "PyMapping_Check(dict) && !PySequence_Check(dict)"

    @rhettinger
    Copy link
    Contributor

    I'm -1 on exposing this API.

    Victor's sandboxing project doesn't warrant cluttering the public toolset.

    As a consultant who gets to see many large codebases in different companies and I see zero need for this type to be exposed. It is not hard to build proxies around any existing type. Victor's use case is unique in that he needs the type to be implemented in C. To that end, it would be okay to expose _dictproxy but not to make it a documented type.

    @bitdancer
    Copy link
    Member

    I believe it was actually Guido who suggested exposing dict_proxy, in response to Victor (but independent of Victor's needs, if I understood the message correctly).

    It has always seemed odd to me ("always" referring my time working with Python3, of course) that a class that implements the mapping protocol has no way to return the same type of objects that real dict methods do.

    @vstinner
    Copy link
    Member Author

    Victor's use case is unique in that he needs the type
    to be implemented in C.

    mxProxy, zope.proxy and zope.security implement object proxies in C (for security purpose). These proxies are not specific to dict, but are generic.

    dictproxy is already used in Python by the type.__dict__ descriptor getter of new-style classes since Python 2.2, since this changeset:

    changeset: 18933:09df3254b49d
    branch: legacy-trunk
    user: Tim Peters <tim.peters@gmail.com>
    date: Thu Aug 02 04:15:00 2001 +0000
    files: Include/Python.h Include/abstract.h Include/ceval.h Include/classobject.h Include/descrobject.h Include/dictobject.h Include/eval.h Include/func
    description:
    Merge of descr-branch back into trunk.

    Guido van Rossum wrote in its rejection notice of the PEP-416: "On the other hand, exposing the existing read-only dict proxy as a built-in type sounds good to me. (It would need to be changed to allow calling the constructor.) GvR."

    --

    To that end, it would be okay to expose _dictproxy
    but not to make it a documented type.

    My sandbox only needs the issue bpo-14385, pysandbox can implement its own dictproxy class.

    If the issue bpo-14385 is accepted, pysandbox can use any mapping, even a mapping implemented in Python! It's just simpler to implement a secure proxy in C.

    So this issue is not directly related to sandboxing.

    @rhettinger
    Copy link
    Contributor

    It's clutter and we should show some restraint before adding new types.

    Enthought's Python Distribution isn't shy about adding tools when they have a demonstrated need. Likewise, Django adds any reusable tools they need. Neither have a dict proxy. I've never run into a use case for it and haven't seen any feature requests for it. PyPi doesn't have package for it (at least not one with any uptake). The languages doesn't *need* this, nor do the other implementations which are trying to keep up with CPython.

    Guido may let you put in it, but that doesn't mean you should. Again, I ask you to show some restraint. The language has already exploded beyond what fits into most developer's heads (even core devs don't even know about many the newer features).

    @vstinner
    Copy link
    Member Author

    It's clutter and we should show some restraint before adding new types.

    dictproxy() doesn't need to be a public builtin type. We can just implement its constructor without documenting the type. Or it may be exposed in a module like collections.

    @gvanrossum
    Copy link
    Member

    Given that dictproxy is needed to implement the proper interface for classes I don't think it is an implementation detail, and so I think it ought to be constructable. We have more esoteric types constructable (e.g. function).

    I also happen to think that its use to implement classes is a pretty darn good use case, and I am sure there are other use cases.

    I don't really care if we make it a builtin but there should be a standard name for it somewhere in the stdlib.

    @rhettinger
    Copy link
    Contributor

    [haypo]

    dictproxy() doesn't need to be a public builtin type.

    +1

    We can just implement its constructor without documenting the type.

    +1

    Or it may be exposed in a module like collections.

    Please don't use collections as a dumping ground for this sort of thing.

    @vstinner
    Copy link
    Member Author

    New patch replacing dictproxy() builtin type with collections.mappingview() type:

    • dictproxy() type doesn't need to be a builtin type
    • it is not specific to dict so replace "dict" prefix with "mapping"
    • "view" is a better suffix than "proxy" to indicate that it is a read-only proxy
    • IMO collections is the best place for new containers

    I don't understand how collections types are called: should it be MappingView or mappingview?

    The code of mappingview is just moved to _collectionsmodule.c to avoid the creation of two short files. PyDictProxy_Type and PyDictProxy_New() are kept and still declared in descrobject.h for backward compatibility.

    I fixed the doc to indicate that mappingview methods operate on the *underlying mapping*.

    I also added test_views() test.

    TODO:

    • mappingview() constructor only accepts dict: it should accept any mapping
    • mappingview.copy() creates a dict, it should create a new object of the same type than the underlying mapping

    @rhettinger
    Copy link
    Contributor

    I would very much like this to go somewhere other than the collections module. As the maintainer of that module, I've worked hard to keep it clutter free (people frequently want to stick things in that module when they can't think of some place else to put it).

    Perhaps consider the existing types module, the builtin namespace, or a new module for proxies (if dict_proxy goes in, other proxy types won't be far behind).

    @gvanrossum
    Copy link
    Member

    Raymond, that sounds like a very irrational way of deciding where it should go.

    @skrah
    Copy link
    Mannequin

    skrah mannequin commented Mar 26, 2012

    I wonder about the name: There is already a collections.abc.MappingView.

    @rhettinger
    Copy link
    Contributor

    The weakref collections objects (sets, dicts, etc) were all put in a separate module and that has worked out well. I suggest doing the same for proxy objects.

    In maintaining the itertools module, I learned long ago that adding new tools to the module made the whole module more difficult to use (increasing the number of choices increases the complexity of choosing the correct tool). I believe that same also applies to the collections module and have been very reserved about adding new types there.

    ISTM that exposing internal types such as dict_proxy or bound/unbound methods could be collected together or that various proxy types and tools could be collected together. I don't feel that dict_proxy belongs with namedtuple, deques, or OrderedDicts.

    As the module maintainer, I request that dict_proxy be put elsewhere. Of course, as BDFL you are free to override that request.

    @gvanrossum
    Copy link
    Member

    I like the suggestion of a library module containing proxy classes.

    @pitrou
    Copy link
    Member

    pitrou commented Mar 27, 2012

    There's already a weak proxy class in the weakref module. Not sure a separate module for the other proxies is a good idea.

    @voidspace
    Copy link
    Contributor

    If we do create a proxy module I'd like to suggest adding all or part of Phillip Eby's ProxyTypes:

    http://pypi.python.org/pypi/ProxyTypes

    I haven't used it directly, but did read through it a while ago and have ended up re-implementing chunks of it myself several times.

    @merwok
    Copy link
    Member

    merwok commented Mar 28, 2012

    weakref.mappingview sounds nice to me.

    @birkenfeld
    Copy link
    Member

    But it has nothing to do with weakrefs, so...

    @gvanrossum
    Copy link
    Member

    Since it has to go *somewhere*, let's put it in the types module. It has a variety of other types that are used in the implementation of classes and functions and related things.

    @rhettinger
    Copy link
    Contributor

    [GvR]

    Since it has to go *somewhere*, let's put it
    in the types module. It has a variety of other
    types that are used in the implementation of
    classes and functions and related things.

    +1

    @poq
    Copy link
    Mannequin

    poq mannequin commented Mar 29, 2012

    It is exposed as types.DictProxyType in Python 2...

    @vstinner
    Copy link
    Member Author

    "It is exposed as types.DictProxyType in Python 2..."

    Yes, but the purpose of the issue is to enable its constructor. You cannot instanciate a DictProxy in Python 2:

    >>> import types
    >>> types.DictProxyType
    <type 'dictproxy'>
    >>> types.DictProxyType({})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: cannot create 'dictproxy' instances

    @vstinner
    Copy link
    Member Author

    Patch version 4, it is ready for a review. Summary of the patch:

    • expose the internal dict_proxy() type (used for __dict__ of user classes) as types.MappingViewType (it was exposed a types.DictProxyType in Python 2)
    • repr(types.MappingViewType) returns 'mappingview(...)' instead of 'dict_proxy(...)'
    • implement mappingview.__new__()
    • mappingview constructors (mappingview.__new__() and PyDictProxy_New) rejects sequences
    • Keep PyDictProxy_New() and PyDictProxy_Type names for backward compatibility
    • mappingview_contains() calls PySequence_Contains() for types other than dict (instead of PyDict_Contains)
    • Write a doc and a lot of tests
    • Rename the internal "dict" attribute to "mapping" (this change may be reverted if it impacts backward compatibility)

    I added the new tests to test_descr. Is test_types a better place for these types (the type is exposed in the types module, but implemented in descrobject.c)? I leaved the object implementation in descrobject.c to not have to add a new short file and to keep the Mercurial history. The file does still contain this funny comment:

    /* This has no reason to be in this file except that adding new files is a
    bit of a pain */

    MappingViewType constructor accepts any mapping: dict, collections.ChainMap, ... Sequences are rejected: tuple, list, dict_view, etc.

    @gvanrossum
    Copy link
    Member

    I'd like to bikeshed a little on the name. I think it should be
    MappingProxy. (We don't use "view" much but the place where we do use
    it, for keys/values/items views, is very different I think. Also
    collections.abc already defines MappingView as the base class for
    KeysView and friends.)

    Also make sure there's no way for someone who has access to the proxy
    to get to the underlying mapping; preventing that access is the
    explicit purpose of the original dict_proxy type. You might even add a
    comment to this effect to the code, so someone doesn't accidentally
    add it in the future. Perhaps in the proxy_methods structure or in the
    type structure just before tp_members.

    @vstinner
    Copy link
    Member Author

    (We don't use "view" much but the place where we do use
    it, for keys/values/items views, is very different I think.)

    You are right, it's closer to a proxy because it has the same methods than a mapping (except of some methods used to modify a mapping), whereas the API of keys/values/items views is very different from the mapping API.

    Also collections.abc already defines MappingView as the base
    class for KeysView and friends.)

    MappingView is a strange ABC: it should probably declare __contains__ as an abstract method. I don't see why it doesn't inherit from Set (which would avoid the need of declaring __contains__, Set is a Container) whereas KeysView, ItemsView and ValuesView do inherit from Set.

    I suppose that MappingView is also present to factorize code but it should not be used directly.

    Anyway, you are not the first one who remarks that we already use "view" to define something else, so I wrote a new patch to use the "mappingproxy" name (exposed as types.MappingProxyType).

    Also make sure there's no way for someone who has access
    to the proxy to get to the underlying mapping;

    I don't think that we can write unit tests for such property. I suppose that the comment below is enough.

    You might even add a comment ...

    Done in my last patch:

    /* WARNING: mappingproxy methods must not give access to the underlying mapping */

    @vstinner
    Copy link
    Member Author

    Oh, collections.abc contains also the mappingview type exposed with the name "dict_proxy":
    dict_proxy = type(type.__dict__)

    It was exposed as _abcoll.dict_proxy in Python 3.2.

    It is not documented. Should we keep it for backward compatibility, or can it be removed? _abcoll was a private module and I didn't find dict_proxy in Python 3.2 doc.

    @merwok
    Copy link
    Member

    merwok commented Mar 29, 2012

    MappingView is a strange ABC: it should probably declare __contains__ as an abstract method.
    I think that was not done because it would not be very useful: MappingView seems to exist only as a base class for the other *View, which define __contains__.

    I don't see why it doesn't inherit from Set (which would avoid the need of declaring __contains__,
    Set is a Container) whereas KeysView, ItemsView and ValuesView do inherit from Set.
    Not ValuesView, because you may have the same object as more than one value.

    @bitdancer bitdancer added the type-feature A feature request or enhancement label Mar 30, 2012
    @vstinner vstinner changed the title Expose dictproxy as a public type Expose dict_proxy internal type as types.MappingViewType Apr 4, 2012
    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 4, 2012

    mappingproxy-5.patch is ready for a review. msg157036 contains a summary except that types.MappingViewType is now called types.MappingProxyType.

    @vstinner vstinner changed the title Expose dict_proxy internal type as types.MappingViewType Expose dict_proxy internal type as types.MappingProxyType Apr 5, 2012
    @rhettinger
    Copy link
    Contributor

    [Victor]
    '''Oh, collections.abc contains also the mappingview type exposed with the name "dict_proxy": dict_proxy = type(type.__dict__)

    It was exposed as _abcoll.dict_proxy in Python 3.2.
    It is not documented. Should we keep it for backward compatibility, or can it be removed? _abcoll was a private module and I didn't find dict_proxy in Python 3.2 doc.
    '''

    You can ignore those. As you saw, these are undocumented and not listed in __all__. If we ever expose these, it will likely be through the types module (or less likely through the collections.abc module which is public in 3.3). Probably, they won't get exposed at all because they are awkward to access directly and because they have implementation specific details (i.e. other implementations have their choice of how to implement various iterators and whatnot).

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 5, 2012

    You can ignore those.

    You mean that I can remove it?

    It's a little bit surprising to find a concrete class in collections.abc. So I think that it's better to expose dict_proxy in types than in collections.abc.

    @rhettinger
    Copy link
    Contributor

    [Victor]

    You mean that I can remove it?

    No, it needs to stay there.
    Those concrete types aren't exposed to users
    but they are used inside to collections.abcs
    to register the types so that isinstance()
    will work as expected.

    Summary: The collections.abc module is fine as-is.

    Go ahead and add dictproxy to the types module.
    Expose it there and document it there.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 15, 2012

    New changeset c3a0197256ee by Victor Stinner in branch 'default':
    Issue bpo-14386: Expose the dict_proxy internal type as types.MappingProxyType
    http://hg.python.org/cpython/rev/c3a0197256ee

    @vstinner
    Copy link
    Member Author

    Summary: The collections.abc module is fine as-is.

    Ok, but there is still an issue: issubclass(types.MappingProxyType, collections.abc.Mapping) is False. Attached registers MappingProxyType as a Mapping. Is it correct?

    The patch also renames dict_proxy to mappingproxy. Is it backward incompatible? (collections.abc module was added to Python 3.3)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 19, 2012

    New changeset 34af3e74292d by Victor Stinner in branch 'default':
    Close bpo-14386: Register types.MappingProxyType as a Mapping
    http://hg.python.org/cpython/rev/34af3e74292d

    @python-dev python-dev mannequin closed this as completed Apr 19, 2012
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    interpreter-core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    8 participants