This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: PEP447: Add type.__getdescriptor__
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, ronaldoussoren
Priority: normal Keywords: patch

Created on 2013-06-10 13:15 by ronaldoussoren, last changed 2022-04-11 14:57 by admin.

File name Uploaded Description Edit ronaldoussoren, 2013-06-11 15:45
issue-18181-poc-v4.txt ronaldoussoren, 2013-07-04 06:35 review
super-hook-proposal.rst ronaldoussoren, 2013-07-04 06:37
issue18181-locallookup-only-super.txt ronaldoussoren, 2013-07-15 14:56 review
issue-18181-full-v1.txt ronaldoussoren, 2013-07-17 14:57
issue-18181-full-v2.txt ronaldoussoren, 2013-07-22 14:20 review
issue-18181-full-v2b.txt ronaldoussoren, 2013-07-22 16:56 review
issue-18181-full-v3.txt ronaldoussoren, 2013-07-29 12:37 review
issue-18181-full-v4.txt ronaldoussoren, 2013-08-01 09:34 review
pep447-2015-07-25.txt ronaldoussoren, 2015-07-25 09:29 review ronaldoussoren, 2015-07-25 09:58
pep447-2015-07-25-v2.txt ronaldoussoren, 2015-07-25 12:55 review
pep447-2015-07-26.txt ronaldoussoren, 2015-07-26 09:53 review
pep447-2015-07-26.txt ronaldoussoren, 2016-07-24 07:56 Updated patch for trunk with minor code cleanup review
Messages (27)
msg190905 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-10 13:15
Super() walks the MRO and looks for the looked-up attribute in the __dict__ of types on the MRO. This can cause problems when classes implement __getattribute__: a class on the MRO can implement a method that super() won't find because it isn't in the class __dict__ (yet).

I'm running into this with PyObjC: the __dict__ for the Python proxies for Objective-C classes are filled lazily (*) by tp_getattro (__getattribute__ in Python) to speed up the bridge and because ObjC is almost as dynamic as Python and methods might appear during runtime (without there being a hook for detecting such changes).

A possible solution to this:

* Add a tp_getattro_super (**) slot to PyTypeObject, with the 
  same signature as tp_getattro, but that only looks at this particular
  class (as opposed to tp_getattro that walks the entire MRO and looks
  in the object's __dict__)  (***)

* The tp_gettro of super calls tp_getattro_super of types of along the
  MRO when that slot is not NULL, and uses the current implementation
  (look in tp_dict) when the slot is NULL.

Would such a change be acceptable?

Open issues:

* Does the new slot get exposed to Python code 
  (and if so, under which name)?

* Should PyObject_GenericGetAttr use the new slot as well?


(*) The current release of PyObjC (2.5) eagerly tries to keep the proxy class __dict__ up to date, an upcoming major release will be as lazy as possible to speed up the bridge. The problem can with some effert be triggered with PyObjC 2.5, and triggering it is easy in the upcoming major release

(**) Or some better name

(***) I'm being very sloppy in my use of terminology here, hopefully my proposal is clear enough anyway.
msg190909 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-10 14:29
Do you have an example in pure Python code?
msg190912 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-10 15:08
I didn't, but the attached script should do the trick. The code suffers badly from copy&paste editing, but more or less has the same behavior as PyObjC.

It defines a subclass of list (MyList) with an append method, and more importantly also defines 3 "proxy" classes: ProxyObject, ProxyList and ProxyMyList that are proxy types for object, list and MyList.

At the end I try to access methods on instances of a proxy objects for MyList, and finally try to access super(MyList, v).append. That only works when "append" is in the __dict__ for ProxyList (for example by uncommenting the 3 lines just above the use of super()).
msg190920 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-10 16:01
issue783528 was a bit similar.
msg190921 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-10 16:09
Looks like it. That issue was closed without a good reason though.
msg190965 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 14:30
I've attached a first very rough patch to demonstrate what I was rambling about.

With this patch and a patched version of PyObjC I can use the builtin super class with the proxy classes for Objective-C classes.

Open issues:

* Does this need a pep?

* The name of the new slot sucks, need to find a better one

* Should it be possible to define the slot in Python code?

  Probably, the '' file is a crude hack but I can imagine
  more realistic scenarios where implementing the slot in Python
  would be useful.

* There are no tests, and no documentation updates
msg190971 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 15:45
Second version of the poc adds a way to implement the hook in Python code, and uses that hook.

Still no tests, no documentation and with awfull naming :-)
msg190972 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 15:46
msg190976 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-11 17:16
Is it possible to avoid a new slot?
for example, super() could walk the tp_base-> chain and call tp_getattro each time the value changes.
msg190979 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 18:01
I don't think it is possible to avoid adding a new slot. The default implementation of tp_getattro (PyObject_GenericGetAttr) looks in the instance dict, while super does not (because it wants a less specific implementation).  

PyObject_GenericGetAttr will also walk the entire MRO, while super.__getattribute__ does not start search at the first entry in the MRO.

Although I'd love to be proven wrong ;-)
msg191020 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-12 11:01
Slightly updated patch (different names, added some documentation)
msg191021 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-12 11:01
added draft of a pep-style proposal
msg192274 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-04 06:35
Updated version of the patch, with some semantic changes and tests for the Python level API.

Known issues:

* Patches uses tabs for indentation

* No tests for the C level API

* I don't like implementation of slot_tp_getattro_super
msg192275 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-04 06:37
Also updated the draft PEP text
msg193100 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-15 13:49
There now is a PEP for this issue: <>.

The current version of the PEP describes a different API than documented by the attached proposal and the prove of concept code. I'm working on an updated implementation.
msg193109 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-15 14:56
issue18181-locallookup-only-super.txt implements the current version of PEP 447, but only for the super object. In this version of the  patch normal attribute lookup (object.__getattribute__ and PyObject_GenericGetAttr) are not yet affected.

Another open issue: the documentation for the super object itself does not yet mention __locallookup__; it probably should but I don't know yet how to cleanly add it there.
msg193232 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-17 14:57
issue-18181-full-v1.txt implements support for __locallookup__ to both super and _PyType_Lookup and should implement the entire PEP.

The patch is not yet 100% and is missing:

  * Tests that add __locallookup__ to a metaclass at runtime, should
    enable the new behavior

  * Tests that use the tp_locallookup slot in an type defined in C code

  * Documentation has not been updated.

Changes to the implementation:

  * _PyType_Lookup is now _PyType_LookupName 
    and returns a new reference (not a borrowed one)

  * _PyType_Lookup and super.__getattribute__ use the 
     __locallookup__ method when present

  * __locallookup__ is not defined on "type", 
    and the type attribute lookup cache is disabled
    for types that have a metatype with __locallookup__

  * The type attribute cache is disabled for types that are 
    not ready (Py_TPFLAGS_READY), primarily
    as a side effect of the implementation.

    The cache is also disabled for types with a metaclass that
    has a tp_locallookup attribute, as mentioned in the PEP.

    The code does not change the VERSION flags of the type because
    dynamicly adding a __locallookup__ method to the metaclass would
    have to change all instances of that metaclass and that's not
    easily possible.
msg193538 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-22 14:20
issue-18181-full-v2.txt adds more tests to v1 and has slightly better documentation (still not good enough).

A major issue with the patch: for some reason the tests pass when run standalone ("./python.exe -m test.regrtest test_pep447"), but crash when I run another test before it (such as when running "make test").

I'm debugging the problem (this is probably a refcounting issue, but you never know...)
msg193547 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-22 16:56
The crash was due to a bug in the test extension. Fixed in version 2b.
msg193861 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-29 12:37
* Improved docmentation

* __locallookup__ is now called unconditionally, the attribute lookup
  cache is now disabled when aType.__locallookup__ != type.__locallookup__

  This gives a slightly nicer API (the only reason the new method
  was optional was due to an implementation detail in _PyType_Lookup)
msg194038 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-08-01 09:34
Updated the patch for PEP 442.
msg247187 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-23 11:46
Updated the title and versions, will work on an updated PEP and patch during and after EP'15.
msg247323 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-25 09:29
I've attached a new version of the patch (pep447-2015-07-25.txt). Changes in this version of the patch:

1) Works with the current trunk (as in "all tests pass")

2) Types in C must explicitly set Py_TPFLAGS_GETDESCRIPTOR in tp_flags to 
   enable the tp_getdescriptor slot. 

   This is primarily done avoid crashing when loading a C extension for
   older CPython versions (even if that's not guaranteed to work anyway).

The PEP needs to be updated for the second change, that's next on my list.
msg247330 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-25 09:58
The attached micro benchmark indicates that method_cache in typeobject.c isn't used when using my patch. I'll have too look into that.

Other than that benchmarks results look OK (but: not using the method_cache is unacceptable as this most definitely changes performance).

What is kind of scary, but is to be expected I guess, is that a __getdescriptor__ method in Python is awfully slow.  I have to look into why this is slow, as the slowdown for using Python is significantly worse than I expected.
msg247349 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-25 12:55
pep447-2015-07-25-v2.txt changes two things w.r.t. the patch earlier today:

1) The performance problems w.r.t. the method_cache are gone

2) Added code for handling python exceptions other than AttributeError in
   calls to __getdescriptor__. 

   This code requires some tests to make sure the error handling code 
   is correct.

The new patch also adds some more tests, and adds extra MCACHE_STATS counters. Those aren't part of the proposal but were helpful to find the problem w.r.t. 1).

As I mentioned before implementing __getdescriptor__ in Python is bad for performance of instances of that metaclass. AFAIK that cannot be helped, doing that will just execute a lot more Python code.
msg247350 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-25 14:15
Note: the error handling code for exceptions in __getdescriptor__ definitely isn't good enough yet. 

I'm writing tests and am hunting down the problems those tests find. I'm getting closer and will post a new version when I think I've found all bugs. Probably some time tomorrow.

For now be warned that unexpected exceptions in __getdescriptor__ might crash the interpreter, especially when it is used to look for dunder methods.
msg247419 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2015-07-26 09:53
Todays version of the patch adds more tests, especially for exceptions in __getdescriptor__ during the lookup of special methods.  Those tests were added because the C code couldn't get exceptions other than AttributeError before this patch and can get those know. This requires extra code, and hence extra tests (which exposed a number of crashers...).

The code should be solid now.

Note that the new error handling code isn't optimal style-wise, at some places I test for PyErr_Occurred() instead of adding error returns to functions now returning 'void' to keep the patch smaller and easier to review.
Date User Action Args
2022-04-11 14:57:46adminsetgithub: 62381
2016-07-24 07:56:54ronaldoussorensetfiles: + pep447-2015-07-26.txt
2015-07-26 09:53:25ronaldoussorensetfiles: + pep447-2015-07-26.txt

messages: + msg247419
2015-07-25 14:15:36ronaldoussorensetmessages: + msg247350
2015-07-25 12:55:18ronaldoussorensetfiles: + pep447-2015-07-25-v2.txt

messages: + msg247349
2015-07-25 09:58:56ronaldoussorensetfiles: +

messages: + msg247330
2015-07-25 09:29:56ronaldoussorensetfiles: + pep447-2015-07-25.txt

messages: + msg247323
2015-07-23 11:46:09ronaldoussorensettitle: PEP447: Add type.__locallookup__ -> PEP447: Add type.__getdescriptor__
messages: + msg247187
versions: + Python 3.6, - Python 3.4
2013-08-01 09:34:14ronaldoussorensetfiles: + issue-18181-full-v4.txt

messages: + msg194038
2013-07-29 12:37:33ronaldoussorensetfiles: + issue-18181-full-v3.txt

messages: + msg193861
2013-07-22 16:56:02ronaldoussorensetfiles: + issue-18181-full-v2b.txt

messages: + msg193547
title: Add type.__locallookup__ -> PEP447: Add type.__locallookup__
2013-07-22 14:20:03ronaldoussorensetfiles: + issue-18181-full-v2.txt

messages: + msg193538
2013-07-17 14:57:52ronaldoussorensetfiles: + issue-18181-full-v1.txt

title: super vs. someclass.__getattribute__ -> Add type.__locallookup__
messages: + msg193232
stage: needs patch -> patch review
2013-07-15 14:56:16ronaldoussorensetfiles: + issue18181-locallookup-only-super.txt

messages: + msg193109
2013-07-15 13:49:26ronaldoussorensetmessages: + msg193100
2013-07-04 06:37:04ronaldoussorensetfiles: + super-hook-proposal.rst

messages: + msg192275
2013-07-04 06:36:12ronaldoussorensetfiles: - super-hook-proposal.rst
2013-07-04 06:36:05ronaldoussorensetfiles: -
2013-07-04 06:35:55ronaldoussorensetfiles: - issue-18181-poc.txt
2013-07-04 06:35:45ronaldoussorensetfiles: - issue-18181-poc-v3.txt
2013-07-04 06:35:18ronaldoussorensetfiles: + issue-18181-poc-v4.txt

messages: + msg192274
2013-06-12 11:01:51ronaldoussorensetfiles: + super-hook-proposal.rst

messages: + msg191021
2013-06-12 11:01:15ronaldoussorensetfiles: + issue-18181-poc-v3.txt

messages: + msg191020
2013-06-11 18:01:52ronaldoussorensetmessages: + msg190979
2013-06-11 17:16:39amaury.forgeotdarcsetmessages: + msg190976
2013-06-11 15:46:29ronaldoussorensetmessages: + msg190972
2013-06-11 15:45:37ronaldoussorensetfiles: +

messages: + msg190971
2013-06-11 14:30:59ronaldoussorensetkeywords: + patch
files: + issue-18181-poc.txt
messages: + msg190965
2013-06-10 16:09:22ronaldoussorensetmessages: + msg190921
2013-06-10 16:01:03amaury.forgeotdarcsetmessages: + msg190920
2013-06-10 15:08:16ronaldoussorensetfiles: +

messages: + msg190912
2013-06-10 14:29:29amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg190909
2013-06-10 13:15:53ronaldoussorencreate