classification
Title: PEP447: Add type.__locallookup__
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, ronaldoussoren
Priority: normal Keywords: patch

Created on 2013-06-10 13:15 by ronaldoussoren, last changed 2013-08-01 09:34 by ronaldoussoren.

Files
File name Uploaded Description Edit
supers_updated.py ronaldoussoren, 2013-06-11 15:45
issue-18181-poc-v4.txt ronaldoussoren, 2013-07-04 06:35 review
super-hook-proposal.rst ronaldoussoren, 2013-07-04 06:37
issue18181-locallookup-only-super.txt ronaldoussoren, 2013-07-15 14:56 review
issue-18181-full-v1.txt ronaldoussoren, 2013-07-17 14:57
issue-18181-full-v2.txt ronaldoussoren, 2013-07-22 14:20 review
issue-18181-full-v2b.txt ronaldoussoren, 2013-07-22 16:56 review
issue-18181-full-v3.txt ronaldoussoren, 2013-07-29 12:37 review
issue-18181-full-v4.txt ronaldoussoren, 2013-08-01 09:34 review
Messages (21)
msg190905 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-10 13:15
Super() walks the MRO and looks for the looked-up attribute in the __dict__ of types on the MRO. This can cause problems when classes implement __getattribute__: a class on the MRO can implement a method that super() won't find because it isn't in the class __dict__ (yet).

I'm running into this with PyObjC: the __dict__ for the Python proxies for Objective-C classes are filled lazily (*) by tp_getattro (__getattribute__ in Python) to speed up the bridge and because ObjC is almost as dynamic as Python and methods might appear during runtime (without there being a hook for detecting such changes).

A possible solution to this:

* Add a tp_getattro_super (**) slot to PyTypeObject, with the 
  same signature as tp_getattro, but that only looks at this particular
  class (as opposed to tp_getattro that walks the entire MRO and looks
  in the object's __dict__)  (***)

* The tp_gettro of super calls tp_getattro_super of types of along the
  MRO when that slot is not NULL, and uses the current implementation
  (look in tp_dict) when the slot is NULL.

Would such a change be acceptable?

Open issues:

* Does the new slot get exposed to Python code 
  (and if so, under which name)?

* Should PyObject_GenericGetAttr use the new slot as well?


Footnotes:

(*) The current release of PyObjC (2.5) eagerly tries to keep the proxy class __dict__ up to date, an upcoming major release will be as lazy as possible to speed up the bridge. The problem can with some effert be triggered with PyObjC 2.5, and triggering it is easy in the upcoming major release

(**) Or some better name

(***) I'm being very sloppy in my use of terminology here, hopefully my proposal is clear enough anyway.
msg190909 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-10 14:29
Do you have an example in pure Python code?
msg190912 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-10 15:08
I didn't, but the attached script should do the trick. The code suffers badly from copy&paste editing, but more or less has the same behavior as PyObjC.

It defines a subclass of list (MyList) with an append method, and more importantly also defines 3 "proxy" classes: ProxyObject, ProxyList and ProxyMyList that are proxy types for object, list and MyList.

At the end I try to access methods on instances of a proxy objects for MyList, and finally try to access super(MyList, v).append. That only works when "append" is in the __dict__ for ProxyList (for example by uncommenting the 3 lines just above the use of super()).
msg190920 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-10 16:01
issue783528 was a bit similar.
msg190921 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-10 16:09
Looks like it. That issue was closed without a good reason though.
msg190965 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 14:30
I've attached a first very rough patch to demonstrate what I was rambling about.

With this patch and a patched version of PyObjC I can use the builtin super class with the proxy classes for Objective-C classes.

Open issues:

* Does this need a pep?

* The name of the new slot sucks, need to find a better one

* Should it be possible to define the slot in Python code?

  Probably, the 'super.py' file is a crude hack but I can imagine
  more realistic scenarios where implementing the slot in Python
  would be useful.

* There are no tests, and no documentation updates
msg190971 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 15:45
Second version of the poc adds a way to implement the hook in Python code, and supers_updated.py uses that hook.

Still no tests, no documentation and with awfull naming :-)
msg190972 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 15:46
.
msg190976 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-11 17:16
Is it possible to avoid a new slot?
for example, super() could walk the tp_base-> chain and call tp_getattro each time the value changes.
msg190979 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-11 18:01
I don't think it is possible to avoid adding a new slot. The default implementation of tp_getattro (PyObject_GenericGetAttr) looks in the instance dict, while super does not (because it wants a less specific implementation).  

PyObject_GenericGetAttr will also walk the entire MRO, while super.__getattribute__ does not start search at the first entry in the MRO.

Although I'd love to be proven wrong ;-)
msg191020 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-12 11:01
Slightly updated patch (different names, added some documentation)
msg191021 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-12 11:01
added draft of a pep-style proposal
msg192274 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-04 06:35
Updated version of the patch, with some semantic changes and tests for the Python level API.

Known issues:

* Patches uses tabs for indentation

* No tests for the C level API

* I don't like implementation of slot_tp_getattro_super
msg192275 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-04 06:37
Also updated the draft PEP text
msg193100 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-15 13:49
There now is a PEP for this issue: <http://www.python.org/dev/peps/pep-0447/>.

The current version of the PEP describes a different API than documented by the attached proposal and the prove of concept code. I'm working on an updated implementation.
msg193109 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-15 14:56
issue18181-locallookup-only-super.txt implements the current version of PEP 447, but only for the super object. In this version of the  patch normal attribute lookup (object.__getattribute__ and PyObject_GenericGetAttr) are not yet affected.

Another open issue: the documentation for the super object itself does not yet mention __locallookup__; it probably should but I don't know yet how to cleanly add it there.
msg193232 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-17 14:57
issue-18181-full-v1.txt implements support for __locallookup__ to both super and _PyType_Lookup and should implement the entire PEP.

The patch is not yet 100% and is missing:

  * Tests that add __locallookup__ to a metaclass at runtime, should
    enable the new behavior

  * Tests that use the tp_locallookup slot in an type defined in C code

  * Documentation has not been updated.

 
Changes to the implementation:

  * _PyType_Lookup is now _PyType_LookupName 
    and returns a new reference (not a borrowed one)

  * _PyType_Lookup and super.__getattribute__ use the 
     __locallookup__ method when present

  * __locallookup__ is not defined on "type", 
    and the type attribute lookup cache is disabled
    for types that have a metatype with __locallookup__

  * The type attribute cache is disabled for types that are 
    not ready (Py_TPFLAGS_READY), primarily
    as a side effect of the implementation.

    The cache is also disabled for types with a metaclass that
    has a tp_locallookup attribute, as mentioned in the PEP.

    The code does not change the VERSION flags of the type because
    dynamicly adding a __locallookup__ method to the metaclass would
    have to change all instances of that metaclass and that's not
    easily possible.
msg193538 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-22 14:20
issue-18181-full-v2.txt adds more tests to v1 and has slightly better documentation (still not good enough).

A major issue with the patch: for some reason the tests pass when run standalone ("./python.exe -m test.regrtest test_pep447"), but crash when I run another test before it (such as when running "make test").

I'm debugging the problem (this is probably a refcounting issue, but you never know...)
msg193547 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-22 16:56
The crash was due to a bug in the test extension. Fixed in version 2b.
msg193861 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-29 12:37
* Improved docmentation

* __locallookup__ is now called unconditionally, the attribute lookup
  cache is now disabled when aType.__locallookup__ != type.__locallookup__

  This gives a slightly nicer API (the only reason the new method
  was optional was due to an implementation detail in _PyType_Lookup)
msg194038 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-08-01 09:34
Updated the patch for PEP 442.
History
Date User Action Args
2013-08-01 09:34:14ronaldoussorensetfiles: + issue-18181-full-v4.txt

messages: + msg194038
2013-07-29 12:37:33ronaldoussorensetfiles: + issue-18181-full-v3.txt

messages: + msg193861
2013-07-22 16:56:02ronaldoussorensetfiles: + issue-18181-full-v2b.txt

messages: + msg193547
title: Add type.__locallookup__ -> PEP447: Add type.__locallookup__
2013-07-22 14:20:03ronaldoussorensetfiles: + issue-18181-full-v2.txt

messages: + msg193538
2013-07-17 14:57:52ronaldoussorensetfiles: + issue-18181-full-v1.txt

title: super vs. someclass.__getattribute__ -> Add type.__locallookup__
messages: + msg193232
stage: needs patch -> patch review
2013-07-15 14:56:16ronaldoussorensetfiles: + issue18181-locallookup-only-super.txt

messages: + msg193109
2013-07-15 13:49:26ronaldoussorensetmessages: + msg193100
2013-07-04 06:37:04ronaldoussorensetfiles: + super-hook-proposal.rst

messages: + msg192275
2013-07-04 06:36:12ronaldoussorensetfiles: - super-hook-proposal.rst
2013-07-04 06:36:05ronaldoussorensetfiles: - supers.py
2013-07-04 06:35:55ronaldoussorensetfiles: - issue-18181-poc.txt
2013-07-04 06:35:45ronaldoussorensetfiles: - issue-18181-poc-v3.txt
2013-07-04 06:35:18ronaldoussorensetfiles: + issue-18181-poc-v4.txt

messages: + msg192274
2013-06-12 11:01:51ronaldoussorensetfiles: + super-hook-proposal.rst

messages: + msg191021
2013-06-12 11:01:15ronaldoussorensetfiles: + issue-18181-poc-v3.txt

messages: + msg191020
2013-06-11 18:01:52ronaldoussorensetmessages: + msg190979
2013-06-11 17:16:39amaury.forgeotdarcsetmessages: + msg190976
2013-06-11 15:46:29ronaldoussorensetmessages: + msg190972
2013-06-11 15:45:37ronaldoussorensetfiles: + supers_updated.py

messages: + msg190971
2013-06-11 14:30:59ronaldoussorensetkeywords: + patch
files: + issue-18181-poc.txt
messages: + msg190965
2013-06-10 16:09:22ronaldoussorensetmessages: + msg190921
2013-06-10 16:01:03amaury.forgeotdarcsetmessages: + msg190920
2013-06-10 15:08:16ronaldoussorensetfiles: + supers.py

messages: + msg190912
2013-06-10 14:29:29amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg190909
2013-06-10 13:15:53ronaldoussorencreate