Title: Zero argument super is broken in 3.6 for methods with a hacked __class__ cell
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.7, Python 3.6
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: bup, ncoghlan
Priority: normal Keywords: patch

Created on 2017-11-30 07:39 by bup, last changed 2017-12-03 13:36 by ncoghlan. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4675 merged ncoghlan, 2017-12-02 11:40
PR 4684 merged ncoghlan, 2017-12-03 01:22
Messages (7)
msg307279 - (view) Author: Dan Snider (bup) * Date: 2017-11-30 07:39
The following code works in 3.3, 3.4, and 3.5, but in 3.6 it throws  RuntimeError: super(): bad __class__ cell. 

from types import FunctionType, CodeType

def create_closure(__class__):
    return (lambda: __class__).__closure__

def new_code(c_or_f):
    '''A new code object with a __class__ cell added to freevars'''
    c = c_or_f.__code__ if isinstance(c_or_f, FunctionType) else c_or_f
    return CodeType(
        c.co_argcount, c.co_kwonlyargcount, c.co_nlocals,
        c.co_stacksize, c.co_flags, c.co_code, c.co_consts, c.co_names,
        c.co_varnames, c.co_filename, c.co_name, c.co_firstlineno,
        c.co_lnotab, c.co_freevars + ('__class__',), c.co_cellvars)

def add_foreign_method(cls, f):
    code = new_code(f.__code__)
    name = f.__name__
    defaults = f.__defaults__
    closure = (f.__closure__ or ()) + create_closure(cls)
    setattr(cls, name, FunctionType(code, globals(), name, defaults, closure))

class List(list):
    def append(self, elem):
    def extend(self, elems):
def __getitem__(self, i):
    print('foreign getitem')
    return super().__getitem__(i)

add_foreign_method(List, __getitem__)

self = List([1,2,3])
msg307352 - (view) Author: Dan Snider (bup) * Date: 2017-11-30 23:55
The hacked cell object using this method appears to be changed to NULL when accessed by frame.f_localsplus. I don't know C well enough to find out what's happening because nothing looks different to me in PyFrame_FastToLocalsWithError.

Also creating a closure with:
from ctypes import pythonapi, py_object

new_cell = pythonapi.PyCell_New
new_cell.argtypes = (py_object, )
new_cell.restype = py_object

doesn't solve this either.
msg307414 - (view) Author: Dan Snider (bup) * Date: 2017-12-01 22:45
So while CO_NOFREE is set in all versions with the example code, it appears only python 3.6 recognizes that flag and disallows the accessing of the __class__ cell. In this case the error message is bad because it implies that the __class__ cell is the wrong type. Disabling the flag when creating the code objects allows the above code to work.
msg307420 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-02 04:16
Given that, I'd say the way to cleanest way to fix this would be to remove these lines from "compute_code_flags" in compile.c:

    if (!PyDict_GET_SIZE(c->u->u_freevars) &&
        !PyDict_GET_SIZE(c->u->u_cellvars)) {
        flags |= CO_NOFREE;

and replace them with a check like the following in PyCode_New just after we ensure the Unicode string for the filename is ready:

    if (!PyTuple_GET_SIZE(freevars) &&
        !PyTuple_GET_SIZE(cellvars)) {
        flags |= CO_NOFREE;

That way CO_NOFREE will be set only when appropriate regardless of how the code object is created, rather than relying on the caller to set it correctly.
msg307480 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-03 01:12
New changeset 078f1814f1a4413a2a0fdb8cf4490ee0fc98ef34 by Nick Coghlan in branch 'master':
bpo-32176: Set CO_NOFREE in the code object constructor (GH-4675)
msg307511 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-03 13:33
New changeset c8f32aae0aa173e122cf4c0592caec620d0d1de9 by Nick Coghlan in branch '3.6':
[3.6] bpo-32176: Set CO_NOFREE in the code object constructor (GH-4684)
msg307512 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-12-03 13:36
Thanks for the issue report! The fix will be released in 3.6.4 and 3.7.0a3 (both of which are expected to be later this month).
Date User Action Args
2017-12-03 13:36:10ncoghlansetstatus: open -> closed
versions: + Python 3.7
messages: + msg307512

resolution: fixed
stage: patch review -> resolved
2017-12-03 13:33:00ncoghlansetmessages: + msg307511
2017-12-03 01:22:07ncoghlansetpull_requests: + pull_request4597
2017-12-03 01:12:22ncoghlansetmessages: + msg307480
2017-12-02 11:40:05ncoghlansetkeywords: + patch
stage: patch review
pull_requests: + pull_request4583
2017-12-02 04:16:57ncoghlansetmessages: + msg307420
2017-12-01 22:45:59bupsetmessages: + msg307414
2017-12-01 06:54:15serhiy.storchakasetnosy: + ncoghlan
2017-11-30 23:55:37bupsetmessages: + msg307352
2017-11-30 07:39:35bupcreate