This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients Mark.Shannon, frenzy, pablogsal, vstinner
Date 2021-02-17.17:42:19
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1613583740.24.0.43193531195.issue43228@roundup.psfhosted.org>
In-reply-to
Content
In short, cloudpickle recreates a function in two steps:

* Create a function object: globals doesn't contain "__builtins__" key
* Update the function attributes (especially __globals__) with cloudpickle.cloudpicke_fast._function_setstate() which always sets "__builtins__" in the function __globals__ dictionary.

def _function_setstate(obj, state):
    ...
    state, slotstate = state
    ...
    obj_globals = slotstate.pop("__globals__")
    ...
    obj.__globals__.update(obj_globals)
    obj.__globals__["__builtins__"] = __builtins__
    ...

The internal PyFunctionObject.func_builtins can only by set when a function is created, it cannot be updated later.

Why not exposing it in funcobject.c (func_memberlist) as we do for closure, doc, globals and module? So cloudpickle can hack it for its usage. Documentation of these function attributes:
https://docs.python.org/dev/library/inspect.html#types-and-members

In Python 3.9, _PyFrame_New_NoTrack() starts by looking at builtins in the parent frame ("back"): use builtins of the parent frame if the parent frame uses exactly the same dictionary of globals. Then it looks for "__builtins__" keys in the globals dictionary.

In the current Python 3.10, the builtins are retrieved from the internal PyFunctionObject.func_builtins member.

Here is the raw pickle bytecode created by cloudpickle:

$ ./python -m pickletools bug.pickle
    0: \x80 PROTO      5
    2: \x95 FRAME      479
   11: \x8c SHORT_BINUNICODE 'cloudpickle.cloudpickle'
   36: \x94 MEMOIZE    (as 0)
   37: \x8c SHORT_BINUNICODE '_builtin_type'
   52: \x94 MEMOIZE    (as 1)
   53: \x93 STACK_GLOBAL
   54: \x94 MEMOIZE    (as 2)
   55: \x8c SHORT_BINUNICODE 'LambdaType'
   67: \x94 MEMOIZE    (as 3)
   68: \x85 TUPLE1
   69: \x94 MEMOIZE    (as 4)
   70: R    REDUCE
   71: \x94 MEMOIZE    (as 5)
   72: (    MARK
   73: h        BINGET     2
   75: \x8c     SHORT_BINUNICODE 'CodeType'
   85: \x94     MEMOIZE    (as 6)
   86: \x85     TUPLE1
   87: \x94     MEMOIZE    (as 7)
   88: R        REDUCE
   89: \x94     MEMOIZE    (as 8)
   90: (        MARK
   91: K            BININT1    1
   93: K            BININT1    0
   95: K            BININT1    0
   97: K            BININT1    1
   99: K            BININT1    2
  101: K            BININT1    67
  103: C            SHORT_BINBYTES b't\x00|\x00\x83\x01S\x00'
  113: \x94         MEMOIZE    (as 9)
  114: N            NONE
  115: \x85         TUPLE1
  116: \x94         MEMOIZE    (as 10)
  117: \x8c         SHORT_BINUNICODE 'len'
  122: \x94         MEMOIZE    (as 11)
  123: \x85         TUPLE1
  124: \x94         MEMOIZE    (as 12)
  125: \x8c         SHORT_BINUNICODE 's'
  128: \x94         MEMOIZE    (as 13)
  129: \x85         TUPLE1
  130: \x94         MEMOIZE    (as 14)
  131: \x8c         SHORT_BINUNICODE '/home/vstinner/python/master/cloudpickle_bug.py'
  180: \x94         MEMOIZE    (as 15)
  181: \x8c         SHORT_BINUNICODE 'func'
  187: \x94         MEMOIZE    (as 16)
  188: K            BININT1    4
  190: C            SHORT_BINBYTES b'\x00\x01'
  194: \x94         MEMOIZE    (as 17)
  195: )            EMPTY_TUPLE
  196: )            EMPTY_TUPLE
  197: t            TUPLE      (MARK at 90)
  198: \x94     MEMOIZE    (as 18)
  199: R        REDUCE
  200: \x94     MEMOIZE    (as 19)
  201: }        EMPTY_DICT
  202: \x94     MEMOIZE    (as 20)
  203: (        MARK
  204: \x8c         SHORT_BINUNICODE '__package__'
  217: \x94         MEMOIZE    (as 21)
  218: N            NONE
  219: \x8c         SHORT_BINUNICODE '__name__'
  229: \x94         MEMOIZE    (as 22)
  230: \x8c         SHORT_BINUNICODE '__main__'
  240: \x94         MEMOIZE    (as 23)
  241: \x8c         SHORT_BINUNICODE '__file__'
  251: \x94         MEMOIZE    (as 24)
  252: h            BINGET     15
  254: u            SETITEMS   (MARK at 203)
  255: N        NONE
  256: N        NONE
  257: N        NONE
  258: t        TUPLE      (MARK at 72)
  259: \x94 MEMOIZE    (as 25)
  260: R    REDUCE
  261: \x94 MEMOIZE    (as 26)
  262: \x8c SHORT_BINUNICODE 'cloudpickle.cloudpickle_fast'
  292: \x94 MEMOIZE    (as 27)
  293: \x8c SHORT_BINUNICODE '_function_setstate'
  313: \x94 MEMOIZE    (as 28)
  314: \x93 STACK_GLOBAL
  315: \x94 MEMOIZE    (as 29)
  316: h    BINGET     26
  318: }    EMPTY_DICT
  319: \x94 MEMOIZE    (as 30)
  320: }    EMPTY_DICT
  321: \x94 MEMOIZE    (as 31)
  322: (    MARK
  323: h        BINGET     22
  325: h        BINGET     16
  327: \x8c     SHORT_BINUNICODE '__qualname__'
  341: \x94     MEMOIZE    (as 32)
  342: h        BINGET     16
  344: \x8c     SHORT_BINUNICODE '__annotations__'
  361: \x94     MEMOIZE    (as 33)
  362: }        EMPTY_DICT
  363: \x94     MEMOIZE    (as 34)
  364: \x8c     SHORT_BINUNICODE '__kwdefaults__'
  380: \x94     MEMOIZE    (as 35)
  381: N        NONE
  382: \x8c     SHORT_BINUNICODE '__defaults__'
  396: \x94     MEMOIZE    (as 36)
  397: N        NONE
  398: \x8c     SHORT_BINUNICODE '__module__'
  410: \x94     MEMOIZE    (as 37)
  411: h        BINGET     23
  413: \x8c     SHORT_BINUNICODE '__doc__'
  422: \x94     MEMOIZE    (as 38)
  423: N        NONE
  424: \x8c     SHORT_BINUNICODE '__closure__'
  437: \x94     MEMOIZE    (as 39)
  438: N        NONE
  439: \x8c     SHORT_BINUNICODE '_cloudpickle_submodules'
  464: \x94     MEMOIZE    (as 40)
  465: ]        EMPTY_LIST
  466: \x94     MEMOIZE    (as 41)
  467: \x8c     SHORT_BINUNICODE '__globals__'
  480: \x94     MEMOIZE    (as 42)
  481: }        EMPTY_DICT
  482: \x94     MEMOIZE    (as 43)
  483: u        SETITEMS   (MARK at 322)
  484: \x86 TUPLE2
  485: \x94 MEMOIZE    (as 44)
  486: \x86 TUPLE2
  487: R    REDUCE
  488: 0    POP
  489: .    STOP
highest protocol among opcodes = 4
History
Date User Action Args
2021-02-17 17:42:20vstinnersetrecipients: + vstinner, Mark.Shannon, frenzy, pablogsal
2021-02-17 17:42:20vstinnersetmessageid: <1613583740.24.0.43193531195.issue43228@roundup.psfhosted.org>
2021-02-17 17:42:20vstinnerlinkissue43228 messages
2021-02-17 17:42:19vstinnercreate