classification
Title: function changed when pickle bound method object
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, christian.heimes, georgexsh, maggyero, pitrou, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2019-06-16 01:40 by georgexsh, last changed 2019-06-17 21:01 by brett.cannon.

Files
File name Uploaded Description Edit
0001-fix-bound-method-__reduce__-bug.patch georgexsh, 2019-06-17 01:46
Messages (4)
msg345721 - (view) Author: George Xie (georgexsh) * Date: 2019-06-16 01:40
if we create a bound method object `f` with function object `A.f` and instance object `B()`,
when pickling this bound method object:

	import pickle

	class A():
	    def f(self):
	        pass

	class B():
	    def f(self):
	        pass

	o = B()
	f = A.f.__get__(o)
	pf = pickle.loads(pickle.dumps(f))
	print(f)
	print(pf)

we get:

	<bound method A.f of <__main__.B object at 0x10b82ac18>>
	<bound method B.f of <__main__.B object at 0x10b82ac88>>

the underlaying function are lost, changed from `A.f` to `B.f`.

as pickle calls `__reduce__` method of `method object`, IMO [its implementation][1] simply ignored the real function, whcih is not right.

I have tried a [wordaround][2]:

	import types
	import copyreg

	def my_reduce(obj):
	    return (obj.__func__.__get__, (obj.__self__,))

	copyreg.pickle(types.MethodType, my_reduce)


[1]: https://github.com/python/cpython/blob/v3.7.3/Objects/classobject.c#L75-L89
[2]: https://stackoverflow.com/a/56614748/4201810
msg345776 - (view) Author: Géry (maggyero) * Date: 2019-06-16 20:20
As you stated on Stackoverflow, the fact that the function `A.f` becomes the function `B.f` after a pickling-unpickling sequence creates an infinite recursive call of `B.f` in worker processes started with `multiprocessing.pool.Pool().apply(super().f)` or `concurrent.futures.ProcessPoolExecutor().submit(super().f)` in `B.f` where B is a subclass of A. This infinite recursion has crashed my laptop several times today, so it looks like it is a rather critical issue.
msg345867 - (view) Author: Géry (maggyero) * Date: 2019-06-17 15:24
@George Xie

There is an issue with the Python workaround `multiprocessing.reduction.register(types.MethodType, my_reduce)` that you proposed. If you then instantiate a `multiprocessing.Manager()` you get this exception:

> python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing
>>> import types
>>> def my_reduce(obj):
...     return (obj.__func__.__get__, (obj.__self__,))
...
>>> multiprocessing.reduction.register(types.MethodType, my_reduce)
>>> multiprocessing.Manager()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 56, in Manager
    m.start()
  File "C:\Program Files\Python37\lib\multiprocessing\managers.py", line 543, in start
    self._process.start()
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Python37\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function BaseManager._run_server at 0x000001AEF8ED1D90>: it's not the same object as multiprocessing.managers.BaseManager._run_server
>>> Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 115, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
msg345872 - (view) Author: Géry (maggyero) * Date: 2019-06-17 15:53
The previous `_pickle.PicklingError` seems to only affect Windows.
History
Date User Action Args
2019-06-28 18:50:54bquinlanlinkissue37294 superseder
2019-06-17 21:01:20brett.cannonsetnosy: - brett.cannon
2019-06-17 15:53:53maggyerosetmessages: + msg345872
2019-06-17 15:32:35vstinnersetnosy: - vstinner
2019-06-17 15:24:52maggyerosetmessages: + msg345867
2019-06-17 09:09:20maggyerosetnosy: + vstinner, christian.heimes, asvetlov
2019-06-17 01:46:26georgexshsetfiles: + 0001-fix-bound-method-__reduce__-bug.patch
keywords: + patch
2019-06-16 20:20:16maggyerosetmessages: + msg345776
2019-06-16 20:05:39maggyerosetnosy: + brett.cannon, rhettinger, pitrou, serhiy.storchaka
2019-06-16 09:34:30maggyerosetnosy: + maggyero
2019-06-16 01:40:57georgexshcreate