This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: "Can't pickle local object" when uses functools.partial with method and args...
Type: behavior Stage: resolved
Components: asyncio, Library (Lib) Versions: Python 3.6
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: DAVID ALEJANDRO Pineda, asvetlov, fdrake, giampaolo.rodola, serhiy.storchaka, vstinner, yselivanov
Priority: normal Keywords:

Created on 2017-02-09 21:41 by DAVID ALEJANDRO Pineda, last changed 2022-04-11 14:58 by admin. This issue is now closed.

File name Uploaded Description Edit
code_with_bug.png DAVID ALEJANDRO Pineda, 2017-02-09 21:41 Code with bug
Messages (10)
msg287459 - (view) Author: DAVID ALEJANDRO Pineda (DAVID ALEJANDRO Pineda) * Date: 2017-02-09 21:41

I'm working in a real time data collector project. I'm using asyncio and multiprocessing (run_in_executor).
In python 3.5 worked fine but in python 3.6 not, gave to me this error:
 "Traceback (most recent call last):
  File "/usr/local/lib/python3.6/multiprocessing/", line 241, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/local/lib/python3.6/multiprocessing/", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'WeakSet.__init__.<locals>._remove'

I tracked and the problem is when i run some method in run_in_executor with the functools.partial.

Before this version the system works fine.

The main file is (to run with local administration using a socket)
And the engine is in

The project (in development):
msg287887 - (view) Author: DAVID ALEJANDRO Pineda (DAVID ALEJANDRO Pineda) * Date: 2017-02-15 21:18
I try again with different ways to run a method with args, without args works fine to insert in the executor.
I find there are big differences with file functools betwen versions 3.5.1 to 3.6.0.
 Here a more simple example for test the functools:
msg289551 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-13 21:23
Please provide your code as a text, not a picture.
msg289553 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-13 21:29
> AttributeError: Can't pickle local object 'WeakSet.__init__.<locals>._remove'

Well, I don't think we can do something about it. Just re-pack your arguments from WeakSet to a regular set.
msg289560 - (view) Author: DAVID ALEJANDRO Pineda (DAVID ALEJANDRO Pineda) * Date: 2017-03-14 03:17
Hello Again.

The problem can be replicated in the same structure when call the 'functools.partial' feature

I wrote a simple example. It uses the asyncio and multiprocessing structure.

With this dependences:

In python 3.5.1 works fine even if in the mprocess file uncomment the 'functools.partial'. But when i use a larger version, like 3.6, fails with the 'weakref'
msg289567 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-14 06:04
If the example worked on old versions of Python, perhaps this is a regression. Either WeakSet was pickleable or it was not used in that asyncio/multiprocessing use. If this is a regression in the stdlib it should be fixed.
msg289575 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-03-14 10:03
Can you write a short Python script producing the bug?
msg296244 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-17 19:02
Since WeakSet never was pickleable, the difference perhaps in the use of WeakSet in asyncio. BaseEventLoop contains a WeakSet attribute since 3.6, this makes it non-pickleable.

I'm not sure this is a cause of the failure.
msg308801 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2017-12-20 20:12
asyncio objects are not pickable and shouldn't be -- they are ephemeral.

Like pickling open socket object doesn't make sense -- there is no possibility to restore connection to peer on unpickling.

Not sure if we can do anything here.

I suggest closing the issue.
msg308802 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-12-20 20:14
Closing this one.

Quoting my other comment from this issue:

> Well, I don't think we can do something about it. Just re-pack your arguments from WeakSet to a regular set.
Date User Action Args
2022-04-11 14:58:43adminsetgithub: 73703
2017-12-20 20:14:36yselivanovsetstatus: open -> closed
resolution: not a bug
messages: + msg308802

stage: resolved
2017-12-20 20:12:51asvetlovsetnosy: + asvetlov
messages: + msg308801
2017-06-17 19:02:32serhiy.storchakasetmessages: + msg296244
components: + Library (Lib)
2017-03-14 10:03:57vstinnersetmessages: + msg289575
2017-03-14 06:04:26serhiy.storchakasetnosy: + fdrake, vstinner, giampaolo.rodola
messages: + msg289567
2017-03-14 03:17:09DAVID ALEJANDRO Pinedasetmessages: + msg289560
2017-03-13 21:29:47yselivanovsetmessages: + msg289553
2017-03-13 21:23:46serhiy.storchakasetmessages: + msg289551
2017-02-15 21:28:40gvanrossumsetnosy: - gvanrossum
2017-02-15 21:18:46DAVID ALEJANDRO Pinedasetmessages: + msg287887
2017-02-10 01:15:51r.david.murraysetnosy: + serhiy.storchaka
2017-02-09 21:41:25DAVID ALEJANDRO Pinedacreate