This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python 3.7.0 multiprocessing forkserver ForkingPickler behaviour change
Type: Stage: resolved
Components: Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Santiago Hernandez, pitrou
Priority: normal Keywords:

Created on 2018-07-03 18:06 by Santiago Hernandez, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
example.py Santiago Hernandez, 2018-07-03 18:06 A minimal example where the problem happens
Messages (4)
msg320995 - (view) Author: Santiago Hernandez (Santiago Hernandez) Date: 2018-07-03 18:06
Hi,

I am splitting some processing among different processes with the forkserver start method using multiprocessing.
In python 3.4 to 3.6 this worked fine.
In python 3.7.0 it's breaking with the following Traceback:

```
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/data/venvs/py37/Python-3.7.0/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle weakref objects
```

I am attaching a minimum example where this happens.
In the example this can be solved by making self._run_single a staticmethod.

However, I would like to understand what is going on and what changed in python for this to throw an exception now, when before it did not.

Thanks.
Santiago
msg321000 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-07-03 19:38
It could be many different things.  The bottom line here, though, is that the Process class is not designed to be picklable (how would it work?), which is probably why you're seeing this.
msg321190 - (view) Author: Santiago Hernandez (Santiago Hernandez) Date: 2018-07-06 19:49
Thanks, I get it now, I hadn't seen that the pickling of the workers was
Processes pickling.
Yeah, I was not doing this anywhere else, it seems like slipped through
once. But it called my attention that it used to work and now it doesn't.

On Tue, Jul 3, 2018 at 4:38 PM, Antoine Pitrou <report@bugs.python.org>
wrote:

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> It could be many different things.  The bottom line here, though, is that
> the Process class is not designed to be picklable (how would it work?),
> which is probably why you're seeing this.
>
> ----------
> nosy: +pitrou
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue34034>
> _______________________________________
>
msg321191 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-07-06 20:05
No problem! I'm going to close this issue as I don't think there's anything we can do here.
History
Date User Action Args
2022-04-11 14:59:02adminsetgithub: 78215
2018-07-06 20:05:11pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg321191

stage: resolved
2018-07-06 19:49:46Santiago Hernandezsetmessages: + msg321190
2018-07-03 19:38:09pitrousetnosy: + pitrou
messages: + msg321000
2018-07-03 18:06:26Santiago Hernandezcreate