This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Started multiprocessing.Process instances are unserialisable
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.11
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: maggyero
Priority: normal Keywords: patch

Created on 2022-03-05 22:19 by maggyero, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 31701 open maggyero, 2022-03-05 23:17
Messages (1)
msg414597 - (view) Author: Géry (maggyero) * Date: 2022-03-05 22:19
The Python program:

import multiprocessing
import time

class Application:

    def __init__(self):
        self._event = multiprocessing.Event()
        self._processes = [
            for _ in range(multiprocessing.cpu_count())]

    def _worker(self):
        while not self._event.is_set():

    def start(self):
        for process in self._processes:

    def stop(self):
        for process in self._processes:

if __name__ == '__main__':
    application = Application()

Its output:

Traceback (most recent call last):
  File "/Users/maggyero/Desktop/", line 31, in <module>
  File "/Users/maggyero/Desktop/", line 21, in start
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 32, in __init__
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 19, in __init__
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

In the function `Application.__init__`, each call `multiprocessing.Process(target=self._worker)` initializes a `multiprocessing.Process` instance with the instance method `self._worker` as its `target` argument. `self._worker` is bound to `self` which has the instance attribute `self._processes`.

In the function `Application.start`, each call `process.start()` serialises the `target` argument and therefore `self._processes`. `self._processes` is a list of `multiprocessing.Process` instances, initially not started yet. The first call `process.start()` starts the first `multiprocessing.Process` instance in that list without issue, but the second call `process.start()` fails.

So a started `multiprocessing.Process` instance cannot be serialised.

The root of the problem is that the `start` method of a `multiprocessing.Process` instance sets its `_popen` instance attribute to a `multiprocessing.popen_*.Popen` instance. The initialization of that instance performs these two steps (among others):

1. For a `multiprocessing.popen_spawn_posix.Popen` instance, a `multiprocessing.popen_spawn_win32.Popen` instance, or a `multiprocessing.popen_forkserver.Popen` instance but not a `multiprocessing.popen_fork.Popen` instance (i.e. for the start method `'spawn'` or the start method `'forkserver'` but not the start method `'fork'`), it [serialises]( the `multiprocessing.Process` instance for writing it to the end of the pipe used by the parent process to communicate with the child process so that the child process can execute the `run` method of the `multiprocessing.Process` instance.

2. It [sets]( its `finalizer` instance attribute to a `multiprocessing.util.Finalize` instance which itself sets its `_weakref` instance attribute to a `weakref.ref` instance for closing at interpreter exit the ends of the pipes used by the parent process to communicate with the child process. In other words, it makes the `multiprocessing.Process` instance hold a weak reference.

Thus if a `multiprocessing.Process` instance holds a reference to a started `multiprocessing.Process` instance then it holds a weak reference (point 2), so starting it will fail since it will serialise (point 1) the weak reference and weak references are not serialisable:

import multiprocessing

if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')  # or 'forkserver' but not 'fork'
    process_a = multiprocessing.Process()
    process_b = multiprocessing.Process() = process_a
    process_a.start()  # creates process_a._popen.finalizer._weakref
    process_b.start()  # TypeError: cannot pickle 'weakref' object

A minimal Python program showing the serialisation issue:

import pickle
import weakref

pickle.dumps(weakref.ref(int))  # TypeError: cannot pickle 'weakref' object
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91090
2022-03-05 23:19:58maggyerosettitle: A started multiprocessing.Process instance cannot be serialised -> Started multiprocessing.Process instances are unserialisable
2022-03-05 23:17:52maggyerosetkeywords: + patch
stage: patch review
pull_requests: + pull_request29821
2022-03-05 22:19:15maggyerocreate