This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: [multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
Status: open Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Yoni Rozenshein, davin, pitrou
Priority: normal Keywords:

Created on 2018-06-17 10:45 by Yoni Rozenshein, last changed 2022-04-11 14:59 by admin.

File name Uploaded Description Edit Yoni Rozenshein, 2018-06-17 10:45
Messages (2)
msg319811 - (view) Author: Yoni Rozenshein (Yoni Rozenshein) Date: 2018-06-17 10:45
multiprocessing will attempt to pickle things using ForkingPickler when starting a new process in spawn mode (in Windows this is the only mode, in Linux this is a non-default but settable mode).

When run within the context of a unit test, if it has to pickle a TestCase subclass, it encounters objects that can't be pickled. The attached file shows a minimum working example (uncomment the two commented lines under __main__ to run with pytest).

When run with unittest.main(), it raises:

TypeError: cannot serialize '_io.TextIOWrapper' object

When run with pytest.main(), it raises:

AttributeError: Can't pickle local object 'ArgumentParser.__init__.<locals>.identity'

Note that changing the _child_process in my example to a classmethod/staticmethod or moving it to a top-level function outside the class works around this issue (both with unittest and with pytest).
msg319826 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-06-17 18:51
While this is limitation of the spawn and forkserver modes, I don't see how this is a bug.  Also, as you point out, making the method a classmethod or staticmethod works around the limitation.
Date User Action Args
2022-04-11 14:59:01adminsetstatus: pending -> open
github: 78065
2018-06-17 18:51:59pitrousetstatus: open -> pending
resolution: not a bug
messages: + msg319826

stage: resolved
2018-06-17 17:51:58ned.deilysetnosy: + pitrou, davin
2018-06-17 10:45:32Yoni Rozensheincreate