This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: parameterize what serialization is used in multiprocessing
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.7, Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: davin Nosy List: Will S, davin, eric.snow, i3v, pablogsal, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2016-09-09 20:58 by davin, last changed 2022-04-11 14:58 by admin.

File name Uploaded Description Edit
issue_28053_missingdocs.patch davin, 2016-09-09 21:52 Patch but missing docs review
Pull Requests
URL Status Linked Edit
PR 9959 closed pablogsal, 2018-10-18 22:20
PR 15058 open pierreglaser, 2019-07-31 16:17
Messages (8)
msg275437 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-09 20:58
Currently multiprocessing uses the pickle module for its serialization of objects to be communicated between processes.  Specifically v2 of the pickle protocols is now exclusively used to provide maximum compatibility, motivated by the desire for multiple versions of Python to be used simultaneously with multiprocessing.

Per conversations in issue26507, issue23403, and others, multiprocessing should offer the option to specify what serialization is to be used for the transport of data between processes.  Besides supporting requests to use a different version of the pickle protocol or using 3rd party tools like dill, a hook to specify the means for reducing objects to a transmittable form opens a door for other creative or higher performance strategies.

Ultimately, this is not an enhancement to add functionality but rather to reorganize the existing internals of multiprocessing to permit better control over its use of serialization.
msg275459 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-09 21:52
Attaching patch containing refactorizations but missing update to docs for the purposes of review.

Introduces three new things:
* a function to get the current serializer/reducer
* a function to set the serializer/reducer
* an abstract base class to assist others in rolling their own reducers
msg275479 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2016-09-09 22:39
msg275486 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-09-09 23:03
New changeset 7381b1b50e00 by Davin Potts in branch 'default':
Issue #28053: Applying refactorings, docs and other cleanup to follow.
msg293947 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2017-05-19 00:45
Docs need updating still.
msg298369 - (view) Author: Will S (Will S) Date: 2017-07-14 18:56
Documentation would be appreciated. I have a project that uses BaseManager, Client, and Listener to create some servers and clients. I would like to update the project to work with Python 3 and would prefer to update the clients and the servers separately (i.e. switch the client to Python 3 while the server is run with Python 2.7). However, BaseManager uses connection.Client which uses connection._ConnectionBase which uses reduction.ForkingPickler without a protocol argument. It seems the default protocol is 3 on Python 3.6 and 2 on Python 2.7 (contrary to the comment above about v2 being used). I just want to set the protocol version to 2 in Python 3.6. Can I do that with the changes added by this patch?

I tried creating like this:

from multiprocessing.reduction import ForkingPickler, AbstractReducer

class ForkingPickler2(ForkingPickler):
    def __init__(self, *args):
        if len(args) > 1:
            args[1] = 2

    def dumps(cls, obj, protocol=2):
        return ForkingPickler.dumps(obj, protocol)

def dump(obj, file, protocol=2):
    ForkingPickler2(file, protocol).dump(obj)

class Pickle2Reducer(AbstractReducer):
    ForkingPickler = ForkingPickler2
    register = ForkingPickler2.register
    dump = dump

and then putting

import pickle2reducer
multiprocessing.reducer = pickle2reducer.Pickle2Reducer()

at the top of my module before

import multiprocessing.connection

but I still see "ValueError: unsupported pickle protocol: 3" on the server when I connect with a Python 3.6 client.
msg299434 - (view) Author: Will S (Will S) Date: 2017-07-28 20:19
Just to follow up in case anyone comes across my last message later:

I just had to change the last line from

multiprocessing.reducer = pickle2reducer.Pickle2Reducer()


multiprocessing.context._default_context.reducer = pickle2reducer.Pickle2Reducer()
msg314744 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-31 17:20
I'd like to know if the work here will be completed soon :-) This currently lacks documentation but also tests.

In particular, looking at the code (and that is supported by Will S' comment above), I feel the API isn't working as intended (i.e. setting the "reducer" property won't actually change the underlying parameters since the "reduction" module is imported eagerly).
Date User Action Args
2022-04-11 14:58:36adminsetgithub: 72240
2019-07-31 16:17:41pierreglasersetpull_requests: + pull_request14807
2018-10-18 22:20:26pablogsalsetpull_requests: + pull_request9306
2018-10-18 17:56:21pablogsalsetnosy: + pablogsal
2018-04-08 16:52:58i3vsetnosy: + i3v
2018-04-03 01:08:54serhiy.storchakaunlinkissue23403 superseder
2018-03-31 17:20:39pitrousetnosy: + pitrou
messages: + msg314744
2017-07-28 20:19:04Will Ssetmessages: + msg299434
2017-07-14 18:56:02Will Ssetnosy: + Will S
messages: + msg298369
2017-05-19 00:45:07davinsetmessages: + msg293947
versions: + Python 3.7
2016-09-09 23:03:29python-devsetnosy: + python-dev
messages: + msg275486
2016-09-09 22:39:20eric.snowsetmessages: + msg275479
2016-09-09 21:52:58davinsetfiles: + issue_28053_missingdocs.patch

nosy: + eric.snow
messages: + msg275459

keywords: + patch
stage: patch review
2016-09-09 21:04:52davinlinkissue23403 superseder
2016-09-09 21:02:12davinlinkissue26507 superseder
2016-09-09 21:00:49davinsetcomponents: + Library (Lib)
2016-09-09 20:58:54davincreate