Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing adds built-in types to the global copyreg.dispatch_table #47600

Closed
avassalotti opened this issue Jul 13, 2008 · 3 comments
Closed
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@avassalotti
Copy link
Member

BPO 3350
Nosy @avassalotti

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-01-25.18:34:49.785>
created_at = <Date 2008-07-13.18:20:41.725>
labels = ['type-bug', 'library']
title = 'multiprocessing adds built-in types to the global copyreg.dispatch_table'
updated_at = <Date 2009-01-25.18:34:49.784>
user = 'https://github.com/avassalotti'

bugs.python.org fields:

activity = <Date 2009-01-25.18:34:49.784>
actor = 'jnoller'
assignee = 'jnoller'
closed = True
closed_date = <Date 2009-01-25.18:34:49.785>
closer = 'jnoller'
components = ['Library (Lib)']
creation = <Date 2008-07-13.18:20:41.725>
creator = 'alexandre.vassalotti'
dependencies = []
files = []
hgrepos = []
issue_num = 3350
keywords = []
message_count = 3.0
messages = ['69615', '70423', '80520']
nosy_count = 3.0
nosy_names = ['alexandre.vassalotti', 'roudkerk', 'jnoller']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue3350'
versions = ['Python 2.6', 'Python 3.0']

@avassalotti
Copy link
Member Author

The multiprocessing module modifies the global copyreg.dispatch_table to
extend pickling to several built-in types (mostly callable types). In my
humble opinion, this is unacceptable for a standard library module.

Here is an example of a behaviour change made by multiprocessing:

  >>> import pickle
  >>> pickle.dumps(str.isalpha)
  Traceback (most recent call last):
    ...
  _pickle.PicklingError: Can't pickle <class 'method_descriptor'>:
attribute lookup builtins.method_descriptor failed
  >>> import multiprocessing.util
  >>> pickle.dumps(str.isalpha)
 
b'\x80\x03cbuiltins\ngetattr\nq\x00cbuiltins\nstr\nq\x01X\x07\x00\x00\x00isalphaq\x02\x86q\x03Rq\x04.'

There was some discussion in bpo-558238 about allowing methods to be
pickled. However, no consensus was reached.

In addition, there is a bug in the method pickling support added by
multiprocessing in Python 3.0:

  Python 2.6b1+ (trunk:64899:64900, Jul 13 2008, 13:33:02) 
  [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import multiprocessing.util, pickle
  >>> class A(object):
  ...   def foo(self): pass
  ... 
  >>> pickle.dumps(A().foo, 2)
 
'\x80\x02c__builtin__\ngetattr\nq\x00c__main__\nA\nq\x01)\x81q\x02}q\x03bU\x03fooq\x04\x86q\x05Rq\x06.'
  >>> pickle.dumps(A.foo, 2)
 
'\x80\x02c__builtin__\ngetattr\nq\x00c__main__\nA\nq\x01U\x03fooq\x02\x86q\x03Rq\x04.'
  
  Python 3.0b1+ (py3k:64876M, Jul 11 2008, 12:20:51) 
  [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import multiprocessing.util, pickle
  >>> class A(object):
  ...   def foo(self): pass
  ... 
  >>> pickle.dumps(A().foo, 2)
  Traceback (most recent call last):
    ...
  pickle.PicklingError: Can't pickle <class 'method'>: it's not found as
builtins.method
  >>> pickle.dumps(A.foo, 2)
  Traceback (most recent call last):
    ...
  pickle.PicklingError: Can't pickle <function foo at 0xb7b1392c>: it's
not found as __main__.foo

A better solution for the interprocess communication needs of
multiprocessing would be to define custom subclass of Pickler with
extended support for built-in types. If needed, I am willing to extend
the _pickle module in Python 3.0 to support modification to its
"dispatch table", like the undocumented dispatch_table attribute in
pickle.Pickler.

@avassalotti avassalotti added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jul 13, 2008
@jnoller
Copy link
Mannequin

jnoller mannequin commented Jul 30, 2008

Alexandre - can you take a look at the solution for bpo-3125 and tell me
if this satisfies your concerns? Note that the merge-forward is blocked in
py3k by bpo-3385 (which is assigned to you)

@jnoller
Copy link
Mannequin

jnoller mannequin commented Jan 25, 2009

Alexandre, this behavior no longer occurs in trunk, and a custom
ForkingPickler was added as part of bpo-3125

I am going to close this as fixed, but please reopen if there is continued
undesired behavior.

@jnoller jnoller mannequin closed this as completed Jan 25, 2009
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

1 participant