classification
Title: Add support for classes/object model in multiprocessing/pickle
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Cezary.Wagner, davin, serhiy.storchaka
Priority: normal Keywords:

Created on 2015-02-24 17:01 by Cezary.Wagner, last changed 2015-11-26 17:45 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
issue_23513_play.py davin, 2015-03-02 03:12 proposed runnable version of described "simple example"
Messages (6)
msg236516 - (view) Author: Cezary Wagner (Cezary.Wagner) Date: 2015-02-24 17:01
Currently I trying to write multiprocessing program but it is impossible to write it in Python since I can not use many features and write unpythonic code.

Simple example - how it should work:

class Thing(object):
  class Result(object):
    def __init__(self, something):
      self.something = something

  def worker(self):
    return self.Result(1)

  def master(self):
    pool.apply_async(self.worker)

Now I must do artificial code - not need for anybody and not solving any problem like this - artificial syntax :)

class Result(object):
    def __init__(self, something):
      self.something = something

def call_work_since_can_not_be_called_normally_by_multiprocessing_from_class(self):
  result = self.worker()
  return result

class Thing(object):
  def worker(self):
    return Result(1) # how to overload it??? - no more inheritance

  def master(self):
    ...
    pool.apply_async(worker, args=(self,))

I my opinion that code is ugly and against python Zen.

Why to not fix multiprocessing and pickle to support class.class or class.method - make code simpler and allow more people use parallelism in Python way not procedural way.
msg236517 - (view) Author: Cezary Wagner (Cezary.Wagner) Date: 2015-02-24 17:04
I am fighting with multiprocessing and still not won - I need to invent new Python coding style to use it and can not reduce ugly code/microcoding. It is not because it is not possible to do it in Python easier but because it is not improved yet.
msg236525 - (view) Author: Cezary Wagner (Cezary.Wagner) Date: 2015-02-24 17:35
I found that 'dill' module will solve some problems with class.class class.method lambda but not tested too much and it has big memory footprint about 7M.

'dill' not looks good since code need some polish to be more readable to be easy to review - I study 'dill' algorithm but not want use since code is not very clear and memory usage is high but it solve pickling many object is some way.
msg237008 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2015-03-02 03:12
The runnable example in the attached file, issue_23513_play.py, suggests a way to preserve the inheritance of the Result class in any subclasses of Thing yet leaves the definition of Thing.worker as the OP first had it (in the most straightforward way).

In creating other processes, the authors of multiprocessing needed a way to communicate objects from one process to another and the choice of pickle for serialization does require us to adjust our thinking a little but it does not mean we have to lose being pythonic.  Independent of whether multiprocessing is involved or not, returning a Result object from an instancemethod of another class might work without the definition of the Result class being in the current namespace, but it feels like a much more comprehensive solution to have the Result class available in the current namespace.  The attached proposes defining the Result class outside of the Thing class yet then adding it as a class attribute of Thing -- this gives both inheritability (which was missing in the "simple example") and visibility for pickling.  The convenience of a function like wrap_worker is just one tool that helps pickle figure out how to do its job -- this is part of a larger discussion around pickle and how to gracefully deal with non-trivial situations.

The attached is not being proposed as a final solution but does it help with your situation in particular?  Do parts of it look useful?
msg244150 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-05-27 09:06
Could you please provide full example that you want to work and test it in Python 3.5? I suppose your issue is fixed in 3.5.
msg255429 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-26 17:45
Support of pickling nested classes and methods with all protocols was added in issue23611.
History
Date User Action Args
2015-11-26 17:45:34serhiy.storchakasetstatus: pending -> closed
resolution: out of date
messages: + msg255429

stage: resolved
2015-09-22 18:31:47serhiy.storchakasetstatus: open -> pending
2015-05-27 09:06:35serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg244150
2015-03-02 03:12:03davinsetfiles: + issue_23513_play.py

messages: + msg237008
2015-02-24 17:35:05Cezary.Wagnersetmessages: + msg236525
2015-02-24 17:29:10davinsetnosy: + davin
2015-02-24 17:04:27Cezary.Wagnersetmessages: + msg236517
2015-02-24 17:01:43Cezary.Wagnercreate