classification
Title: Can't launch multiproccessing.Process on methods
Type: behavior Stage: resolved
Components: Interpreter Core, Library (Lib), Windows Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, cool-RR, ezio.melotti, jnoller, petri.lehtinen, pitrou, r.david.murray, terry.h, terry.reedy
Priority: normal Keywords: patch

Created on 2011-05-01 13:25 by cool-RR, last changed 2014-07-30 00:18 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
patch_2.diff petri.lehtinen, 2011-05-27 11:28 Test case review
Messages (16)
msg134907 - (view) Author: Ram Rachum (cool-RR) * Date: 2011-05-01 13:25
Hello,

I found this bit in my inbox, I forgot why I cared about it, but it raises an exception (at least on Windows):

>>> import multiprocessing
>>> p = multiprocessing.Process(target=bytes.maketrans, args=(b'abc', b'xyz'))
>>> p.start()
Traceback (most recent call last):
  File "C:\Python32\Lib\pickle.py", line 679, in save_global
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'maketrans'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python32\Lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python32\Lib\multiprocessing\forking.py", line 267, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python32\Lib\multiprocessing\forking.py", line 190, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python32\Lib\pickle.py", line 237, in dump
    self.save(obj)
  File "C:\Python32\Lib\pickle.py", line 344, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python32\Lib\pickle.py", line 432, in save_reduce
    save(state)
  File "C:\Python32\Lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python32\Lib\pickle.py", line 623, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Python32\Lib\pickle.py", line 656, in _batch_setitems
    save(v)
  File "C:\Python32\Lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python32\Lib\pickle.py", line 683, in save_global
    (obj, module, name))
_pickle.PicklingError: Can't pickle <built-in function maketrans>: it's not found as __main__.maketrans

If you do the same things with `threading.Thread`, it works, but for `multiprocessing.Process` it doesn't. Is this a general problem with pickling "unbound methods"?
msg135392 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-07 00:33
Running on winxp with IDLE, I get the second traceback, all the same after the first line. Given "target is the callable object to be invoked by the run() method.", I would have expected this to work too.

Problem is not builtins:
class C:
    def f(s): print( 'here')

p = multiprocessing.Process(target=C.f, args=(C(),))
p.start()

gives same error, ending in
_pickle.PicklingError: Can't pickle <function f at 0x00FE5AE0>: it's not found as __main__.f
msg135460 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2011-05-07 13:05
Do things like this in the REPL are known not to work. Things are not protected in the if __name__ block so we can import the script properly to run it.
msg136270 - (view) Author: Ram Rachum (cool-RR) * Date: 2011-05-19 09:15
Test attached.
msg136935 - (view) Author: Ram Rachum (cool-RR) * Date: 2011-05-26 09:51
Why is this still marked as "test needed"?
msg136936 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-05-26 09:54
The test should be a diff against Lib/test/test_multiprocessing.py that adds a unit test.
msg137044 - (view) Author: Ram Rachum (cool-RR) * Date: 2011-05-27 10:59
Diff attached, is it good? I'm not very experienced with diffs, I usually work with pull requests.
msg137045 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-05-27 11:21
Your patch is good in this case, as the person who applies the patch knows which file is affected.

In the future, use diff -u original_file modified_file to get a unified diff. It's the "de facto" format for patches.
msg137046 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-05-27 11:28
Actually, I only commented on the patch format and not on the actual contents of the patch, sorry :)

Your test method missed the self parameter, and the test case needed to be added to the testcases_other list for the test to be actually run. I attached a corrected patch.

And, now that I tested it, the test runs correctly on Linux, so maybe this is a Windows specific issue?
msg137061 - (view) Author: Ram Rachum (cool-RR) * Date: 2011-05-27 15:26
Thanks for the `-u` tip and the correction to the code, Petri. I removed my previous files since yours is the definite one. And yeah, it's a Windows issue.
msg137066 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-27 15:48
Jesse, I do not understand your comment, including 'REPL'
msg137079 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-27 16:31
REPL is the Python interactive prompt in this case (REPL is Read Eval Print Loop).  So Jesse is saying that using multiprocessing from the REPL (at least on Windows) isn't supported.  This is because on Windows multiprocessing needs to re-import the main program in order to start the function in a worker process.  When using the REPL, there is no main program from which to import the function.

The test case failing on windows may or may not be a related issue; I'm not familiar enough with mulitprocessing to say.
msg137090 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-05-27 17:57
After looking at the doc chapter, I get that 'if __name__' block is needed on Windows.

OK. batch mode with if __name__ block:

testmp.py
---------
print('Top of Module')
class C:
    def f(s): print('Method C.f')

if __name__ == '__main__':
    print('Start of main block')
    import multiprocessing
    p = multiprocessing.Process(target=C.f, args=(C(),))
    p.start()
    p.join()

command-prompt..\python32> python @misc2\testmp.py

output (similar to that of OP):

Top of Module
Start of main block
Traceback (most recent call last):
  File "C:\Programs\Python32\lib\pickle.py", line 679, in save_global
    klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'f'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "@misc2\testmp.py", line 9, in <module>
    p.start()
  File "C:\Programs\Python32\lib\multiprocessing\process.py", line 130, in start

    self._popen = Popen(self)
  File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 267, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 190, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Programs\Python32\lib\pickle.py", line 237, in dump
    self.save(obj)
  File "C:\Programs\Python32\lib\pickle.py", line 344, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Programs\Python32\lib\pickle.py", line 432, in save_reduce
    save(state)
  File "C:\Programs\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Programs\Python32\lib\pickle.py", line 623, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Programs\Python32\lib\pickle.py", line 656, in batch_setitems
    save(v)
  File "C:\Programs\Python32\lib\pickle.py", line 299, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Programs\Python32\lib\pickle.py", line 683, in save_global
    (obj, module, name))
_pickle.PicklingError: Can't pickle <function f at 0x00B4B4B0>: it's not found as __main__.f

C:\Programs\Python32>Top of Module
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 370, in main
    self = load(from_parent)
EOFError

The essential problem seems to be the attempt to pickle the method attribute of the class as a class attribute of the module.
msg224242 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-29 20:04
This works perfectly on 64 bit Windows 8.1 for 3.4.1 and 3.5.0a0.
msg224243 - (view) Author: Ram Rachum (cool-RR) * Date: 2014-07-29 20:07
Confirmed here it's working in Python 3.4, I guess it was fixed sometime in the last few years.

I guess the only thing we'd care about now is ensuring a test for this was added to the test suite, so there wouldn't be a regression. Can anyone confirm that?
msg224277 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-30 00:18
Pickling of builtin functions and methods was indeed improved thanks to __qualname__ support. Closing.
History
Date User Action Args
2014-07-30 00:18:22pitrousetstatus: open -> closed

nosy: + pitrou
messages: + msg224277

resolution: out of date
stage: needs patch -> resolved
2014-07-29 20:07:48cool-RRsetmessages: + msg224243
2014-07-29 20:04:04BreamoreBoysetnosy: + BreamoreBoy
messages: + msg224242
2011-08-28 16:56:15terry.hsetnosy: + terry.h
2011-07-08 09:12:34petri.lehtinensetstage: test needed -> needs patch
2011-05-27 17:57:20terry.reedysetmessages: + msg137090
2011-05-27 16:31:06r.david.murraysetnosy: + r.david.murray
messages: + msg137079
2011-05-27 15:48:08terry.reedysetmessages: + msg137066
components: + Windows
versions: + Python 3.2, - Python 3.4
2011-05-27 15:26:00cool-RRsetmessages: + msg137061
2011-05-27 15:23:10cool-RRsetfiles: - test.py
2011-05-27 15:23:01cool-RRsetfiles: - patch.diff
2011-05-27 11:28:08petri.lehtinensetfiles: + patch_2.diff

messages: + msg137046
2011-05-27 11:21:08petri.lehtinensetnosy: + petri.lehtinen
messages: + msg137045
2011-05-27 10:59:11cool-RRsetfiles: + patch.diff
keywords: + patch
messages: + msg137044
2011-05-26 09:54:11ezio.melottisetnosy: + ezio.melotti
messages: + msg136936
2011-05-26 09:51:41cool-RRsetmessages: + msg136935
2011-05-19 09:15:26cool-RRsetfiles: + test.py

messages: + msg136270
2011-05-07 13:05:59jnollersetmessages: + msg135460
2011-05-07 00:36:08terry.reedysettitle: Can't launch Process on built-in static method -> Can't launch multiproccessing.Process on methods
stage: test needed
2011-05-07 00:33:16terry.reedysetnosy: + terry.reedy, jnoller
messages: + msg135392
2011-05-01 13:25:55cool-RRcreate