msg134907 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-01 13:25 |
Hello,
I found this bit in my inbox, I forgot why I cared about it, but it raises an exception (at least on Windows):
>>> import multiprocessing
>>> p = multiprocessing.Process(target=bytes.maketrans, args=(b'abc', b'xyz'))
>>> p.start()
Traceback (most recent call last):
File "C:\Python32\Lib\pickle.py", line 679, in save_global
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'maketrans'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python32\Lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Python32\Lib\multiprocessing\forking.py", line 267, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python32\Lib\multiprocessing\forking.py", line 190, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Python32\Lib\pickle.py", line 237, in dump
self.save(obj)
File "C:\Python32\Lib\pickle.py", line 344, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python32\Lib\pickle.py", line 432, in save_reduce
save(state)
File "C:\Python32\Lib\pickle.py", line 299, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python32\Lib\pickle.py", line 623, in save_dict
self._batch_setitems(obj.items())
File "C:\Python32\Lib\pickle.py", line 656, in _batch_setitems
save(v)
File "C:\Python32\Lib\pickle.py", line 299, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python32\Lib\pickle.py", line 683, in save_global
(obj, module, name))
_pickle.PicklingError: Can't pickle <built-in function maketrans>: it's not found as __main__.maketrans
If you do the same things with `threading.Thread`, it works, but for `multiprocessing.Process` it doesn't. Is this a general problem with pickling "unbound methods"?
|
msg135392 - (view) |
Author: Terry J. Reedy (terry.reedy) *  |
Date: 2011-05-07 00:33 |
Running on winxp with IDLE, I get the second traceback, all the same after the first line. Given "target is the callable object to be invoked by the run() method.", I would have expected this to work too.
Problem is not builtins:
class C:
def f(s): print( 'here')
p = multiprocessing.Process(target=C.f, args=(C(),))
p.start()
gives same error, ending in
_pickle.PicklingError: Can't pickle <function f at 0x00FE5AE0>: it's not found as __main__.f
|
msg135460 - (view) |
Author: Jesse Noller (jnoller) *  |
Date: 2011-05-07 13:05 |
Do things like this in the REPL are known not to work. Things are not protected in the if __name__ block so we can import the script properly to run it.
|
msg136270 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-19 09:15 |
Test attached.
|
msg136935 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-26 09:51 |
Why is this still marked as "test needed"?
|
msg136936 - (view) |
Author: Ezio Melotti (ezio.melotti) *  |
Date: 2011-05-26 09:54 |
The test should be a diff against Lib/test/test_multiprocessing.py that adds a unit test.
|
msg137044 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-27 10:59 |
Diff attached, is it good? I'm not very experienced with diffs, I usually work with pull requests.
|
msg137045 - (view) |
Author: Petri Lehtinen (petri.lehtinen) *  |
Date: 2011-05-27 11:21 |
Your patch is good in this case, as the person who applies the patch knows which file is affected.
In the future, use diff -u original_file modified_file to get a unified diff. It's the "de facto" format for patches.
|
msg137046 - (view) |
Author: Petri Lehtinen (petri.lehtinen) *  |
Date: 2011-05-27 11:28 |
Actually, I only commented on the patch format and not on the actual contents of the patch, sorry :)
Your test method missed the self parameter, and the test case needed to be added to the testcases_other list for the test to be actually run. I attached a corrected patch.
And, now that I tested it, the test runs correctly on Linux, so maybe this is a Windows specific issue?
|
msg137061 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2011-05-27 15:26 |
Thanks for the `-u` tip and the correction to the code, Petri. I removed my previous files since yours is the definite one. And yeah, it's a Windows issue.
|
msg137066 - (view) |
Author: Terry J. Reedy (terry.reedy) *  |
Date: 2011-05-27 15:48 |
Jesse, I do not understand your comment, including 'REPL'
|
msg137079 - (view) |
Author: R. David Murray (r.david.murray) *  |
Date: 2011-05-27 16:31 |
REPL is the Python interactive prompt in this case (REPL is Read Eval Print Loop). So Jesse is saying that using multiprocessing from the REPL (at least on Windows) isn't supported. This is because on Windows multiprocessing needs to re-import the main program in order to start the function in a worker process. When using the REPL, there is no main program from which to import the function.
The test case failing on windows may or may not be a related issue; I'm not familiar enough with mulitprocessing to say.
|
msg137090 - (view) |
Author: Terry J. Reedy (terry.reedy) *  |
Date: 2011-05-27 17:57 |
After looking at the doc chapter, I get that 'if __name__' block is needed on Windows.
OK. batch mode with if __name__ block:
testmp.py
---------
print('Top of Module')
class C:
def f(s): print('Method C.f')
if __name__ == '__main__':
print('Start of main block')
import multiprocessing
p = multiprocessing.Process(target=C.f, args=(C(),))
p.start()
p.join()
command-prompt..\python32> python @misc2\testmp.py
output (similar to that of OP):
Top of Module
Start of main block
Traceback (most recent call last):
File "C:\Programs\Python32\lib\pickle.py", line 679, in save_global
klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'f'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "@misc2\testmp.py", line 9, in <module>
p.start()
File "C:\Programs\Python32\lib\multiprocessing\process.py", line 130, in start
self._popen = Popen(self)
File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 267, in __init__
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 190, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Programs\Python32\lib\pickle.py", line 237, in dump
self.save(obj)
File "C:\Programs\Python32\lib\pickle.py", line 344, in save
self.save_reduce(obj=obj, *rv)
File "C:\Programs\Python32\lib\pickle.py", line 432, in save_reduce
save(state)
File "C:\Programs\Python32\lib\pickle.py", line 299, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Programs\Python32\lib\pickle.py", line 623, in save_dict
self._batch_setitems(obj.items())
File "C:\Programs\Python32\lib\pickle.py", line 656, in batch_setitems
save(v)
File "C:\Programs\Python32\lib\pickle.py", line 299, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Programs\Python32\lib\pickle.py", line 683, in save_global
(obj, module, name))
_pickle.PicklingError: Can't pickle <function f at 0x00B4B4B0>: it's not found as __main__.f
C:\Programs\Python32>Top of Module
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 370, in main
self = load(from_parent)
EOFError
The essential problem seems to be the attempt to pickle the method attribute of the class as a class attribute of the module.
|
msg224242 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2014-07-29 20:04 |
This works perfectly on 64 bit Windows 8.1 for 3.4.1 and 3.5.0a0.
|
msg224243 - (view) |
Author: Ram Rachum (cool-RR) * |
Date: 2014-07-29 20:07 |
Confirmed here it's working in Python 3.4, I guess it was fixed sometime in the last few years.
I guess the only thing we'd care about now is ensuring a test for this was added to the test suite, so there wouldn't be a regression. Can anyone confirm that?
|
msg224277 - (view) |
Author: Antoine Pitrou (pitrou) *  |
Date: 2014-07-30 00:18 |
Pickling of builtin functions and methods was indeed improved thanks to __qualname__ support. Closing.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:16 | admin | set | github: 56178 |
2014-07-30 00:18:22 | pitrou | set | status: open -> closed
nosy:
+ pitrou messages:
+ msg224277
resolution: out of date stage: needs patch -> resolved |
2014-07-29 20:07:48 | cool-RR | set | messages:
+ msg224243 |
2014-07-29 20:04:04 | BreamoreBoy | set | nosy:
+ BreamoreBoy messages:
+ msg224242
|
2011-08-28 16:56:15 | terry.h | set | nosy:
+ terry.h
|
2011-07-08 09:12:34 | petri.lehtinen | set | stage: test needed -> needs patch |
2011-05-27 17:57:20 | terry.reedy | set | messages:
+ msg137090 |
2011-05-27 16:31:06 | r.david.murray | set | nosy:
+ r.david.murray messages:
+ msg137079
|
2011-05-27 15:48:08 | terry.reedy | set | messages:
+ msg137066 components:
+ Windows versions:
+ Python 3.2, - Python 3.4 |
2011-05-27 15:26:00 | cool-RR | set | messages:
+ msg137061 |
2011-05-27 15:23:10 | cool-RR | set | files:
- test.py |
2011-05-27 15:23:01 | cool-RR | set | files:
- patch.diff |
2011-05-27 11:28:08 | petri.lehtinen | set | files:
+ patch_2.diff
messages:
+ msg137046 |
2011-05-27 11:21:08 | petri.lehtinen | set | nosy:
+ petri.lehtinen messages:
+ msg137045
|
2011-05-27 10:59:11 | cool-RR | set | files:
+ patch.diff keywords:
+ patch messages:
+ msg137044
|
2011-05-26 09:54:11 | ezio.melotti | set | nosy:
+ ezio.melotti messages:
+ msg136936
|
2011-05-26 09:51:41 | cool-RR | set | messages:
+ msg136935 |
2011-05-19 09:15:26 | cool-RR | set | files:
+ test.py
messages:
+ msg136270 |
2011-05-07 13:05:59 | jnoller | set | messages:
+ msg135460 |
2011-05-07 00:36:08 | terry.reedy | set | title: Can't launch Process on built-in static method -> Can't launch multiproccessing.Process on methods stage: test needed |
2011-05-07 00:33:16 | terry.reedy | set | nosy:
+ terry.reedy, jnoller messages:
+ msg135392
|
2011-05-01 13:25:55 | cool-RR | create | |