Message284755
> Have a look at the sample.py program.
It fails on Python 3.4.3 on my Linux system (just renamed to mp_sample.py):
python3 mp_sample.py
starting (without multiprocessing pool)...
worker process 0
worker process 3
worker process 4
worker process 6
worker process 5
worker process 9
worker process 8
worker process 7
worker process 1
worker process 2
starting (with multiprocessing pool)...
Traceback (most recent call last):
File "mp_sample.py", line 82, in <module>
op.start()
File "mp_sample.py", line 43, in start
result.get()
File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
File "/usr/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
put(task)
File "/usr/lib/python3.4/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
File "/usr/lib/python3.4/multiprocessing/queues.py", line 57, in __getstate__
context.assert_spawning(self)
File "/usr/lib/python3.4/multiprocessing/context.py", line 347, in assert_spawning
' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance
Also:
QueueHandler isn't designed to be pickleable, nor is that necessary for use with multiprocessing. The logging cookbook contains a working example with QueueHandler and multiple processes using multiprocessing, and I've just run it on Python 3.6 (and 3.7) with no problems. See
https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
No idea why the sample program isn't working, and I'm sorry I haven't the time to look at it in detail. Please confirm if the linked sample works on your system, and if it does, see if you can adapt it to your needs. Note that it doesn't pickle any handlers. |
|
Date |
User |
Action |
Args |
2017-01-05 15:54:43 | vinay.sajip | set | recipients:
+ vinay.sajip, serhiy.storchaka, davin, cxss |
2017-01-05 15:54:43 | vinay.sajip | set | messageid: <1483631683.1.0.523588728803.issue29168@psf.upfronthosting.co.za> |
2017-01-05 15:54:43 | vinay.sajip | link | issue29168 messages |
2017-01-05 15:54:43 | vinay.sajip | create | |
|