➜
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2011-12-09 19:57 by jango, last changed 2022-04-11 14:57 by admin. This issue is now closed.
| Messages (16) | |||
|---|---|---|---|
| msg149118 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-09 19:57 | |
I've wrote a little application that uses multiprocessing module: https://github.com/jango/PC/blob/master/pc/pc-example.py When I run it in my Linux setup, I get the expected output (Python 2.7.1+): 2011-12-09 14:16:29,014 Started Producer 0. 2011-12-09 14:16:29,076 Started Producer 1. 2011-12-09 14:16:29,076 Producer process (0) exited... 2011-12-09 14:16:29,131 Started Producer 2. 2011-12-09 14:16:29,188 Started Producer 3. 2011-12-09 14:16:29,218 Started Producer 4. 2011-12-09 14:16:29,281 Started Producer 5. 2011-12-09 14:16:29,327 Started Producer 6. 2011-12-09 14:16:29,377 Started Producer 7. 2011-12-09 14:16:29,403 Producer process (1) exited... 2011-12-09 14:16:29,412 Started Producer 8. 2011-12-09 14:16:29,504 Started Producer 9. 2011-12-09 14:16:29,528 Producer process (4) exited... 2011-12-09 14:16:29,570 Started Producer 10. 2011-12-09 14:16:29,615 Started Producer 11. 2011-12-09 14:16:29,622 Producer process (3) exited... 2011-12-09 14:16:29,637 Producer process (7) exited... 2011-12-09 14:16:29,653 Producer process (5) exited... 2011-12-09 14:16:29,692 Started Producer 12. 2011-12-09 14:16:29,731 Producer process (12) exited... 2011-12-09 14:16:29,747 Started Producer 13. 2011-12-09 14:16:29,809 Started Producer 14. 2011-12-09 14:16:29,860 Started Producer 15. 2011-12-09 14:16:29,903 Producer process (2) exited... 2011-12-09 14:16:29,905 Started Producer 16. 2011-12-09 14:16:29,918 Producer process (8) exited... 2011-12-09 14:16:29,970 Started Producer 17. 2011-12-09 14:16:30,001 Started Producer 18. 2011-12-09 14:16:30,070 Started Producer 19. 2011-12-09 14:16:30,090 Producer process (11) exited... 2011-12-09 14:16:30,105 Producer process (10) exited... 2011-12-09 14:16:30,137 Started Producer 20. 2011-12-09 14:16:30,152 Producer process (6) exited... 2011-12-09 14:16:30,183 Producer process (9) exited... 2011-12-09 14:16:30,183 Producer process (17) exited... 2011-12-09 14:16:30,251 Started Producer 21. 2011-12-09 14:16:30,288 Started Producer 22. 2011-12-09 14:16:30,308 Producer process (18) exited... 2011-12-09 14:16:30,308 Producer process (20) exited... 2011-12-09 14:16:30,355 Producer process (16) exited... 2011-12-09 14:16:30,380 Started Producer 23. 2011-12-09 14:16:30,436 Started Producer 24. 2011-12-09 14:16:30,472 Started Producer 25. 2011-12-09 14:16:30,480 Producer process (22) exited... 2011-12-09 14:16:30,509 Started Producer 26. 2011-12-09 14:16:30,554 Started Producer 27. 2011-12-09 14:16:30,609 Started Producer 28. 2011-12-09 14:16:30,620 Producer process (14) exited... 2011-12-09 14:16:30,636 Producer process (15) exited... 2011-12-09 14:16:30,660 Started Producer 29. 2011-12-09 14:16:30,667 Producer process (19) exited... 2011-12-09 14:16:30,667 Producer process (13) exited... 2011-12-09 14:16:30,667 Producer process (27) exited... 2011-12-09 14:16:30,750 Started Consumer 0. 2011-12-09 14:16:30,761 --> 0 produces 0. 2011-12-09 14:16:30,776 Producer process (28) exited... 2011-12-09 14:16:30,770 --> 1 produces 10. 2011-12-09 14:16:30,783 --> 4 produces 40. 2011-12-09 14:16:30,797 Started Consumer 1. 2011-12-09 14:16:30,807 --> 3 produces 30. 2011-12-09 14:16:30,816 --> 5 produces 50. 2011-12-09 14:16:30,817 --> 12 produces 120. 2011-12-09 14:16:30,819 --> 2 produces 20. 2011-12-09 14:16:30,826 --> 8 produces 80. 2011-12-09 14:16:30,835 --> 11 produces 110. 2011-12-09 14:16:30,848 --> 10 produces 100. 2011-12-09 14:16:30,849 --> 6 produces 60. 2011-12-09 14:16:30,852 --> 17 produces 170. 2011-12-09 14:16:30,853 --> 9 produces 90. 2011-12-09 14:16:30,859 --> 18 produces 180. 2011-12-09 14:16:30,860 --> 20 produces 200. 2011-12-09 14:16:30,865 --> 16 produces 160. 2011-12-09 14:16:30,866 --> 22 produces 220. 2011-12-09 14:16:30,867 --> 14 produces 140. 2011-12-09 14:16:30,868 --> 15 produces 150. 2011-12-09 14:16:30,869 --> 19 produces 190. 2011-12-09 14:16:30,874 --> 13 produces 130. 2011-12-09 14:16:30,823 Producer process (21) exited... 2011-12-09 14:16:30,812 --> 7 produces 70. 2011-12-09 14:16:30,893 --> 28 produces 280. 2011-12-09 14:16:30,854 Producer process (25) exited... 2011-12-09 14:16:30,894 Started Consumer 2. 2011-12-09 14:16:30,875 --> 27 produces 270. 2011-12-09 14:16:30,905 --> 25 produces 250. 2011-12-09 14:16:30,917 --> 21 produces 210. 2011-12-09 14:16:30,943 Started Consumer 3. 2011-12-09 14:16:31,002 Started Consumer 4. 2011-12-09 14:16:31,049 Started Consumer 5. 2011-12-09 14:16:31,091 Started Consumer 6. 2011-12-09 14:16:31,137 Started Consumer 7. 2011-12-09 14:16:31,197 Producer process (23) exited... 2011-12-09 14:16:31,213 --> 23 produces 230. 2011-12-09 14:16:31,206 Started Consumer 8. 2011-12-09 14:16:31,263 Started Consumer 9. 2011-12-09 14:16:31,275 Producer process (24) exited... 2011-12-09 14:16:31,307 Started Consumer 10. 2011-12-09 14:16:31,322 --> 24 produces 240. 2011-12-09 14:16:31,357 Started Consumer 11. 2011-12-09 14:16:31,386 Started Consumer 12. 2011-12-09 14:16:31,400 Producer process (26) exited... 2011-12-09 14:16:31,432 Started Consumer 13. 2011-12-09 14:16:31,431 --> 26 produces 260. 2011-12-09 14:16:31,476 Started Consumer 14. 2011-12-09 14:16:31,527 Started Consumer 15. 2011-12-09 14:16:31,556 Producer process (29) exited... 2011-12-09 14:16:31,570 Started Consumer 16. 2011-12-09 14:16:31,587 --> 29 produces 290. 2011-12-09 14:16:31,630 Started Consumer 17. 2011-12-09 14:16:31,664 Started Consumer 18. 2011-12-09 14:16:31,714 Started Consumer 19. 2011-12-09 14:16:31,768 Started Consumer 20. 2011-12-09 14:16:31,815 Started Consumer 21. 2011-12-09 14:16:31,847 Started Consumer 22. 2011-12-09 14:16:31,903 Started Consumer 23. 2011-12-09 14:16:31,932 Started Consumer 24. 2011-12-09 14:16:31,981 Started Consumer 25. 2011-12-09 14:16:32,023 Started Consumer 26. 2011-12-09 14:16:32,073 Started Consumer 27. 2011-12-09 14:16:32,121 Started Consumer 28. 2011-12-09 14:16:32,175 Started Consumer 29. 2011-12-09 14:16:32,181 Consumer (0): instance received shutdown call... 2011-12-09 14:16:32,182 Consumer (1): instance received shutdown call... 2011-12-09 14:16:32,185 Consumer (2): instance received shutdown call... 2011-12-09 14:16:32,186 Consumer (3): instance received shutdown call... 2011-12-09 14:16:32,187 Consumer (4): instance received shutdown call... 2011-12-09 14:16:32,188 Consumer (5): instance received shutdown call... 2011-12-09 14:16:32,189 Consumer (6): instance received shutdown call... 2011-12-09 14:16:32,190 Consumer (7): instance received shutdown call... 2011-12-09 14:16:32,191 Consumer (8): instance received shutdown call... 2011-12-09 14:16:32,192 Consumer (9): instance received shutdown call... 2011-12-09 14:16:32,193 Consumer (10): instance received shutdown call... 2011-12-09 14:16:32,194 Consumer (11): instance received shutdown call... 2011-12-09 14:16:32,195 Consumer (12): instance received shutdown call... 2011-12-09 14:16:32,196 Consumer (13): instance received shutdown call... 2011-12-09 14:16:32,197 Consumer (14): instance received shutdown call... 2011-12-09 14:16:32,198 Consumer (15): instance received shutdown call... 2011-12-09 14:16:32,199 Consumer (16): instance received shutdown call... 2011-12-09 14:16:32,201 Consumer (17): instance received shutdown call... 2011-12-09 14:16:32,207 Consumer (18): instance received shutdown call... 2011-12-09 14:16:32,208 Consumer (19): instance received shutdown call... 2011-12-09 14:16:32,209 Consumer (20): instance received shutdown call... 2011-12-09 14:16:32,210 Consumer (21): instance received shutdown call... 2011-12-09 14:16:32,211 Consumer (22): instance received shutdown call... 2011-12-09 14:16:32,212 Consumer (23): instance received shutdown call... 2011-12-09 14:16:32,213 Consumer (24): instance received shutdown call... 2011-12-09 14:16:32,213 Consumer (25): instance received shutdown call... 2011-12-09 14:16:32,214 Consumer (26): instance received shutdown call... 2011-12-09 14:16:32,215 Consumer (27): instance received shutdown call... 2011-12-09 14:16:32,216 Consumer (28): instance received shutdown call... 2011-12-09 14:16:32,217 Consumer (29): instance received shutdown call... 2011-12-09 14:16:32,226 Last call to consumer. 2011-12-09 14:16:32,231 Consumer instance (8) exited... 2011-12-09 14:16:32,274 Last call to consumer. 2011-12-09 14:16:32,276 Consumer instance (9) exited... 2011-12-09 14:16:32,337 Last call to consumer. 2011-12-09 14:16:32,338 Consumer instance (10) exited... 2011-12-09 14:16:32,367 Last call to consumer. 2011-12-09 14:16:32,371 Consumer instance (11) exited... 2011-12-09 14:16:32,399 Last call to consumer. 2011-12-09 14:16:32,404 Consumer instance (12) exited... 2011-12-09 14:16:32,440 Last call to consumer. 2011-12-09 14:16:32,441 Consumer instance (13) exited... 2011-12-09 14:16:32,492 Last call to consumer. 2011-12-09 14:16:32,504 Consumer instance (14) exited... 2011-12-09 14:16:32,539 Last call to consumer. 2011-12-09 14:16:32,540 Consumer instance (15) exited... 2011-12-09 14:16:32,598 Last call to consumer. 2011-12-09 14:16:32,599 Consumer instance (16) exited... 2011-12-09 14:16:32,633 Last call to consumer. 2011-12-09 14:16:32,634 Consumer instance (17) exited... 2011-12-09 14:16:32,679 Last call to consumer. 2011-12-09 14:16:32,687 Consumer instance (18) exited... 2011-12-09 14:16:32,726 Last call to consumer. 2011-12-09 14:16:32,727 Consumer instance (19) exited... 2011-12-09 14:16:32,789 Last call to consumer. 2011-12-09 14:16:32,790 Consumer instance (20) exited... 2011-12-09 14:16:32,820 Last call to consumer. 2011-12-09 14:16:32,833 Consumer instance (21) exited... 2011-12-09 14:16:32,867 Last call to consumer. 2011-12-09 14:16:32,873 Consumer instance (22) exited... 2011-12-09 14:16:32,913 Last call to consumer. 2011-12-09 14:16:32,918 Consumer instance (23) exited... 2011-12-09 14:16:32,936 Last call to consumer. 2011-12-09 14:16:32,940 Consumer instance (0) exited... 2011-12-09 14:16:32,936 Last call to consumer. 2011-12-09 14:16:32,960 Consumer instance (1) exited... 2011-12-09 14:16:32,945 Last call to consumer. 2011-12-09 14:16:32,960 Last call to consumer. 2011-12-09 14:16:32,994 Consumer instance (3) exited... 2011-12-09 14:16:32,991 Last call to consumer. 2011-12-09 14:16:33,015 Consumer instance (25) exited... 2011-12-09 14:16:32,946 Last call to consumer. 2011-12-09 14:16:33,063 Consumer instance (2) exited... 2011-12-09 14:16:33,023 Last call to consumer. 2011-12-09 14:16:33,080 Consumer instance (4) exited... 2011-12-09 14:16:32,989 Consumer instance (24) exited... 2011-12-09 14:16:33,038 Last call to consumer. 2011-12-09 14:16:33,191 Consumer instance (26) exited... 2011-12-09 14:16:33,085 Last call to consumer. 2011-12-09 14:16:33,242 Consumer instance (5) exited... 2011-12-09 14:16:33,147 Last call to consumer. 2011-12-09 14:16:33,265 Consumer instance (28) exited... 2011-12-09 14:16:33,194 Last call to consumer. 2011-12-09 14:16:33,309 Consumer instance (29) exited... 2011-12-09 14:16:33,101 Last call to consumer. 2011-12-09 14:16:33,336 Consumer instance (6) exited... 2011-12-09 14:16:33,085 Last call to consumer. 2011-12-09 14:16:33,380 Consumer instance (27) exited... 2011-12-09 14:16:33,147 Last call to consumer. 2011-12-09 14:16:33,455 Consumer instance (7) exited... However, when the same is run from Windows (Python 2.7.2), I get a very long and strange exception trace which revolves around pickle.py: Traceback (most recent call last): File "pc-example.py", line 39, in <module> pc.start() File "C:\cygwin\home\jango\workspace\pc\pc\pc.py", line 103, in start p_proc.start() File "C:\Python27\lib\multiprocessing\process.py", line 130, in start self._popen = Popen(self) File "C:\Python27\lib\multiprocessing\forking.py", line 271, in __init__ dump(process_obj, to_child, HIGHEST_PROTOCOL) File "C:\Python27\lib\multiprocessing\forking.py", line 193, in dump ForkingPickler(file, protocol).dump(obj) File "C:\Python27\lib\pickle.py", line 224, in dump self.save(obj) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 600, in save_list self._batch_appends(iter(obj)) File "C:\Python27\lib\pickle.py", line 636, in _batch_appends save(tmp[0]) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 419, in save_reduce save(state) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 649, in save_dict self._batch_setitems(obj.iteritems()) File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems save(v) File "C:\Python27\lib\pickle.py", line 331, in save self.save_reduce(obj=obj, *rv) File "C:\Python27\lib\pickle.py", line 396, in save_reduce save(cls) File "C:\Python27\lib\pickle.py", line 286, in save f(self, obj) # Call unbound method with explicit self File "C:\Python27\lib\pickle.py", line 748, in save_global (obj, module, name)) pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Python27\lib\multiprocessing\forking.py", line 374, in main self = load(from_parent) File "C:\Python27\lib\pickle.py", line 1378, in load return Unpickler(file).load() File "C:\Python27\lib\pickle.py", line 858, in load dispatch[key](self) File "C:\Python27\lib\pickle.py", line 880, in load_eof raise EOFError EOFError |
|||
| msg149119 - (view) | Author: Antoine Pitrou (pitrou) *  |
Date: 2011-12-09 20:00 | |
Yes, Windows needs to pickle objects which are sent to (or returned from) a child process. Now you should wonder why you are sending a threading lock to the child. Are you sure this is deliberate? |
|||
| msg149121 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-09 20:28 | |
I am not sending locks explicetly (i.e. I am not using locks), but I do pass a Queue object from PC instance to _Consumer and _Producer instances that get/put values from/to the queue -- this is done deliberately. 2011/12/9 Antoine Pitrou <report@bugs.python.org> > > Antoine Pitrou <pitrou@free.fr> added the comment: > > Yes, Windows needs to pickle objects which are sent to (or returned from) > a child process. Now you should wonder why you are sending a threading lock > to the child. Are you sure this is deliberate? > > ---------- > nosy: +pitrou > versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6 > > _______________________________________ > Python tracker <report@bugs.python.org> > <http://bugs.python.org/issue13569> > _______________________________________ > |
|||
| msg149122 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2011-12-09 20:34 | |
> I am not sending locks explicetly (i.e. I am not using locks), but I do > pass a Queue object from PC instance to _Consumer and _Producer instances > that get/put values from/to the queue -- this is done deliberately. Is it a Queue.Queue or a multiprocessing.Queue? |
|||
| msg149123 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-09 20:48 | |
It's multiprocessing Queue: from multiprocessing import Process, Queue, Event |
|||
| msg149127 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2011-12-09 21:30 | |
I don't think it's the queue. Try removing the logger instead (or creating it in the child). |
|||
| msg149136 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-09 23:20 | |
Hi Antoine, * If I don't pass a logger and do print statements instead, works like a charm. * If I getLogger() in the child instead, example fails with the same trace. However, according to this ( http://docs.python.org/library/logging.html ): "The logging module is intended to be thread-safe without any special work needing to be done by its clients. It achieves this though using threading locks; there is one lock to serialize access to the module’s shared data, and each handler also creates a lock to serialize access to its underlying I/O." Which is why I assumed I could use logging safely. Thanks, Nikita On 11-12-09 04:30 PM, Antoine Pitrou wrote: > Antoine Pitrou<pitrou@free.fr> added the comment: > > I don't think it's the queue. Try removing the logger instead (or creating it in the child). > > ---------- > > _______________________________________ > Python tracker<report@bugs.python.org> > <http://bugs.python.org/issue13569> > _______________________________________ |
|||
| msg149137 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2011-12-10 00:23 | |
> If I getLogger() in the child instead, example fails with the same trace. That sounds rather unlikely. Are you sure you don't store the logger in your process' __init__ method? The __init__ method is called in the parent and the process instance is transfered to the child when you call start() on the process. |
|||
| msg149138 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-10 00:55 | |
>>Are you sure you don't store the logger in your process' __init__ method? The __init__ method is called in the parent and the process instance is transfered to the child when you call start() on the process. To make sure we are on the same page, here is what I changed it to: https://github.com/jango/PC/blob/test/pc/pc.py -> lines 47-50 commented out (init of PC class that initialises threads); -> lines 64, 65, 77, 78 getLogger within the PC class and print some messages; -> lines 59, 61 71, 74 -- None is passed to the constructor of _Consumer and _Producer; -> lines 135, 203 -- children user logger; With this code, I still get the exception. On 11-12-09 07:23 PM, Antoine Pitrou wrote: > Antoine Pitrou<pitrou@free.fr> added the comment: > >> If I getLogger() in the child instead, example fails with the same trace. > That sounds rather unlikely. Are you sure you don't store the logger in your process' __init__ method? The __init__ method is called in the parent and the process instance is transfered to the child when you call start() on the process. > > ---------- > > _______________________________________ > Python tracker<report@bugs.python.org> > <http://bugs.python.org/issue13569> > _______________________________________ |
|||
| msg149144 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2011-12-10 09:43 | |
> To make sure we are on the same page, here is what I changed it to: > https://github.com/jango/PC/blob/test/pc/pc.py > > -> lines 47-50 commented out (init of PC class that initialises threads); > -> lines 64, 65, 77, 78 getLogger within the PC class and print some > messages; > -> lines 59, 61 71, 74 -- None is passed to the constructor of _Consumer > and _Producer; > -> lines 135, 203 -- children user logger; Ok, can you try to move the getLogger call from _Producer.__init__ to _Producer.run? (same for _Consumer) (note these aren't threads, but processes) |
|||
| msg149154 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-10 13:44 | |
>> (note these aren't threads, but processes) Fair enough, I should be more careful with terminology. > Ok, can you try to move the getLogger call from _Producer.__init__ to,> _Producer.run?,> (same for _Consumer) Yes, that seems to work fine. One thing, however, that this would mean that I would have to declare logger in any method outside of __init__, like in shutdown() for example. |
|||
| msg149175 - (view) | Author: Antoine Pitrou (pitrou) * |
Date: 2011-12-10 17:46 | |
> One thing, however, that this would mean > that I would have to declare logger in any method outside of __init__, > like in shutdown() for example. If you only use the logger from the child process, you can declare and store it in the run() method. That said, this still can be a feature request: allow loggers to be pickled (and unpickled). |
|||
| msg149176 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-10 17:54 | |
Fair enough. Thanks for help with investigating this issue! |
|||
| msg149200 - (view) | Author: Vinay Sajip (vinay.sajip) * |
Date: 2011-12-10 23:52 | |
You shouldn't need to pickle loggers, because loggers just represent where in your application something happens (that being determined by the logger name). The logger name is just a string, and if you need to communicate that across e.g. a process boundary, just sending the name should be enough. Ideally in a multiprocessing application, logging initialisation should happen after the fork() or CreateProcess() which creates the child process. It's best if loggers are named after the module which logs events (using __name__, which means you don't need to pass the name around - but even if you need to for some reason, you still don't need to pickle a logger or pass it around as a parameter). You *can* use logging safely in terms of the locks it creates to serialise access to I/O resources and to protect shared data structures, but you still need to understand what's happening and allow for it in your own coding. To summarise, I don't quite see from the information in this issue why it is desirable to pickle loggers, and the GitHub link doesn't work, so I can't look at Nikita's example, either. So if a better justification isn't available, I would like to close the issue. |
|||
| msg149202 - (view) | Author: Nikita Pchelin (jango) | Date: 2011-12-11 00:54 | |
Here is a link for example in question: https://github.com/jango/calculon/blob/master/calculon/calculon.py Previously, I would pass logger instance from Calculon class to _Producer and _Consumer classes. In the current revision, I removed the logging completely since I don't feel like it was serving much purpose anyway. On 11-12-10 06:52 PM, Vinay Sajip wrote: > Vinay Sajip<vinay_sajip@yahoo.co.uk> added the comment: > > You shouldn't need to pickle loggers, because loggers just represent where in your application something happens (that being determined by the logger name). The logger name is just a string, and if you need to communicate that across e.g. a process boundary, just sending the name should be enough. > > Ideally in a multiprocessing application, logging initialisation should happen after the fork() or CreateProcess() which creates the child process. It's best if loggers are named after the module which logs events (using __name__, which means you don't need to pass the name around - but even if you need to for some reason, you still don't need to pickle a logger or pass it around as a parameter). > > You *can* use logging safely in terms of the locks it creates to serialise access to I/O resources and to protect shared data structures, but you still need to understand what's happening and allow for it in your own coding. > > To summarise, I don't quite see from the information in this issue why it is desirable to pickle loggers, and the GitHub link doesn't work, so I can't look at Nikita's example, either. So if a better justification isn't available, I would like to close the issue. > > ---------- > > _______________________________________ > Python tracker<report@bugs.python.org> > <http://bugs.python.org/issue13569> > _______________________________________ |
|||
| msg149230 - (view) | Author: Vinay Sajip (vinay.sajip) * |
Date: 2011-12-11 16:48 | |
So, if I understand your last comment correctly, it's OK to close this issue, so I'm going to. If I misunderstood, then please feel free to reopen. |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022-04-11 14:57:24 | admin | set | github: 57778 |
| 2011-12-11 16:48:01 | vinay.sajip | set | status: open -> closed resolution: not a bug messages: + msg149230 |
| 2011-12-11 00:54:49 | jango | set | messages: + msg149202 |
| 2011-12-10 23:52:24 | vinay.sajip | set | messages: + msg149200 |
| 2011-12-10 17:54:45 | jango | set | messages: + msg149176 |
| 2011-12-10 17:46:43 | pitrou | set | nosy:
+ vinay.sajip title: multiprocessing module: Process.start() fails with EOFError: pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock -> Loggers cannot be pickled messages: + msg149175 versions: - Python 2.7, Python 3.2 type: behavior -> enhancement |
| 2011-12-10 13:44:23 | jango | set | messages: + msg149154 |
| 2011-12-10 09:43:20 | pitrou | set | messages: + msg149144 |
| 2011-12-10 00:55:33 | jango | set | messages: + msg149138 |
| 2011-12-10 00:23:40 | pitrou | set | messages: + msg149137 |
| 2011-12-09 23:20:19 | jango | set | messages: + msg149136 |
| 2011-12-09 21:30:24 | pitrou | set | messages: + msg149127 |
| 2011-12-09 20:48:02 | jango | set | messages: + msg149123 |
| 2011-12-09 20:34:44 | pitrou | set | messages: + msg149122 |
| 2011-12-09 20:28:27 | jango | set | messages: + msg149121 |
| 2011-12-09 20:00:46 | pitrou | set | nosy:
+ pitrou messages: + msg149119 versions: + Python 2.7, Python 3.2, Python 3.3, - Python 2.6 |
| 2011-12-09 19:57:41 | jango | create | |