classification
Title: Loggers cannot be pickled
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: jango, pitrou, vinay.sajip
Priority: normal Keywords:

Created on 2011-12-09 19:57 by jango, last changed 2011-12-11 16:48 by vinay.sajip. This issue is now closed.

Messages (16)
msg149118 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-09 19:57
I've wrote a little application that uses multiprocessing module:
https://github.com/jango/PC/blob/master/pc/pc-example.py

When I run it in my Linux setup, I get the expected output (Python 2.7.1+):
2011-12-09 14:16:29,014 Started Producer 0.
2011-12-09 14:16:29,076 Started Producer 1.
2011-12-09 14:16:29,076 Producer process (0) exited...
2011-12-09 14:16:29,131 Started Producer 2.
2011-12-09 14:16:29,188 Started Producer 3.
2011-12-09 14:16:29,218 Started Producer 4.
2011-12-09 14:16:29,281 Started Producer 5.
2011-12-09 14:16:29,327 Started Producer 6.
2011-12-09 14:16:29,377 Started Producer 7.
2011-12-09 14:16:29,403 Producer process (1) exited...
2011-12-09 14:16:29,412 Started Producer 8.
2011-12-09 14:16:29,504 Started Producer 9.
2011-12-09 14:16:29,528 Producer process (4) exited...
2011-12-09 14:16:29,570 Started Producer 10.
2011-12-09 14:16:29,615 Started Producer 11.
2011-12-09 14:16:29,622 Producer process (3) exited...
2011-12-09 14:16:29,637 Producer process (7) exited...
2011-12-09 14:16:29,653 Producer process (5) exited...
2011-12-09 14:16:29,692 Started Producer 12.
2011-12-09 14:16:29,731 Producer process (12) exited...
2011-12-09 14:16:29,747 Started Producer 13.
2011-12-09 14:16:29,809 Started Producer 14.
2011-12-09 14:16:29,860 Started Producer 15.
2011-12-09 14:16:29,903 Producer process (2) exited...
2011-12-09 14:16:29,905 Started Producer 16.
2011-12-09 14:16:29,918 Producer process (8) exited...
2011-12-09 14:16:29,970 Started Producer 17.
2011-12-09 14:16:30,001 Started Producer 18.
2011-12-09 14:16:30,070 Started Producer 19.
2011-12-09 14:16:30,090 Producer process (11) exited...
2011-12-09 14:16:30,105 Producer process (10) exited...
2011-12-09 14:16:30,137 Started Producer 20.
2011-12-09 14:16:30,152 Producer process (6) exited...
2011-12-09 14:16:30,183 Producer process (9) exited...
2011-12-09 14:16:30,183 Producer process (17) exited...
2011-12-09 14:16:30,251 Started Producer 21.
2011-12-09 14:16:30,288 Started Producer 22.
2011-12-09 14:16:30,308 Producer process (18) exited...
2011-12-09 14:16:30,308 Producer process (20) exited...
2011-12-09 14:16:30,355 Producer process (16) exited...
2011-12-09 14:16:30,380 Started Producer 23.
2011-12-09 14:16:30,436 Started Producer 24.
2011-12-09 14:16:30,472 Started Producer 25.
2011-12-09 14:16:30,480 Producer process (22) exited...
2011-12-09 14:16:30,509 Started Producer 26.
2011-12-09 14:16:30,554 Started Producer 27.
2011-12-09 14:16:30,609 Started Producer 28.
2011-12-09 14:16:30,620 Producer process (14) exited...
2011-12-09 14:16:30,636 Producer process (15) exited...
2011-12-09 14:16:30,660 Started Producer 29.
2011-12-09 14:16:30,667 Producer process (19) exited...
2011-12-09 14:16:30,667 Producer process (13) exited...
2011-12-09 14:16:30,667 Producer process (27) exited...
2011-12-09 14:16:30,750 Started Consumer 0.
2011-12-09 14:16:30,761 --> 0 produces 0.
2011-12-09 14:16:30,776 Producer process (28) exited...
2011-12-09 14:16:30,770 --> 1 produces 10.
2011-12-09 14:16:30,783 --> 4 produces 40.
2011-12-09 14:16:30,797 Started Consumer 1.
2011-12-09 14:16:30,807 --> 3 produces 30.
2011-12-09 14:16:30,816 --> 5 produces 50.
2011-12-09 14:16:30,817 --> 12 produces 120.
2011-12-09 14:16:30,819 --> 2 produces 20.
2011-12-09 14:16:30,826 --> 8 produces 80.
2011-12-09 14:16:30,835 --> 11 produces 110.
2011-12-09 14:16:30,848 --> 10 produces 100.
2011-12-09 14:16:30,849 --> 6 produces 60.
2011-12-09 14:16:30,852 --> 17 produces 170.
2011-12-09 14:16:30,853 --> 9 produces 90.
2011-12-09 14:16:30,859 --> 18 produces 180.
2011-12-09 14:16:30,860 --> 20 produces 200.
2011-12-09 14:16:30,865 --> 16 produces 160.
2011-12-09 14:16:30,866 --> 22 produces 220.
2011-12-09 14:16:30,867 --> 14 produces 140.
2011-12-09 14:16:30,868 --> 15 produces 150.
2011-12-09 14:16:30,869 --> 19 produces 190.
2011-12-09 14:16:30,874 --> 13 produces 130.
2011-12-09 14:16:30,823 Producer process (21) exited...
2011-12-09 14:16:30,812 --> 7 produces 70.
2011-12-09 14:16:30,893 --> 28 produces 280.
2011-12-09 14:16:30,854 Producer process (25) exited...
2011-12-09 14:16:30,894 Started Consumer 2.
2011-12-09 14:16:30,875 --> 27 produces 270.
2011-12-09 14:16:30,905 --> 25 produces 250.
2011-12-09 14:16:30,917 --> 21 produces 210.
2011-12-09 14:16:30,943 Started Consumer 3.
2011-12-09 14:16:31,002 Started Consumer 4.
2011-12-09 14:16:31,049 Started Consumer 5.
2011-12-09 14:16:31,091 Started Consumer 6.
2011-12-09 14:16:31,137 Started Consumer 7.
2011-12-09 14:16:31,197 Producer process (23) exited...
2011-12-09 14:16:31,213 --> 23 produces 230.
2011-12-09 14:16:31,206 Started Consumer 8.
2011-12-09 14:16:31,263 Started Consumer 9.
2011-12-09 14:16:31,275 Producer process (24) exited...
2011-12-09 14:16:31,307 Started Consumer 10.
2011-12-09 14:16:31,322 --> 24 produces 240.
2011-12-09 14:16:31,357 Started Consumer 11.
2011-12-09 14:16:31,386 Started Consumer 12.
2011-12-09 14:16:31,400 Producer process (26) exited...
2011-12-09 14:16:31,432 Started Consumer 13.
2011-12-09 14:16:31,431 --> 26 produces 260.
2011-12-09 14:16:31,476 Started Consumer 14.
2011-12-09 14:16:31,527 Started Consumer 15.
2011-12-09 14:16:31,556 Producer process (29) exited...
2011-12-09 14:16:31,570 Started Consumer 16.
2011-12-09 14:16:31,587 --> 29 produces 290.
2011-12-09 14:16:31,630 Started Consumer 17.
2011-12-09 14:16:31,664 Started Consumer 18.
2011-12-09 14:16:31,714 Started Consumer 19.
2011-12-09 14:16:31,768 Started Consumer 20.
2011-12-09 14:16:31,815 Started Consumer 21.
2011-12-09 14:16:31,847 Started Consumer 22.
2011-12-09 14:16:31,903 Started Consumer 23.
2011-12-09 14:16:31,932 Started Consumer 24.
2011-12-09 14:16:31,981 Started Consumer 25.
2011-12-09 14:16:32,023 Started Consumer 26.
2011-12-09 14:16:32,073 Started Consumer 27.
2011-12-09 14:16:32,121 Started Consumer 28.
2011-12-09 14:16:32,175 Started Consumer 29.
2011-12-09 14:16:32,181 Consumer (0): instance received shutdown call...
2011-12-09 14:16:32,182 Consumer (1): instance received shutdown call...
2011-12-09 14:16:32,185 Consumer (2): instance received shutdown call...
2011-12-09 14:16:32,186 Consumer (3): instance received shutdown call...
2011-12-09 14:16:32,187 Consumer (4): instance received shutdown call...
2011-12-09 14:16:32,188 Consumer (5): instance received shutdown call...
2011-12-09 14:16:32,189 Consumer (6): instance received shutdown call...
2011-12-09 14:16:32,190 Consumer (7): instance received shutdown call...
2011-12-09 14:16:32,191 Consumer (8): instance received shutdown call...
2011-12-09 14:16:32,192 Consumer (9): instance received shutdown call...
2011-12-09 14:16:32,193 Consumer (10): instance received shutdown call...
2011-12-09 14:16:32,194 Consumer (11): instance received shutdown call...
2011-12-09 14:16:32,195 Consumer (12): instance received shutdown call...
2011-12-09 14:16:32,196 Consumer (13): instance received shutdown call...
2011-12-09 14:16:32,197 Consumer (14): instance received shutdown call...
2011-12-09 14:16:32,198 Consumer (15): instance received shutdown call...
2011-12-09 14:16:32,199 Consumer (16): instance received shutdown call...
2011-12-09 14:16:32,201 Consumer (17): instance received shutdown call...
2011-12-09 14:16:32,207 Consumer (18): instance received shutdown call...
2011-12-09 14:16:32,208 Consumer (19): instance received shutdown call...
2011-12-09 14:16:32,209 Consumer (20): instance received shutdown call...
2011-12-09 14:16:32,210 Consumer (21): instance received shutdown call...
2011-12-09 14:16:32,211 Consumer (22): instance received shutdown call...
2011-12-09 14:16:32,212 Consumer (23): instance received shutdown call...
2011-12-09 14:16:32,213 Consumer (24): instance received shutdown call...
2011-12-09 14:16:32,213 Consumer (25): instance received shutdown call...
2011-12-09 14:16:32,214 Consumer (26): instance received shutdown call...
2011-12-09 14:16:32,215 Consumer (27): instance received shutdown call...
2011-12-09 14:16:32,216 Consumer (28): instance received shutdown call...
2011-12-09 14:16:32,217 Consumer (29): instance received shutdown call...
2011-12-09 14:16:32,226 Last call to consumer.
2011-12-09 14:16:32,231 Consumer instance (8) exited...
2011-12-09 14:16:32,274 Last call to consumer.
2011-12-09 14:16:32,276 Consumer instance (9) exited...
2011-12-09 14:16:32,337 Last call to consumer.
2011-12-09 14:16:32,338 Consumer instance (10) exited...
2011-12-09 14:16:32,367 Last call to consumer.
2011-12-09 14:16:32,371 Consumer instance (11) exited...
2011-12-09 14:16:32,399 Last call to consumer.
2011-12-09 14:16:32,404 Consumer instance (12) exited...
2011-12-09 14:16:32,440 Last call to consumer.
2011-12-09 14:16:32,441 Consumer instance (13) exited...
2011-12-09 14:16:32,492 Last call to consumer.
2011-12-09 14:16:32,504 Consumer instance (14) exited...
2011-12-09 14:16:32,539 Last call to consumer.
2011-12-09 14:16:32,540 Consumer instance (15) exited...
2011-12-09 14:16:32,598 Last call to consumer.
2011-12-09 14:16:32,599 Consumer instance (16) exited...
2011-12-09 14:16:32,633 Last call to consumer.
2011-12-09 14:16:32,634 Consumer instance (17) exited...
2011-12-09 14:16:32,679 Last call to consumer.
2011-12-09 14:16:32,687 Consumer instance (18) exited...
2011-12-09 14:16:32,726 Last call to consumer.
2011-12-09 14:16:32,727 Consumer instance (19) exited...
2011-12-09 14:16:32,789 Last call to consumer.
2011-12-09 14:16:32,790 Consumer instance (20) exited...
2011-12-09 14:16:32,820 Last call to consumer.
2011-12-09 14:16:32,833 Consumer instance (21) exited...
2011-12-09 14:16:32,867 Last call to consumer.
2011-12-09 14:16:32,873 Consumer instance (22) exited...
2011-12-09 14:16:32,913 Last call to consumer.
2011-12-09 14:16:32,918 Consumer instance (23) exited...
2011-12-09 14:16:32,936 Last call to consumer.
2011-12-09 14:16:32,940 Consumer instance (0) exited...
2011-12-09 14:16:32,936 Last call to consumer.
2011-12-09 14:16:32,960 Consumer instance (1) exited...
2011-12-09 14:16:32,945 Last call to consumer.
2011-12-09 14:16:32,960 Last call to consumer.
2011-12-09 14:16:32,994 Consumer instance (3) exited...
2011-12-09 14:16:32,991 Last call to consumer.
2011-12-09 14:16:33,015 Consumer instance (25) exited...
2011-12-09 14:16:32,946 Last call to consumer.
2011-12-09 14:16:33,063 Consumer instance (2) exited...
2011-12-09 14:16:33,023 Last call to consumer.
2011-12-09 14:16:33,080 Consumer instance (4) exited...
2011-12-09 14:16:32,989 Consumer instance (24) exited...
2011-12-09 14:16:33,038 Last call to consumer.
2011-12-09 14:16:33,191 Consumer instance (26) exited...
2011-12-09 14:16:33,085 Last call to consumer.
2011-12-09 14:16:33,242 Consumer instance (5) exited...
2011-12-09 14:16:33,147 Last call to consumer.
2011-12-09 14:16:33,265 Consumer instance (28) exited...
2011-12-09 14:16:33,194 Last call to consumer.
2011-12-09 14:16:33,309 Consumer instance (29) exited...
2011-12-09 14:16:33,101 Last call to consumer.
2011-12-09 14:16:33,336 Consumer instance (6) exited...
2011-12-09 14:16:33,085 Last call to consumer.
2011-12-09 14:16:33,380 Consumer instance (27) exited...
2011-12-09 14:16:33,147 Last call to consumer.
2011-12-09 14:16:33,455 Consumer instance (7) exited...

However, when the same is run from Windows (Python 2.7.2), I get a very long and strange exception trace which revolves around pickle.py:
Traceback (most recent call last):
  File "pc-example.py", line 39, in <module>
    pc.start() 
  File "C:\cygwin\home\jango\workspace\pc\pc\pc.py", line 103, in start
    p_proc.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 271, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 193, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "C:\Python27\lib\pickle.py", line 636, in _batch_appends
    save(tmp[0])
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 374, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError
msg149119 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-09 20:00
Yes, Windows needs to pickle objects which are sent to (or returned from) a child process. Now you should wonder why you are sending a threading lock to the child. Are you sure this is deliberate?
msg149121 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-09 20:28
I am not sending locks explicetly (i.e. I am not using locks), but I do
pass a Queue object from PC instance to _Consumer and _Producer instances
that get/put values from/to the queue -- this is done deliberately.

2011/12/9 Antoine Pitrou <report@bugs.python.org>

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Yes, Windows needs to pickle objects which are sent to (or returned from)
> a child process. Now you should wonder why you are sending a threading lock
> to the child. Are you sure this is deliberate?
>
> ----------
> nosy: +pitrou
> versions: +Python 2.7, Python 3.2, Python 3.3 -Python 2.6
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue13569>
> _______________________________________
>
msg149122 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-09 20:34
> I am not sending locks explicetly (i.e. I am not using locks), but I do
> pass a Queue object from PC instance to _Consumer and _Producer instances
> that get/put values from/to the queue -- this is done deliberately.

Is it a Queue.Queue or a multiprocessing.Queue?
msg149123 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-09 20:48
It's multiprocessing Queue:

from multiprocessing import Process, Queue, Event
msg149127 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-09 21:30
I don't think it's the queue. Try removing the logger instead (or creating it in the child).
msg149136 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-09 23:20
Hi Antoine,

* If I don't pass a logger and do print statements instead, works like a 
charm.

* If I getLogger() in the child instead, example fails with the same trace.

However, according to this ( http://docs.python.org/library/logging.html 
): "The logging module is intended to be thread-safe without any special 
work needing to be done by its clients. It achieves this though using 
threading locks; there is one lock to serialize access to the module’s 
shared data, and each handler also creates a lock to serialize access to 
its underlying I/O."

Which is why I assumed I could use logging safely.

Thanks,
Nikita

On 11-12-09 04:30 PM, Antoine Pitrou wrote:
> Antoine Pitrou<pitrou@free.fr>  added the comment:
>
> I don't think it's the queue. Try removing the logger instead (or creating it in the child).
>
> ----------
>
> _______________________________________
> Python tracker<report@bugs.python.org>
> <http://bugs.python.org/issue13569>
> _______________________________________
msg149137 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-10 00:23
> If I getLogger() in the child instead, example fails with the same trace.

That sounds rather unlikely. Are you sure you don't store the logger in your process' __init__ method? The __init__ method is called in the parent and the process instance is transfered to the child when you call start() on the process.
msg149138 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-10 00:55
>>Are you sure you don't store the logger in your process' __init__ 
method? The __init__ method is called in the parent and the process 
instance is transfered to the child when you call start() on the process.

To make sure we are on the same page, here is what I changed it to:
https://github.com/jango/PC/blob/test/pc/pc.py

-> lines 47-50 commented out (init of PC class that initialises threads);
-> lines 64, 65, 77, 78 getLogger within the PC class and print some 
messages;
-> lines 59, 61 71, 74 -- None is passed to the constructor of _Consumer 
and _Producer;
-> lines 135, 203 -- children user logger;

With this code, I still get the exception.

On 11-12-09 07:23 PM, Antoine Pitrou wrote:
> Antoine Pitrou<pitrou@free.fr>  added the comment:
>
>> If I getLogger() in the child instead, example fails with the same trace.
> That sounds rather unlikely. Are you sure you don't store the logger in your process' __init__ method? The __init__ method is called in the parent and the process instance is transfered to the child when you call start() on the process.
>
> ----------
>
> _______________________________________
> Python tracker<report@bugs.python.org>
> <http://bugs.python.org/issue13569>
> _______________________________________
msg149144 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-10 09:43
> To make sure we are on the same page, here is what I changed it to:
> https://github.com/jango/PC/blob/test/pc/pc.py
> 
> -> lines 47-50 commented out (init of PC class that initialises threads);
> -> lines 64, 65, 77, 78 getLogger within the PC class and print some 
> messages;
> -> lines 59, 61 71, 74 -- None is passed to the constructor of _Consumer 
> and _Producer;
> -> lines 135, 203 -- children user logger;

Ok, can you try to move the getLogger call from _Producer.__init__ to
_Producer.run?
(same for _Consumer)

(note these aren't threads, but processes)
msg149154 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-10 13:44
>> (note these aren't threads, but processes)
Fair enough, I should be more careful with terminology.

 > Ok, can you try to move the getLogger call from _Producer.__init__ 
to,> _Producer.run?,> (same for _Consumer)

Yes, that seems to work fine. One thing, however, that this would mean 
that I would have to declare logger in any method outside of __init__, 
like in shutdown()  for example.
msg149175 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-10 17:46
> One thing, however, that this would mean 
> that I would have to declare logger in any method outside of __init__, 
> like in shutdown()  for example.

If you only use the logger from the child process, you can declare and store it in the run() method.

That said, this still can be a feature request: allow loggers to be pickled (and unpickled).
msg149176 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-10 17:54
Fair enough. Thanks for help with investigating this issue!
msg149200 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-10 23:52
You shouldn't need to pickle loggers, because loggers just represent where in your application something happens (that being determined by the logger name). The logger name is just a string, and if you need to communicate that across e.g. a process boundary, just sending the name should be enough.

Ideally in a multiprocessing application, logging initialisation should happen after the fork() or CreateProcess() which creates the child process. It's best if loggers are named after the module which logs events (using __name__, which means you don't need to pass the name around - but even if you need to for some reason, you still don't need to pickle a logger or pass it around as a parameter).

You *can* use logging safely in terms of the locks it creates to serialise access to I/O resources and to protect shared data structures, but you still need to understand what's happening and allow for it in your own coding.

To summarise, I don't quite see from the information in this issue why it is desirable to pickle loggers, and the GitHub link doesn't work, so I can't look at Nikita's example, either. So if a better justification isn't available, I would like to close the issue.
msg149202 - (view) Author: Nikita Pchelin (jango) Date: 2011-12-11 00:54
Here is a link for example in question:
https://github.com/jango/calculon/blob/master/calculon/calculon.py

Previously, I would pass logger instance from Calculon class to 
_Producer and _Consumer classes. In the current revision, I removed the 
logging completely since I don't feel like it was serving much purpose 
anyway.

On 11-12-10 06:52 PM, Vinay Sajip wrote:
> Vinay Sajip<vinay_sajip@yahoo.co.uk>  added the comment:
>
> You shouldn't need to pickle loggers, because loggers just represent where in your application something happens (that being determined by the logger name). The logger name is just a string, and if you need to communicate that across e.g. a process boundary, just sending the name should be enough.
>
> Ideally in a multiprocessing application, logging initialisation should happen after the fork() or CreateProcess() which creates the child process. It's best if loggers are named after the module which logs events (using __name__, which means you don't need to pass the name around - but even if you need to for some reason, you still don't need to pickle a logger or pass it around as a parameter).
>
> You *can* use logging safely in terms of the locks it creates to serialise access to I/O resources and to protect shared data structures, but you still need to understand what's happening and allow for it in your own coding.
>
> To summarise, I don't quite see from the information in this issue why it is desirable to pickle loggers, and the GitHub link doesn't work, so I can't look at Nikita's example, either. So if a better justification isn't available, I would like to close the issue.
>
> ----------
>
> _______________________________________
> Python tracker<report@bugs.python.org>
> <http://bugs.python.org/issue13569>
> _______________________________________
msg149230 - (view) Author: Vinay Sajip (vinay.sajip) * (Python committer) Date: 2011-12-11 16:48
So, if I understand your last comment correctly, it's OK to close this issue, so I'm going to. If I misunderstood, then please feel free to reopen.
History
Date User Action Args
2011-12-11 16:48:01vinay.sajipsetstatus: open -> closed
resolution: not a bug
messages: + msg149230
2011-12-11 00:54:49jangosetmessages: + msg149202
2011-12-10 23:52:24vinay.sajipsetmessages: + msg149200
2011-12-10 17:54:45jangosetmessages: + msg149176
2011-12-10 17:46:43pitrousetnosy: + vinay.sajip
title: multiprocessing module: Process.start() fails with EOFError: pickle.PicklingError: Can't pickle <type 'thread.lock'>: it's not found as thread.lock -> Loggers cannot be pickled
messages: + msg149175

versions: - Python 2.7, Python 3.2
type: behavior -> enhancement
2011-12-10 13:44:23jangosetmessages: + msg149154
2011-12-10 09:43:20pitrousetmessages: + msg149144
2011-12-10 00:55:33jangosetmessages: + msg149138
2011-12-10 00:23:40pitrousetmessages: + msg149137
2011-12-09 23:20:19jangosetmessages: + msg149136
2011-12-09 21:30:24pitrousetmessages: + msg149127
2011-12-09 20:48:02jangosetmessages: + msg149123
2011-12-09 20:34:44pitrousetmessages: + msg149122
2011-12-09 20:28:27jangosetmessages: + msg149121
2011-12-09 20:00:46pitrousetnosy: + pitrou

messages: + msg149119
versions: + Python 2.7, Python 3.2, Python 3.3, - Python 2.6
2011-12-09 19:57:41jangocreate