➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_multiprocessing: pickling failures
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, jnoller, loewis, michael.foord, skrah, tim.golden
Priority: release blocker Keywords: buildbot, patch

Created on 2010-04-07 13:34 by skrah, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
issue8333.patch skrah, 2010-04-07 18:55
Messages (12)
msg102539 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-04-07 13:34 
On Windows/amd64, I get loads of pickling errors in test_multiprocessing.

Type 1 error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 347, in main
    self = load(from_parent)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError


Type 2 error:

======================================================================
ERROR: test_fork (__main__.WithManagerTestQueue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "..\..\Lib\test\test_multiprocessing.py", line 485, in test_fork
    p.start()
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\process.py", line 104, in start
    self._popen = Popen(self)
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 244, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 167, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'cStringIO.StringO'>: it's not found as cStringIO.StringO
msg102542 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2010-04-07 14:48 
Seeing the same thing on 32-bit WinXP on x86

On 07/04/2010 14:34, Stefan Krah wrote:
>
> New submission from Stefan Krah<stefan-usenet@bytereef.org>:
>
> On Windows/amd64, I get loads of pickling errors in test_multiprocessing.
>
> Type 1 error:
>
> Traceback (most recent call last):
>    File "<string>", line 1, in<module>
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 347, in main
>      self = load(from_parent)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 1378, in load
>      return Unpickler(file).load()
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 858, in load
>      dispatch[key](self)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 880, in load_eof
>      raise EOFError
> EOFError
>
>
> Type 2 error:
>
> ======================================================================
> ERROR: test_fork (__main__.WithManagerTestQueue)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "..\..\Lib\test\test_multiprocessing.py", line 485, in test_fork
>      p.start()
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\process.py", line 104, in start
>      self._popen = Popen(self)
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 244, in __init__
>      dump(process_obj, to_child, HIGHEST_PROTOCOL)
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 167, in dump
>      ForkingPickler(file, protocol).dump(obj)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 224, in dump
>      self.save(obj)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
>      save(state)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
>      self._batch_setitems(obj.iteritems())
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
>      save(v)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\multiprocessing\forking.py", line 40, in dispatcher
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 401, in save_reduce
>      save(args)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 548, in save_tuple
>      save(element)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
>      save(state)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
>      self._batch_setitems(obj.iteritems())
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
>      save(v)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 419, in save_reduce
>      save(state)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 649, in save_dict
>      self._batch_setitems(obj.iteritems())
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 681, in _batch_setitems
>      save(v)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 331, in save
>      self.save_reduce(obj=obj, *rv)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 396, in save_reduce
>      save(cls)
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 286, in save
>      f(self, obj) # Call unbound method with explicit self
>    File "C:\Users\stefan\svn\trunk\lib\pickle.py", line 748, in save_global
>      (obj, module, name))
> PicklingError: Can't pickle<type 'cStringIO.StringO'>: it's not found as cStringIO.StringO
>
> ----------
> components: Library (Lib)
> messages: 102539
> nosy: skrah
> priority: high
> severity: normal
> stage: needs patch
> status: open
> title: test_multiprocessing: pickling failures
> type: behavior
> versions: Python 2.7
>
> _______________________________________
> Python tracker<report@bugs.python.org>
> <http://bugs.python.org/issue8333>
> _______________________________________
> _______________________________________________
> Python-bugs-list mailing list
> Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/mail%40timgolden.me.uk
msg102555 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-04-07 18:05 
When did this problem start?
msg102556 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-04-07 18:54 
It was introduced in r79623, and the attached patch makes the problem
go away. I would be happy to apply it, but I perhaps you and Michael
could analyze if there are unwanted dependencies between multiprocessing
and unittest.
msg102561 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-04-07 19:17 
I expect the unittest tests will fail with that patch applied however... No problem with switching to just using StringIO in unittest though - preferably with a comment as to why.
msg102564 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-07 20:51 
Perhaps we could run the multiprocessing tests unbuffered?
msg102565 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-04-07 20:51 
BTW, this is also causing all Windows buildbots to fail.
msg102573 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-04-07 22:49 
TestResult creates its StringIO buffers even if they aren't use. I can create them on first use and use StringIO.StringIO rather than cStringIO. Will fix now.
msg102574 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-04-07 22:56 
Although merely switching to StringIO may not be enough. TestResult objects keep a reference to sys.stdout when they are instantiated and you can't pickle that either.

It is preferable for the reference to be taken at instantiation time rather than stored as a module level reference to be compatible with test runners (like regrtest) that *themselves* replace sys.stdout...
msg102577 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-04-07 23:31 
Changing unittest.TestResult to use StringIO instead of cStringIO isn't sufficient. regrtest.py also needs to use StringIO...
msg102579 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-04-08 00:15 
I believe this issue to now be fixed on trunk. We can close this when the buildbots are passing again.
msg102581 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-04-08 00:44 
Green build on Windows XP, trunk.

http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%20trunk/builds/282
History
Date User Action Args
2022-04-11 14:56:59adminsetnosy: + benjamin.peterson
github: 52580
2010-04-08 00:44:10michael.foordsetstatus: open -> closed
resolution: accepted
messages: + msg102581

stage: needs patch -> resolved
2010-04-08 00:15:08michael.foordsetmessages: + msg102579
2010-04-07 23:31:26michael.foordsetmessages: + msg102577
2010-04-07 22:56:10michael.foordsetmessages: + msg102574
2010-04-07 22:49:51michael.foordsetmessages: + msg102573
2010-04-07 20:52:52loewissetpriority: high -> release blocker
2010-04-07 20:51:35loewissetkeywords: + buildbot

messages: + msg102565
2010-04-07 20:51:02loewissetnosy: + loewis
messages: + msg102564
2010-04-07 19:17:02michael.foordsetmessages: + msg102561
2010-04-07 18:55:08skrahsetfiles: + issue8333.patch
keywords: + patch
2010-04-07 18:54:52skrahsetnosy: + michael.foord
messages: + msg102556
2010-04-07 18:05:29jnollersetmessages: + msg102555
2010-04-07 18:00:43r.david.murraysetnosy: + jnoller
2010-04-07 14:48:07tim.goldensetnosy: + tim.golden
messages: + msg102542
2010-04-07 13:34:34skrahcreate