This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_multiprocessing crashes under Windows when run in verbose mode
Type: behavior Stage: patch review
Components: Library (Lib), Tests Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: jnoller, michael.foord, pitrou
Priority: high Keywords: patch

Created on 2009-10-24 11:13 by pitrou, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
testrunner.patch pitrou, 2009-10-24 11:49
Messages (6)
msg94406 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-24 11:13
When run under Windows in verbose mode ("python -m test.regrtest -v
test_multiprocessing"), most tests in test_multiprocessing fail with a
recursion limit error.

The explanation is that most tests are written in the following manner:

class _TestArray(BaseTestCase):
    [...]
    def test_array(self, raw=False):
        [...]
        p = self.Process(target=self.f, args=(arr,))

Running a Process under Windows involved pickling it to send it to the
child process. This also pickles the `target` function, which here is a
method of an unittest instance. This consequently pickles the unittest
instance, which has a reference to the unittest runner, which has a
reference to a unittest.runner._WritelnDecorator instance.

The infinite recursion occurs when unpickling the _WritelnDecorator
instance, because the `stream` attribute is not restored when calling
__new__, and the __getattr__ method then recurses when trying to return
`getattr(self.stream,attr)`.

I see two possible resolutions:
- make unittest.runner._WritelnDecorator properly (un)pickleable 
- change all tests in test_multiprocessing to avoid pickling instances
of unittest.TestCase

The former is simpler and probably more future-proof than the latter.

(NB: in non-verbose mode, test.support uses a custom test runner which
doesn't involve the _WritelnDecorator)


Appendix: here is a traceback example (noticed on the newgil branch but
it really happens on stock trunk and py3k):

test_notify (test.test_multiprocessing.WithProcessesTestCondition) ...
Traceback
 (most recent call last):
  File "<string>", line 1, in <module>
  File "Z:\py3k\newgil\lib\multiprocessing\forking.py", line 339, in main
    self = load(from_parent)
  File "Z:\py3k\newgil\lib\pickle.py", line 1365, in load
    encoding=encoding, errors=errors).load()
  File "Z:\py3k\newgil\lib\unittest\runner.py", line 21, in __getattr__
    return getattr(self.stream,attr)
  File "Z:\py3k\newgil\lib\unittest\runner.py", line 21, in __getattr__
    return getattr(self.stream,attr)
  File "Z:\py3k\newgil\lib\unittest\runner.py", line 21, in __getattr__
    return getattr(self.stream,attr)
[... snipped ...]
RuntimeError: maximum recursion depth exceeded while calling a Python
object
msg94407 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-24 11:33
Amusingly, there is a Test_TextTestRunner but it's not run by
test_unittest....
msg94408 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-24 11:49
Here is a patch.
msg95129 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-10 20:29
Ok, if nobody objects, I will commit it since it's simple enough and
blocks any attempt at diagnosing test_multiprocessing under Windows.
msg95130 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-11-10 20:31
Please do; I've been swamped - I'm sorry
msg95141 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-11-11 15:42
This has been committed to trunk (r76196) and py3k (r76198).
I'm reluctant to backport it to 2.6/3.1 because unittest has changed a
lot in-between, and the old version doesn't have a lot of tests. Please
do if you think it's useful :)
History
Date User Action Args
2022-04-11 14:56:54adminsetgithub: 51446
2009-11-11 15:42:47pitrousetstatus: open -> closed
resolution: fixed
messages: + msg95141
2009-11-10 20:31:23jnollersetmessages: + msg95130
2009-11-10 20:29:45pitrousetmessages: + msg95129
2009-10-24 11:49:15pitrousetfiles: + testrunner.patch
keywords: + patch
messages: + msg94408

stage: patch review
2009-10-24 11:33:15pitrousetmessages: + msg94407
2009-10-24 11:13:44pitroucreate