With the "pydoc.synopsis() for 'binary' modules" changes (d6c3fb8d5f84) to pydoc and test_pydoc for Issue20123, running test_pydoc can leave the regrtest environment altered and cause subsequent tests to fail in strange ways.  I ran into this issue running regression tests for 3.4.0 and was able to isolate the failure to this change set.

A somewhat simplified (if not necessarily minimal) set of steps to reproduce follow.  I've not had time to further investigate the cause.  I've also only seen the problem when running the tests with an installed Python or from a build directory when there is also a Python installed in the configured prefix but not from the build directory with no Python installed at the configured prefix.  I've reproduced the failure on both Debian Linux and on OS X.  The following output is from the former.

$ rm -rf /tmp/root/*
$ hg purge --all
$ hg update 6d72617cae64  # the parent change set of d6c3fb8d5f84
$ hg log -r .
changeset:   88306:6d72617cae64
user:        R David Murray <rdmurray@bitdance.com>
date:        Sat Jan 04 21:17:52 2014 -0500
summary:     whatsnew: removal of TYPE_INT64 from marshal.
$ ./configure --prefix=/tmp/root
$ make && make install
$ ( cd /tmp && /tmp/root/bin/python3.4 -m test -w test_pydoc test_distutils )
[1/2] test_pydoc
[2/2] test_distutils
All 2 tests OK.
$ hg update d6c3fb8d5f84
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg log -r .
changeset:   88307:d6c3fb8d5f84
user:        Eric Snow <ericsnowcurrently@gmail.com>
date:        Sat Jan 04 20:38:11 2014 -0700
summary:     Issue 20123: Fix pydoc.synopsis() for "binary" modules.
$ make && make install
$ ( cd /tmp && /tmp/root/bin/python3.4 -m test -w test_pydoc test_distutils )
[1/2] test_pydoc
Warning -- warnings.filters was modified by test_pydoc
Warning -- logging._handlerList was modified by test_pydoc
[2/2] test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for details
1 test failed:
    test_distutils
1 test altered the execution environment:
    test_pydoc
Re-running failed tests in verbose mode
Re-running test 'test_distutils' in verbose mode
test_simple_built (distutils.tests.test_bdist_dumb.BuildDumbTestCase) ... FAIL
test_check_all (distutils.tests.test_check.CheckTestCase) ... ok
[...]

======================================================================
ERROR: test_metadata_check_option (distutils.tests.test_sdist.SDistTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_sdist.py", line 247, in test_metadata_check_option
    cmd.run()
  File "/tmp/root/lib/python3.4/distutils/command/sdist.py", line 158, in run
    self.get_file_list()
  File "/tmp/root/lib/python3.4/distutils/command/sdist.py", line 204, in get_file_list
    self.add_defaults()
  File "/tmp/root/lib/python3.4/distutils/command/sdist.py", line 262, in add_defaults
    self.filelist.extend(build_py.get_source_files())
  File "/tmp/root/lib/python3.4/distutils/command/build_py.py", line 301, in get_source_files
    return [module[-1] for module in self.find_all_modules()]
  File "/tmp/root/lib/python3.4/distutils/command/build_py.py", line 295, in find_all_modules
    package_dir = self.get_package_dir(package)
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/build_py.py", line 187, in get_package_dir
    return os.path.join(self.distribution.src_root, res)
  File "/tmp/root/lib/python3.4/posixpath.py", line 83, in join
    path += b
TypeError: unsupported operand type(s) for +=: 'dict' and 'str'

======================================================================
ERROR: test_debug_mode (distutils.tests.test_install.InstallTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_install.py", line 236, in test_debug_mode
    self.test_record()
  File "/tmp/root/lib/python3.4/distutils/tests/test_install.py", line 186, in test_record
    cmd.run()
  File "/tmp/root/lib/python3.4/distutils/command/install.py", line 566, in run
    self.run_command(cmd_name)
  File "/tmp/root/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/root/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/install_scripts.py", line 18, in run
    self.run_command("egg_info")
  File "/tmp/root/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/root/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 163, in run
    self.find_sources()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 186, in find_sources
    mm.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 246, in run
    self.add_defaults()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 282, in add_defaults
    sdist.add_defaults(self)
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/sdist.py", line 148, in add_defaults
    if os.path.exists(fn):
  File "/tmp/root/lib/python3.4/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: can't specify None for path argument

======================================================================
ERROR: test_record (distutils.tests.test_install.InstallTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_install.py", line 186, in test_record
    cmd.run()
  File "/tmp/root/lib/python3.4/distutils/command/install.py", line 566, in run
    self.run_command(cmd_name)
  File "/tmp/root/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/root/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/install_scripts.py", line 18, in run
    self.run_command("egg_info")
  File "/tmp/root/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/root/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 163, in run
    self.find_sources()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 186, in find_sources
    mm.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 246, in run
    self.add_defaults()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 282, in add_defaults
    sdist.add_defaults(self)
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/sdist.py", line 148, in add_defaults
    if os.path.exists(fn):
  File "/tmp/root/lib/python3.4/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: can't specify None for path argument

======================================================================
ERROR: test_record_extensions (distutils.tests.test_install.InstallTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_install.py", line 217, in test_record_extensions
    cmd.run()
  File "/tmp/root/lib/python3.4/distutils/command/install.py", line 566, in run
    self.run_command(cmd_name)
  File "/tmp/root/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/root/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/install_egg_info.py", line 29, in run
    self.run_command('egg_info')
  File "/tmp/root/lib/python3.4/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/root/lib/python3.4/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 163, in run
    self.find_sources()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 186, in find_sources
    mm.run()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 246, in run
    self.add_defaults()
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/egg_info.py", line 282, in add_defaults
    sdist.add_defaults(self)
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/sdist.py", line 148, in add_defaults
    if os.path.exists(fn):
  File "/tmp/root/lib/python3.4/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: can't specify None for path argument

======================================================================
ERROR: test_get_inputs (distutils.tests.test_install_lib.InstallLibTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_install_lib.py", line 91, in test_get_inputs
    inputs = cmd.get_inputs()
  File "/tmp/root/lib/python3.4/distutils/command/install_lib.py", line 211, in get_inputs
    inputs.extend(build_py.get_outputs())
  File "/tmp/root/lib/python3.4/distutils/command/build_py.py", line 308, in get_outputs
    modules = self.find_all_modules()
  File "/tmp/root/lib/python3.4/distutils/command/build_py.py", line 295, in find_all_modules
    package_dir = self.get_package_dir(package)
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/build_py.py", line 187, in get_package_dir
    return os.path.join(self.distribution.src_root, res)
  File "/tmp/root/lib/python3.4/posixpath.py", line 83, in join
    path += b
TypeError: unsupported operand type(s) for +=: 'dict' and 'str'

======================================================================
ERROR: test_get_outputs (distutils.tests.test_install_lib.InstallLibTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_install_lib.py", line 71, in test_get_outputs
    outputs = cmd.get_outputs()
  File "/tmp/root/lib/python3.4/distutils/command/install_lib.py", line 188, in get_outputs
    self.install_dir)
  File "/tmp/root/lib/python3.4/distutils/command/install_lib.py", line 148, in _mutate_outputs
    build_files = build_cmd.get_outputs()
  File "/tmp/root/lib/python3.4/distutils/command/build_py.py", line 308, in get_outputs
    modules = self.find_all_modules()
  File "/tmp/root/lib/python3.4/distutils/command/build_py.py", line 295, in find_all_modules
    package_dir = self.get_package_dir(package)
  File "/tmp/root/lib/python3.4/site-packages/setuptools/command/build_py.py", line 187, in get_package_dir
    return os.path.join(self.distribution.src_root, res)
  File "/tmp/root/lib/python3.4/posixpath.py", line 83, in join
    path += b
TypeError: unsupported operand type(s) for +=: 'dict' and 'str'

======================================================================
FAIL: test_simple_built (distutils.tests.test_bdist_dumb.BuildDumbTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_bdist_dumb.py", line 91, in test_simple_built
    self.assertEqual(contents, sorted(wanted))
AssertionError: Lists differ: ['PKG-INFO', 'SOURCES.txt', 'dependency_lin[52 chars]txt'] != ['foo-0.1-py3.4.egg-info', 'foo.cpython-34.pyc', 'foo.py']

First differing element 0:
PKG-INFO
foo-0.1-py3.4.egg-info

First list contains 3 additional elements.
First extra element 3:
foo.cpython-34.pyc

+ ['foo-0.1-py3.4.egg-info', 'foo.cpython-34.pyc', 'foo.py']
- ['PKG-INFO',
-  'SOURCES.txt',
-  'dependency_links.txt',
-  'foo.cpython-34.pyc',
-  'foo.py',
-  'top_level.txt']

======================================================================
FAIL: test_formats (distutils.tests.test_bdist.BuildTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_bdist.py", line 26, in test_formats
    self.assertEqual(found, formats)
AssertionError: Lists differ: ['bztar', 'egg', 'gztar', 'msi', 'rpm', 'tar', 'wininst', 'zip', 'ztar'] != ['bztar', 'gztar', 'msi', 'rpm', 'tar', 'wininst', 'zip', 'ztar']

First differing element 1:
egg
gztar

First list contains 1 additional elements.
First extra element 8:
ztar

- ['bztar', 'egg', 'gztar', 'msi', 'rpm', 'tar', 'wininst', 'zip', 'ztar']
?           -------

+ ['bztar', 'gztar', 'msi', 'rpm', 'tar', 'wininst', 'zip', 'ztar']

======================================================================
FAIL: test_dir_in_package_data (distutils.tests.test_build_py.BuildPyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_build_py.py", line 153, in test_dir_in_package_data
    dist.run_commands()
distutils.errors.DistutilsFileError: can't copy 'pkg/doc/otherdir': doesn't exist or not a regular file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_build_py.py", line 155, in test_dir_in_package_data
    self.fail("failed package_data when data dir includes a dir")
AssertionError: failed package_data when data dir includes a dir

======================================================================
FAIL: test_empty_package_dir (distutils.tests.test_build_py.BuildPyTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_build_py.py", line 87, in test_empty_package_dir
    dist.run_commands()
distutils.errors.DistutilsFileError: can't copy 'oc/testfile': doesn't exist or not a regular file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/root/lib/python3.4/distutils/tests/test_build_py.py", line 89, in test_empty_package_dir
    self.fail("failed package_data test when package_dir is ''")
AssertionError: failed package_data test when package_dir is ''

----------------------------------------------------------------------
Ran 201 tests in 0.915s

FAILED (failures=4, errors=6, skipped=19)
test test_distutils failed
$ rm -rf /tmp/root/*
$ hg purge --all
$ hg update default  # current tip
339 files updated, 0 files merged, 2 files removed, 0 files unresolved
$ hg log -r .
changeset:   88889:d7ac90c0463a
tag:         tip
user:        Victor Stinner <victor.stinner@gmail.com>
date:        Sat Feb 01 22:49:59 2014 +0100
summary:     Issue #20400: Merge Tulip into Python: add the new asyncio.subprocess module
$ ./configure --prefix=/tmp/root
$ make && make install
$ ( cd /tmp && /tmp/root/bin/python3.4 -m test -w test_pydoc test_distutils )
[1/2] test_pydoc
Warning -- warnings.filters was modified by test_pydoc
Warning -- logging._handlerList was modified by test_pydoc
[2/2] test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for details
1 test failed:
    test_distutils
1 test altered the execution environment:
    test_pydoc
Re-running failed tests in verbose mode
Re-running test 'test_distutils' in verbose mode
test_simple_built (distutils.tests.test_bdist_dumb.BuildDumbTestCase) ... FAIL
[...]
----------------------------------------------------------------------
Ran 201 tests in 0.959s

FAILED (failures=4, errors=6, skipped=19)
test test_distutils failed
$ # problem still exists

Here's another example of an odd test failure apparently caused by test_pydoc side effects, this using current default tip (same results with the current cherry pick for 3.4.0rc2):

./bin/python3.4 -m test -w -uall,-largefile test_pydoc test_unicode
[1/2] test_pydoc
Warning -- warnings.filters was modified by test_pydoc
Warning -- logging._handlerList was modified by test_pydoc
[2/2] test_unicode
test test_unicode failed -- Traceback (most recent call last):
  File "/py/dev/3x/root/fwn/Library/Frameworks/pytest_10.9.framework/Versions/3.4/lib/python3.4/test/test_unicode.py", line 1072, in test_formatting
    string_tests.MixinStrUnicodeUserStringTest.test_formatting(self)
  File "/py/dev/3x/root/fwn/Library/Frameworks/pytest_10.9.framework/Versions/3.4/lib/python3.4/test/string_tests.py", line 1181, in test_formatting
    with self.assertWarns(DeprecationWarning):
  File "/py/dev/3x/root/fwn/Library/Frameworks/pytest_10.9.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 202, in __enter__
    for v in sys.modules.values():
RuntimeError: dictionary changed size during iteration

1 test failed:
    test_unicode
1 test altered the execution environment:
    test_pydoc
Re-running failed tests in verbose mode
Re-running test 'test_unicode' in verbose mode
test_formatter_field_name_split (test.test_unicode.StringModuleTest) ... ok
test_formatter_parser (test.test_unicode.StringModuleTest) ... ok
test___contains__ (test.test_unicode.UnicodeTest) ... ok
test_additional_rsplit (test.test_unicode.UnicodeTest) ... ok
test_additional_split (test.test_unicode.UnicodeTest) ... ok
test_ascii (test.test_unicode.UnicodeTest) ... ok
test_aswidechar (test.test_unicode.UnicodeTest) ... ok
test_aswidecharstring (test.test_unicode.UnicodeTest) ... ok
test_bug1001011 (test.test_unicode.UnicodeTest) ... ok
test_bytes_comparison (test.test_unicode.UnicodeTest) ... ok
test_capitalize (test.test_unicode.UnicodeTest) ... ok
test_casefold (test.test_unicode.UnicodeTest) ... ok
test_center (test.test_unicode.UnicodeTest) ... ok
test_codecs (test.test_unicode.UnicodeTest) ... ok
test_codecs_charmap (test.test_unicode.UnicodeTest) ... ok
test_codecs_errors (test.test_unicode.UnicodeTest) ... ok
test_codecs_idna (test.test_unicode.UnicodeTest) ... ok
test_codecs_utf7 (test.test_unicode.UnicodeTest) ... ok
test_codecs_utf8 (test.test_unicode.UnicodeTest) ... ok
test_compare (test.test_unicode.UnicodeTest) ... ok
test_comparison (test.test_unicode.UnicodeTest) ... ok
test_concatenation (test.test_unicode.UnicodeTest) ... ok
test_constructor (test.test_unicode.UnicodeTest) ... ok
test_constructor_defaults (test.test_unicode.UnicodeTest)
Check the constructor argument defaults. ... ok
test_constructor_keyword_args (test.test_unicode.UnicodeTest)
Pass various keyword argument combinations to the constructor. ... ok
test_contains (test.test_unicode.UnicodeTest) ... ok
test_conversion (test.test_unicode.UnicodeTest) ... ok
test_count (test.test_unicode.UnicodeTest) ... ok
test_encode_decimal (test.test_unicode.UnicodeTest) ... ok
test_endswith (test.test_unicode.UnicodeTest) ... ok
test_expandtabs (test.test_unicode.UnicodeTest) ... ok
test_expandtabs_optimization (test.test_unicode.UnicodeTest) ... ok
test_expandtabs_overflows_gracefully (test.test_unicode.UnicodeTest) ... skipped 'only applies to 32-bit platforms'
test_extended_getslice (test.test_unicode.UnicodeTest) ... ok
test_find (test.test_unicode.UnicodeTest) ... ok
test_find_etc_raise_correct_error_messages (test.test_unicode.UnicodeTest) ... ok
test_floatformatting (test.test_unicode.UnicodeTest) ... ok
test_format (test.test_unicode.UnicodeTest) ... ok
test_format_auto_numbering (test.test_unicode.UnicodeTest) ... ok
test_format_float (test.test_unicode.UnicodeTest) ... ok
test_format_huge_item_number (test.test_unicode.UnicodeTest) ... ok
test_format_huge_precision (test.test_unicode.UnicodeTest) ... ok
test_format_huge_width (test.test_unicode.UnicodeTest) ... ok
test_format_map (test.test_unicode.UnicodeTest) ... ok
test_format_subclass (test.test_unicode.UnicodeTest) ... ok
test_formatting (test.test_unicode.UnicodeTest) ... ERROR
test_formatting_c_limits (test.test_unicode.UnicodeTest) ... ok
test_formatting_huge_precision (test.test_unicode.UnicodeTest) ... ok
test_formatting_huge_precision_c_limits (test.test_unicode.UnicodeTest) ... ok
test_formatting_huge_width (test.test_unicode.UnicodeTest) ... ok
test_formatting_with_enum (test.test_unicode.UnicodeTest) ... ok
test_from_format (test.test_unicode.UnicodeTest) ... ok
test_getnewargs (test.test_unicode.UnicodeTest) ... ok
test_hash (test.test_unicode.UnicodeTest) ... ok
test_index (test.test_unicode.UnicodeTest) ... ok
test_inplace_rewrites (test.test_unicode.UnicodeTest) ... ok
test_invalid_cb_for_2bytes_seq (test.test_unicode.UnicodeTest) ... ok
test_invalid_cb_for_3bytes_seq (test.test_unicode.UnicodeTest) ... ok
test_invalid_cb_for_4bytes_seq (test.test_unicode.UnicodeTest) ... ok
test_invalid_start_byte (test.test_unicode.UnicodeTest) ... ok
test_isalnum (test.test_unicode.UnicodeTest) ... ok
test_isalpha (test.test_unicode.UnicodeTest) ... ok
test_isdecimal (test.test_unicode.UnicodeTest) ... ok
test_isdigit (test.test_unicode.UnicodeTest) ... ok
test_isidentifier (test.test_unicode.UnicodeTest) ... ok
test_islower (test.test_unicode.UnicodeTest) ... ok
test_isnumeric (test.test_unicode.UnicodeTest) ... ok
test_isprintable (test.test_unicode.UnicodeTest) ... ok
test_isspace (test.test_unicode.UnicodeTest) ... ok
test_issue18183 (test.test_unicode.UnicodeTest) ... ok
test_issue8271 (test.test_unicode.UnicodeTest) ... ok
test_istitle (test.test_unicode.UnicodeTest) ... ok
test_isupper (test.test_unicode.UnicodeTest) ... ok
test_iterators (test.test_unicode.UnicodeTest) ... ok
test_join (test.test_unicode.UnicodeTest) ... ok
test_literals (test.test_unicode.UnicodeTest) ... ok
test_ljust (test.test_unicode.UnicodeTest) ... ok
test_lower (test.test_unicode.UnicodeTest) ... ok
test_maketrans_translate (test.test_unicode.UnicodeTest) ... ok
test_mul (test.test_unicode.UnicodeTest) ... ok
test_none_arguments (test.test_unicode.UnicodeTest) ... ok
test_partition (test.test_unicode.UnicodeTest) ... ok
test_printable_repr (test.test_unicode.UnicodeTest) ... ok
test_printing (test.test_unicode.UnicodeTest) ... ok
test_raiseMemError (test.test_unicode.UnicodeTest) ... ok
test_replace (test.test_unicode.UnicodeTest) ... ok
test_replace_id (test.test_unicode.UnicodeTest) ... ok
test_replace_overflow (test.test_unicode.UnicodeTest) ... skipped 'only applies to 32-bit platforms'
test_repr (test.test_unicode.UnicodeTest) ... ok
test_resize (test.test_unicode.UnicodeTest) ... ok
test_rfind (test.test_unicode.UnicodeTest) ... ok
test_rindex (test.test_unicode.UnicodeTest) ... ok
test_rjust (test.test_unicode.UnicodeTest) ... ok
test_rpartition (test.test_unicode.UnicodeTest) ... ok
test_rsplit (test.test_unicode.UnicodeTest) ... ok
test_slice (test.test_unicode.UnicodeTest) ... ok
test_split (test.test_unicode.UnicodeTest) ... ok
test_splitlines (test.test_unicode.UnicodeTest) ... ok
test_startswith (test.test_unicode.UnicodeTest) ... ok
test_startswith_endswith_errors (test.test_unicode.UnicodeTest) ... ok
test_strip (test.test_unicode.UnicodeTest) ... ok
test_subclass_add (test.test_unicode.UnicodeTest) ... ok
test_subscript (test.test_unicode.UnicodeTest) ... ok
test_surrogates (test.test_unicode.UnicodeTest) ... ok
test_swapcase (test.test_unicode.UnicodeTest) ... ok
test_title (test.test_unicode.UnicodeTest) ... ok
test_transform_decimal (test.test_unicode.UnicodeTest) ... ok
test_ucs4 (test.test_unicode.UnicodeTest) ... ok
test_unexpected_end_of_data (test.test_unicode.UnicodeTest) ... ok
test_unicode_repr (test.test_unicode.UnicodeTest) ... ok
test_upper (test.test_unicode.UnicodeTest) ... ok
test_utf8_decode_invalid_sequences (test.test_unicode.UnicodeTest) ... ok
test_utf8_decode_valid_sequences (test.test_unicode.UnicodeTest) ... ok
test_zfill (test.test_unicode.UnicodeTest) ... ok

======================================================================
ERROR: test_formatting (test.test_unicode.UnicodeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/py/dev/3x/root/fwn/Library/Frameworks/pytest_10.9.framework/Versions/3.4/lib/python3.4/test/test_unicode.py", line 1072, in test_formatting
    string_tests.MixinStrUnicodeUserStringTest.test_formatting(self)
  File "/py/dev/3x/root/fwn/Library/Frameworks/pytest_10.9.framework/Versions/3.4/lib/python3.4/test/string_tests.py", line 1181, in test_formatting
    with self.assertWarns(DeprecationWarning):
  File "/py/dev/3x/root/fwn/Library/Frameworks/pytest_10.9.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 202, in __enter__
    for v in sys.modules.values():
RuntimeError: dictionary changed size during iteration

----------------------------------------------------------------------
Ran 114 tests in 1.122s

FAILED (errors=1, skipped=2)
test test_unicode failed

Eric, can you take a look at this?  It is playing minor havoc with regression testing for 3.4.0.

Sorry for the delay.  It will likely be tomorrow before I can take a look but I'll do so as soon as possible.

TL;DR new tests (improving coverage) uncovered existing "bugs".  We should probably disable the tests for now.

I'm glad you found this.  Out of curiosity, how often do you run the test suite against a clean checkout?  Typically I only run it out of the same "tainted" directory that I develop in and not very frequently with -w.  Have you seen related failures on any buildbots?

Anyway, it looks like the actual pydoc changes in the patch aren't at fault.  Rather, the 3 test_modules* tests I added are.  Specifically, something in pydoc.Helper (i.e. help() in the REPL) is the problem.  That's what I get for trying to add test coverage (where there was none) for code I'm fixing! <wink>

If I recall correctly, passing 'modules' to help() does something funny like actually loading *every* module it can find.  I can understand how this might have side effects, maybe expose bugs in other modules, and even cause weird failures when running the test suite! :P

I'm not sure what it will take to get this sorted out.  This may actually be the way that pydoc.Helper()('modules') is supposed to work.  In that case we'd need to fix the modules that are having an issue, namely disutils (and probably logging).  We should open separate issues for each module that needs fixing.  Until then we should probably disable those three tests, particularly for the upcoming 3.4 release (and rc2 if we can squeeze it in).

Any objections before I disable those 3 tests?

> Any objections before I disable those 3 tests?

Not from me. In case I was added to nosy because of logging - AFAICT, test_pydoc is not cleaning up after itself, and one of the problems is a logger which is created / added but not removed / closed.

I concur: let's disable the tests for now.  If we can do a quick patch and request a cherry pick from Larry, we can still get this into 3.4.0.  To answer your question, Eric, I normally run tests from a clean, installed location as part of OS X installer testing and I run them in non-random order so that regressions are more apparent.

New changeset 13edfab6c3c0 by Eric Snow in branch 'default':
Issue #20484: Disable the 2 remaining "modules" tests in test_pydoc.
http://hg.python.org/cpython/rev/13edfab6c3c0

I've disabled 2 of the 3 tests (the other was already disabled for issue20123).  I'll pick up re-enabling the tests in issue20128.

Thanks again, Ned, for finding this.

(mental note: stay away from pydoc!]

Thanks for looking into it, Eric.  Can you open a 3.4 cherry-pick issue for this so it gets into 3.4.0?

I've opened issue20734 for the 3.4.0 cherry-pick.

New changeset 7cfb3db36ec6 by Eric Snow in branch '3.4':
Issue #20484: Disable the 2 remaining "modules" tests in test_pydoc.
http://hg.python.org/cpython/rev/7cfb3db36ec6