seen with the 2.7.11 release candidate; I'm not yet 100% if this is the fix for #22995 causing these test regressions.

[ 93/395] test_cpickle
test test_cpickle failed -- multiple errors occurred; run in verbose mode for details
[101/395/2] test_decimal
test test_decimal failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_decimal.py", line 1774, in test_pickle
    e = pickle.loads(pickle.dumps(c, proto))
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 306, in save
    rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[108/395/4] test_dictviews
test test_dictviews failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_dictviews.py", line 201, in test_pickle
    pickle.dumps, d.viewkeys(), proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/unittest/case.py", line 473, in assertRaises
    callableObj(*args, **kwargs)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 1380, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/pickle.py", line 306, in save
    rv = reduce(self.proto)
AttributeError: 'module' object has no attribute '_reduce_ex'

[177/395/7] test_imp
test test_imp failed -- Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_imp.py", line 60, in test_source
    imp.reload(os)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/os.py", line 727, in <module>
    _copy_reg.pickle(stat_result, _pickle_stat_result, _make_stat_result)
AttributeError: 'module' object has no attribute 'pickle'

[184/395/10] test_int_literal
[185/395/10] test_ioctl
test test_ioctl crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[186/395/11] test_isinstance
[187/395/11] test_iter
[188/395/11] test_iterlen
[189/395/11] test_itertools
[190/395/11] test_json
test test_json crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[191/395/12] test_kqueue
test_kqueue skipped -- test works only on BSD
[192/395/12] test_largefile
test test_largefile crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[193/395/13] test_lib2to3
test test_lib2to3 crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[194/395/14] test_linecache
test test_linecache crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[195/395/15] test_list
test test_list crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[196/395/16] test_locale
[197/395/16] test_logging
test test_logging crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[198/395/17] test_long
[199/395/17] test_long_future
[200/395/17] test_longexp
[201/395/17] test_macos
test test_macos crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[202/395/18] test_macostools
test test_macostools crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[203/395/19] test_macpath
[204/395/19] test_macurl2path
[205/395/19] test_mailbox
test test_mailbox crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[206/395/20] test_marshal
test test_marshal crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[207/395/21] test_math
test test_math crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'
[208/395/22] test_md5
[209/395/22] test_memoryio
test test_memoryio failed -- multiple errors occurred; run in verbose mode for details
[210/395/23] test_memoryview
[211/395/23] test_mhlib
test test_mhlib crashed -- <type 'exceptions.AttributeError'>: 'module' object has no attribute 'pickle'

Duplicate or related to Issue 25601?

Yes, it is a duplicate of issue25601. And I can't reproduce it locally.

Matthias, can you reproduce test failure every time when run tests? If yes, could you please run tests in verbose mode and found what tests in test_cpickle were failed first?

looks like re-running the tests in verbose mode using -w is affected by this issue.

the first cpickle subtest failing is:

======================================================================
ERROR: test_recursive_dict_subclass_and_inst (test.test_cpickle.cStringIOCPicklerFastTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 175, in wrapper
    func(self)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 819, in test_recursive_dict_subclass_a
nd_inst
    self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 790, in check_recursive_collection_and
_inst
    s = self.dumps(y, proto)
  File "/home/packages/python/2.7/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 141, in dumps
    p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

In your report test_cpickle has number 93. What other 92 tests were ran before? Could you run tests few times and find minimal list of tests that precede failing test_cpickle?

This failure is not related to issue22995 because the same failure was reported in issue25601 5 commits before first commit in issue22995.

This can be related to issue25083. Can you look at the size of copy_reg.pyc and copy_reg.pyo files?

This seems entirely reproducible in a local Xenial sbuild using the source packages in https://launchpad.net/~doko/+archive/ubuntu/toolchain/+packages

Interestingly, inspection of the built artifacts in the schroot after all is done yields:

# find . -name copy_reg.py? | xargs ls -l
-rw-rw-r-- 1 barry barry 5597 Nov 30 16:52 ./Lib/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5553 Nov 30 16:53 ./Lib/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:03 ./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:03 ./debian/tmp-dbg/usr/lib/python2.7/copy_reg.pyo
-rw-rw-r-- 1 barry barry 5091 Nov 30 17:02 ./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyc
-rw-rw-r-- 1 barry barry 5047 Nov 30 17:02 ./debian/tmp-shdbg/usr/lib/python2.7/copy_reg.pyo

Those are debug builds, so it seems reasonable they are different sizes than the non-debug artifacts in Lib, right?  And since they are self-consistent in size, it seems like it wouldn't be caused by a pyc/pyo write bug.

First failure in test run afaict:

======================================================================
ERROR: test_recursive_dict_subclass_and_inst (test.test_cpickle.cStringIOCPicklerFastTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 175, in wrapper
    func(self)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 819, in test_recursive_dict_subclass_and_inst
    self.check_recursive_collection_and_inst(MyDict.fromkeys)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/pickletester.py", line 790, in check_recursive_collection_and_inst
    s = self.dumps(y, proto)
  File "/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/test/test_cpickle.py", line 141, in dumps
    p.dump(arg)
AttributeError: 'module' object has no attribute '_reduce_ex'

======================================================================

But if I just run test_cpickle.py, it succeeds!

Could you please add debugging output to check that the copy_reg module is empty if tests failed?

I set a breakpoint where the error occurs.  I'm in test_cpickle.py dumps().  Doing cPickle.dumps(arg)  in the bp, I get the error.  arg is:

(Pdb) arg
{<__main__.H object at 0x7ff8e2a870d0>: None}
(Pdb) type(arg)
<class 'test.pickletester.MyDict'>

Look again at the AttributeError:

(Pdb) cPickle.dumps(arg)
*** AttributeError: 'module' object has no attribute '_reduce_ex'

First, why is that trying to find the attribute of a module object instead of the MyDict instance?  I even tried this just to discount the effects of __main__.H:

(Pdb) cPickle.dumps(type(arg)())
*** AttributeError: 'module' object has no attribute '_reduce_ex'

(as you'd expect, type(arg)() is just an instance of a MyDict.

Two questions: why is it trying to get the attribute on a module object, and why is the attribute it's trying to get '_reduce_ex' instead of '__reduce_ex__'?

The attribute to get is taken from the __reduce_ex___str in cPickle.c, which gets defined as:

#define INIT_STR(S) if (!( S ## _str=PyString_InternFromString(#S)))  return -1;
...
    INIT_STR(__reduce_ex__);

This is a compile-time definition, so it makes no sense that it would be trying to find the wrong attribute in some cases (e.g. when running the full regrtest.py) but not in others (e.g. running just test_cpickle.py).  But I can find no other references to a string containing "reduce_ex" so the cPickle version has to be it.  And that gets initialized in init_stuff(), called from initcPickle() so I don't see any other way for it to get corrupted.

cPickle.dumps({}) works just fine, so it's an artifact of the MyDict.  That's all I've figured out for now.

Default __reduce_ex__ and __reduce__ implementations call copy_reg._reduce_ex. cPickle.dumps({}) works because __reduce_ex__ is not called, dict is supported by pickle directly.

... for protocols < 2.

http://buildbot.python.org/all/builders/PPC64LE%20Fedora%202.7/builds/164/steps/test/logs/stdio

Failed a number of tests that directly or indirectly use copy_reg. First failed test is test_cpickle.

Well, one thing I noticed is that init_stuff() in cPickle.c can leak the copy_reg module object, if any of the subsequent PyObject_GetAttr*() calls fail.  They return -1 without decref'ing copyreg.  Probably not a horrible leak since it's probably rare, but it should be fixed on principle.

At the problematic breakpoint mentioned in msg255630, copy_reg is nearly empty.

(Pdb) sys.modules['copy_reg']
<module 'copy_reg' from '/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/copy_reg.pyc'>
(Pdb) dir(sys.modules['copy_reg'])
['__builtins__', '__doc__', '__file__', '__name__', '__package__']
(Pdb) sys.modules['copy_reg'].__file__
'/build/python2.7-96oGYh/python2.7-2.7.11~rc1/Lib/copy_reg.pyc'
(Pdb) sys.modules['copy_reg'].__doc__ 
(Pdb) sys.modules['copy_reg'].__name__
'copy_reg'
(Pdb) sys.modules['copy_reg'].__package__

Was it empty before running any test_cpickle tests? Could you find after what test it becomes empty?

This just gets weirder.  I've narrowed it down to running this command:

build-static/python Lib/test/regrtest.py test_doctest test_cpickle

fails every time.  What's even weirder is that I hacked regrtest to print some useful information before and after each test run.  It prints the id(sys.modules['copy_reg']) and len(dir(sys.modules['copy_reg']))).  Here's the output for two test runs.  You can see that if test_doctest is run before test_cpickle, regrtest sees a different module in sys.modules['copy_reg'], but only *after* test_cpickle runs.  Using test_doctest2 instead is fine.

I'll note one other thing.  Doko mentioned that in our build environment, cPickle is built in, not an extension.

# build-static/python Lib/test/regrtest.py test_doctest test_cpickle
[1/2] test_doctest
('BEFORE =====>', 140559940644712, 22)
('AFTER =====>', 140559940644712, 22)
[2/2] test_cpickle
('BEFORE =====>', 140559940644712, 22)
test test_cpickle failed -- multiple errors occurred; run in verbose mode for details
('AFTER =====>', 140559897320928, 5)
1 test OK.
1 test failed:
    test_cpickle

# build-static/python Lib/test/regrtest.py test_doctest2 test_cpickle
[1/2] test_doctest2
('BEFORE =====>', 139794866929512, 22)
('AFTER =====>', 139794866929512, 22)
[2/2] test_cpickle
('BEFORE =====>', 139794866929512, 22)
('AFTER =====>', 139794866929512, 22)
All 2 tests OK.

On Dec 01, 2015, at 06:26 PM, Barry A. Warsaw wrote:

>I'll note one other thing.  Doko mentioned that in our build environment,
>cPickle is built in, not an extension.

Which may be relevant because in a from-hg-head build of Python 2.7, this
problem doesn't occur.  I haven't tried 2.7.11 tarball yet.

Thanks Barry. Running test_cpickle after test_doctest I now can reproduce the problem in a from-hg-head build of Python 2.7.

On Dec 01, 2015, at 06:47 PM, Serhiy Storchaka wrote:

>Thanks Barry. Running test_cpickle after test_doctest I now can reproduce the
>problem in a from-hg-head build of Python 2.7.

That's interesting because I can't!  (Neither from the 2.7.11rc1 tarball.)

Did you make cPickle a built-in like Doko did?

Ah, sorry, I missed and mislead you. My tests were specially hacked to simulate a bug (sys.modules['copy_reg'] = object()). No, actually I can't reproduce it.

I added cPickle to Modules/Setup.local and rebuilt from hg.  cPickle's a built-in but I still can't reproduce the problem.

I can reproduce with

% ./python Lib/test/regrtest.py test___all__ test_memoryio test_zipimport test_cpickle


The underlying issue is classic: PyDict_SetItem returns NULL if the stack is too deep. This confuses import.

Not new, so not a release blocker.

Thank you Benjamin! This stably reproduce the issue. Just wondering how you found this minimal sequence?

On Dec 05, 2015, at 07:03 AM, Benjamin Peterson wrote:

>The underlying issue is classic: PyDict_SetItem returns NULL if the stack is
>too deep. This confuses import.

Thanks for digging into this Benjamin.  I'm not able to investigate further
atm, but could you please explain your analysis in more detail?  Where's the
faulty setitem?  Where is import getting confused?  Why does this (or the
combination I found before) cause the problem?

Err, sorry, it's actually PyDict_GetItem which is the problem. Basically
if you try to import with a very deep stack, doing PyDict_GetItem on
sys.modules can return NULL (fro copy_reg). That confuses import a lot.

I haven't exactly figured out why those tests cause failure. One of them
probably puts unicode strings in sys.modules.

On Sat, Dec 5, 2015, at 06:28, Barry A. Warsaw wrote:
> 
> Barry A. Warsaw added the comment:
> 
> On Dec 05, 2015, at 07:03 AM, Benjamin Peterson wrote:
> 
> >The underlying issue is classic: PyDict_SetItem returns NULL if the stack is
> >too deep. This confuses import.
> 
> Thanks for digging into this Benjamin.  I'm not able to investigate
> further
> atm, but could you please explain your analysis in more detail?  Where's
> the
> faulty setitem?  Where is import getting confused?  Why does this (or the
> combination I found before) cause the problem?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue25698>
> _______________________________________

Err, sorry, it's actually PyDict_GetItem which is the problem. Basically
if you try to import with a very deep stack, doing PyDict_GetItem on
sys.modules can return NULL (fro copy_reg). That confuses import a lot.

I haven't exactly figured out why those tests cause failure. One of them
probably puts unicode strings in sys.modules.

On Sat, Dec 5, 2015, at 06:28, Barry A. Warsaw wrote:
> 
> Barry A. Warsaw added the comment:
> 
> On Dec 05, 2015, at 07:03 AM, Benjamin Peterson wrote:
> 
> >The underlying issue is classic: PyDict_SetItem returns NULL if the stack is
> >too deep. This confuses import.
> 
> Thanks for digging into this Benjamin.  I'm not able to investigate
> further
> atm, but could you please explain your analysis in more detail?  Where's
> the
> faulty setitem?  Where is import getting confused?  Why does this (or the
> combination I found before) cause the problem?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue25698>
> _______________________________________

One solution is to skip tests caused deep recursion.

The better solution would be to fix import machinery to not produce empty modules.

Here is a patch that makes import machinery to not left empty module in case of recursion error. I prefer this solution.

Benjamin, could you please make a review?

I don't think this patch is the best way. The correct way is to have and
use PyDict_GetWithError. I wouldn't be surprised if you could crash
Python under this patch by putting some function with an import in
__cmp__ in sys.modules.

On Sun, Feb 7, 2016, at 14:23, Serhiy Storchaka wrote:
> 
> Serhiy Storchaka added the comment:
> 
> Benjamin, could you please make a review?
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue25698>
> _______________________________________

Agreed. Here is a patch that makes PyImport_AddModule to use PyDict_GetWithError and fail earlier. I think it should be applied in 3.x too.

That seems fine except you might as well just put the GetWithError prototype in the header file.

New changeset 81dba6e392c3 by Serhiy Storchaka in branch '2.7':
Issue #25698: Importing module if the stack is too deep no longer replaces
https://hg.python.org/cpython/rev/81dba6e392c3

New changeset 1c2de3b0a474 by Serhiy Storchaka in branch '3.5':
Issue #25698: Prevent possible replacing imported module with the empty one
https://hg.python.org/cpython/rev/1c2de3b0a474

New changeset c711c36cf988 by Serhiy Storchaka in branch 'default':
Issue #25698: Prevent possible replacing imported module with the empty one
https://hg.python.org/cpython/rev/c711c36cf988

> That seems fine except you might as well just put the GetWithError prototype in the header file.

Yes, of course. It was temporary placed here to avoid rebuilding all sources.

Thank you Barry and Benjamin for investigating this issue.

Thanks for digging deeply and fixing this!

I see test_cpickle failing with NPE while running the tests with IBM Java.

Exception Stack Trace
--------------------------------
<testsuite tests="144" errors="2" failures="0" skipped="0" name="test.test_cpickle.cPickleTests" time="0.049">
  <testcase name="test_attribute_name_interning" time="0.036"/>
  <testcase name="test_bad_input" time="0.786">
    <error type="java.lang.NullPointerException" message="java.lang.NullPointerException"><![CDATA[Traceback (most recent call last):
  File "/jtctest/apps/ascii/jython/80files/jython_2.7.0/Lib/unittest/case.py", line 332, in run
    testMethod()
  File "/jtctest/apps/ascii/jython/80files/jython_2.7.0/Lib/test/pickletester.py", line 1136, in test_bad_input
    self.assertRaises((IndexError, cPickle.UnpicklingError),
  File "/jtctest/apps/ascii/jython/80files/jython_2.7.0/Lib/unittest/case.py", line 476, in assertRaises
    callableObj(*args, **kwargs)
NullPointerException: java.lang.NullPointerException

Jython used
-------------------
2.7.0

IBM Java version
-------------------
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 8.0.5.0 - pxp3280sr5-20170725_01(SR5 Beta-1)

Please let me know if the issue is similar or it requires a new bug. Thanks.