The attached patch adds more tests for pickling:

• Add tests for the module-level load() and dump() functions.
• Add tests for cPickle's internal data structures, stressing workloads
  with many gets/puts.
• Add tests for the Pickler and Unpickler classes, in particular the
  memo attribute.
• test_xpickle is extended to test backwards compatibility with Python
  2.4, 2.5 and 2.6 by round-tripping pickled objects through a worker
  process. 2.3 was too difficult to support.

I'll port to py3k after reviewed for trunk.

Overall, the patch looks good. I haven't reviewed the patch thoroughly
yet, but there is a few things I am not sure about. First, why do you
bother supporting 2.4 (i.e. with a copy of run_with_locale) in a patch
aimed at 2.7?

In test_many_puts_and_gets(), I believe this:

  for proto in [0, 1, 2]:
    ...

should be changed to:

  for proto in protocols:
    ...

Also, I think the tests in AbstractPicklerUnpicklerObjectTests could be
stronger if they used the pickletools module to decompile to check that
only PUT opcodes have changed or not. However, the extra complexity may
not worth it.

And one last thing, why AbstractCompatTests is hard-coded to use
cPickle? Shouldn't the standard pickle module be tested too?

I've made test_xpickle support Python 2.4 because it uses Python 2.4,
2.5 and 2.6 to check that we haven't broken backwards compatibility with
those versions.

I made AbstractCompatTests use cPickle because that's what I've been
changing :) I can add tests for the pure-Python pickle module if you
want. At that point, though, test_xpickle will be very slow and we'll
want to add a regrtest resource to control how much time test_xpickle
spends running. Does xpickle work for the resource name (ie, regrtest.py
-u xpickle)?

I agree with your other points.

How much is "very slow"? If it's less than 10-15 seconds I think it's ok.

test_xpickle currently takes over three minutes on my MacBook Pro to
test compatibility with Python 2.4, 2.5 and 2.6, and adding tests for
the pickle module will at least double that, if not push it well above
10 minutes. That's why I recommend using the resource flag if we want to
add the same tests for the pickle module.

We took a similar issue in test_decimal and came up with a better
approach to using the resource flag. If the resource is not enabled,
the tester runs a random subset of the tests. Over time, that means
that all the tests will get run even if testers are routinely not
enabling the resource.

test_xpickle currently takes over three minutes on my MacBook Pro to
test compatibility with Python 2.4, 2.5 and 2.6, and adding tests for
the pickle module will at least double that, if not push it well above
10 minutes.

Ouch! A couple of minutes is already too much for a single test IMHO. I
fully agree with the desire to test as deeply as possible, but if all
modules were to become as well tested, nobody would bother running the
regression suite anymore :-)

Raymond, how would you reconcile your random approach with the fact that
buildbots only display errors after a second verification run?

The buildbots should be running with the resource enabled (run the full
set of tests, every time).

If we need the reproducible results with the resource disabled, then a
random seed (different for each successful run) can be saved in a
logfile so that a particular pattern of tests can be re-run.

The idea to run some random tests when the resource flag is disabled
provides more coverage than not running any tests at all. This was very
helpful during the early development of the decimal module.

Updated the patch to include a regrtest xpickle resource, which
restricts if the backwards compat tests will be run. For normal
development, the existing tests should be fine; you only need these
tests if you're mucking with the pickle output.

Added backwards compat tests for the pure-Python pickle module. Total
run time with -uxpickle maxes out a little above five minutes on my
machine, with a lot of variation between runs.

Also addressed the comment about [0, 1, 2] -> protocols.

If no-one has any objections to the xpickle resource included in the
latest version of the patch, I'd like to commit this soon so that we can
be more confident in the other changes I have queued up. If I no-one
objects, I'll commit this sometime early tomorrow morning PDT.

If no-one has any objections to the xpickle resource included in the
latest version of the patch, I'd like to commit this soon so that we can
be more confident in the other changes I have queued up. If I no-one
objects, I'll commit this sometime early tomorrow morning PDT.

Ok if it doesn't take too long to run the tests (which may imply
implementing something like Raymond's suggestion of randomizing test
order, if you haven't already done so).

Thanks for your work,

cheers

Antoine.

Ok if it doesn't take too long to run the tests (which may imply
implementing something like Raymond's suggestion of randomizing test
order, if you haven't already done so).

I did something similar: if you don't pass the -uxpickle flag to
regrtest, only a small subset of the tests will be run, which takes less
than half a second on my machine; if you do pass -uxpickle, the full
battery of tests will run, which takes ~5 minutes.

Collin, can you port these new tests to Py3k?

Yes, I'm porting them. I got started, but got distracted by other things
since there were a lot of conflicts in the merge

Ok. Thank you!

Committed as r71408 (trunk) and r71638 (py3k).