In the attached demo I create a graph of 250 nodes, all of which are
connected to every other node, and this is represented by a set
attribute of the Node objects.

When I try to pickle this graph, it fails in various ways. In regular
pickle it is always a "maximum recursion depth exceeded" error. In
cPickle it is either a KeyError or a silent termination of the process.
(As tested on Windows XP 32 bits.) It depends on the size of the graph.
For smaller graphs (say 100 nodes) it works fine.

If connections are described using a dictionary or even a list, I get
the same errors.

So now I've learned that this is a result of the way Pickler is
implemented. I think that it would make sense to create an
implementation that is not that recursive and that would handle such
structures better.

I have now written one such implementation that is a subclass of the
original pickler and handles recursive data structures. I presume that
it is inferior in a couple of ways (performance and memory efficiency
probably), but a complete rewrite could probably create an
implementation that handles recursive data structures and is not inferior.

Sidenote: If I click "edit" for nonrecursivepickler.py, I get told that
"File has been classified as spam." Strange. Should I not use Viagra as
a classname? :) (j/k I didn't do anything like that)

However I have now fixed a mistake in my code.

I just ran into this behavior with an attempt to pickle a dom tree for
an XML document whose nesting level never got deeper than nine child
nodes, and indeed it crashed the interpreter. Throwing an exception
would be preferable, of course, to silent blowing up Python, but even
the exception seems to fly in the face of the documentation for the
pickle module [1] which claims (summarizing) that serializing recursive
objects using marshal will fail but pickling recursive objects will not
fail. I can provide a repro for the XML/DOM pickling case if you think
that would be helpful, but that seems redundant since essentially you've
already illustrated the problem with your own repro case. Thanks for
your work on the solution.

[1] http://docs.python.org/lib/node314.html

Bob Kline wrote:

I just ran into this behavior with an attempt to pickle a dom tree
for an XML document whose nesting level never got deeper than nine
child nodes, and indeed it crashed the interpreter.

Pickling recursive data-structure should not crash the interpreter.
Please open a new issue and don't forget to provide an example case.

the documentation for the pickle module 1 which claims (summarizing)
that serializing recursive objects using marshal will fail but
pickling recursive objects will not fail.

The section of documentation, you are referring to, uses the term
"recursive object" to means an object which contains a reference to
itself. Anyway, the documentation 1 states clearly:

Trying to pickle a highly recursive data structure may exceed the
maximum recursion depth, a RuntimeError will be raised in this
case. You can carefully raise this limit with sys.setrecursionlimit().

I have also described the crash, but it makes sense to handle it
separately. So I have created bpo-2702, and changed the title of this
issue.

Please open a new issue and don't forget to provide an example case.

Looks like Daniel beat me to the punch.

I've provided an alternate implementation of this that works with very
minimal modification to pickle.py. See bpo-3119 for the patch.

I am closing this issue in favour of bpo-3119, since Aaron's patch is
cleaner and slightly faster.

Thank you Daniel for the idea!