Issue $19219 added new tokens making marshal format smaller and faster.
This patch adds two new tokens:
TYPE_SHORT_REF for which the ref index is a byte and
TYPE_VERSION for which the operand is the protocol version.

The former helps because it catches common singletons such as 0, 1, () and so on which typically show up early in a pickle.  they then need only two bytes to encoded.
This shrinks the code for the decimal.py module from 172K to 162K.


The second can help break backwards compatibility requirements in the future.  The format (if 4 or larger) is now put into the stream, so that future new formats can re-assign opcodes if needed.

I don't reassign the version number, leaving it at the new value of 4.  This change is still backwards compatible with the previous '4' so there should be no problem.

For size / performance comparison, try:
python.exe -m timeit -s "import decimal; c=compile(open(decimal.__file__).read(), decimal.__file__, 'exec'); import marshal; d=marshal.dumps(c); print(len(d))" "marshal.loads(d)"

Let defer version token until it be needed.

Good idea to add a version. But it should not be a token, but a mandatory header at the beginning. For example, 16 unsigned bits at the beginning.

Many file formats use a "magic" key, like "MZ" for Windows executable or "GIF" for GIF pictures. What do you think of adding such magic string (if it does not exist yet, I don't know the marshal format... is it documented somewhere?).

I actually agree with Kristjan that an opcode is the least disruptive
choice here. I also agree it's useful if we want to be able to evolve
the protocol without being tied by backwards compatibility constraints.

"I actually agree with Kristjan that an opcode is the least disruptive
choice here."

Does you mean that data serialized with python 3.3 can be read with
python 3.4? But not the opposite (version token unknown in Python
3.3)?

> "I actually agree with Kristjan that an opcode is the least disruptive
> choice here."
> 
> Does you mean that data serialized with python 3.3 can be read with
> python 3.4? But not the opposite (version token unknown in Python
> 3.3)?

Yes, indeed. I'm not sure it's very important but it's safer in case
people have old "frozen" modules around.

Unlike pickle, the marshal module makes no promises about keeping the format consistent between Python versions.

The version token needed only when we want break backward compatibility (change the meaning of existing opcodes).

> The version token needed only when we want break backward
> compatibility (change the meaning of existing opcodes).

This is true. For example, my first hunch was to make w_long()
emit variable-length data, (0-254: one byte integer, 255: four
bytes integer following the 255 prefix).

Right, the idea of the version token is to introduce it now, as early as possible, even if it is not needed, for prudence.
For example, if version 5 decides to change the semantics of some of the opcodes, we can then support both kinds, in the future.  Read old files _and_ the new ones.

Despite the fact that we claim that we don't guarantee interoperability, in reality it is very desirable.  During development, for instance. Frozen modules, .pyc files, and all that.

Should we support serialization in an older version? For example,
disable the new SMALL and ASCII tokens?

> Should we support serialization in an older version? For example,
> disable the new SMALL and ASCII tokens?

It is officially supported with the "version" parameter:
http://docs.python.org/3.4/library/marshal.html#marshal.dump

We have done so previously and should continue to do that.  Thanks for pointing out that the new SHORT_REF needs that fix :)

> Right, the idea of the version token is to introduce it now, as early as possible, even if it is not needed, for prudence.

How it will help? If we will break backward compatibility in version 10, then we will introduce the version token for version 10 and larger. If the file will not contain the version token, then it will compatible with version 9.

Only output TYPE_SHORT_REF for version >= 4

Quoting myself: "for prudence."
We probably should have had this from the beginning.  Adding this now makes it easier to make such changes in future, because then you don't have to re-invent the versioning mechanism.

What is difference between introducing the version token right now or only when (if?) we will need it?

Never put off to tomorrow what you can do today :)

Largest modules in the stdlib have up to 800 references. So it would be worth to have 4 opcodes for short (10-bit) ref indices.