>>> s = pickle.dumps(timezone.utc)
>>> pickle.loads(s)
Traceback (most recent call last):
 ..
TypeError: ("Required argument 'offset' (pos 1) not found", <class 'datetime.timezone'>, ())

Python version is fixed in sandbox and committed in r82154.

Committed in r82184.  Leaving the issue open pending a more thorough review of pickling in datetime module.

The datetime module provides compact pickled representation for date, datetime, time and timedelta instances:

type: size
date: 34
datetime: 44
time: 36
timedelta: 37


On the other hand, current pickle size for timezone is 64 and the size of an aware datetime instance is 105 bytes.

Since stability is important for pickle format, the best format should be developed before the first release to include timezone class.

As part of this, we should ensure references to common timezones, like
UTC, only create references to a single instance rather than filling
memory with multiple instances.

One consequence of this is that shared instances should probably be immutable.

I am attaching a python prototype implementing interned UTC instance pickling.  The patch is against sandbox revision r82218 of datetime.py.

Note that the pickling protocol requires that an instance or factory function is defined at the module level.

The pickle size saving is substantial:


>>> len(dumps(datetime.now(timezone.utc)))
61
>>> len(dumps(datetime.now(timezone.min)))
163

but there is still room for improvement:

>>> len(dumps(datetime.now()))
44

I do feel, however, that further improvements will see diminishing returns.

Your latest patch doesn't have a review link. Would you like to regenerate it against the latest default?

I updated the patch.  Please note that it only includes python code, so you need to suppress _datetime acceleration to test:

>>> import sys
>>> sys.modules['_datetime'] = None
>>> from pickle import *
>>> from datetime import *
>>> dumps(timezone.utc)
b'\x80\x03cdatetime\n_utc\nq\x00.'

Hmm.  Still no "review" link.  Is there something special that I need to do to get it?  Use different name, maybe?

FWIW, I see review links for both of your files: all the way over to the right in the Files section.

> I see review links for both of your files

Now I do too.  I guess they just take time to appear.  Note that I unlinked the file that Antoine complained about.

For stability it is better to use public name 'timezone.utc' instead of '_utc'. Dotted names now are supported with all protocols.

> but there is still room for improvement:

Don't worry about this. When you pickle a number of datetime objects with the same timezone, the timezone is pickled only once and for all other datetime objects only a reference is used.

> For stability it is better to use public name 'timezone.utc' instead of '_utc'.

Can you elaborate on the "stability" consideration?

I would like to revisit this issue since we will have some changes made to datetime pickles in the context of PEP 495.

For now "_utc" is not a part of API. When use it in pickling, you implicitly make it a part of API, and should support it forever (and force other Python implementations to support it). As far as it looks as implementation detail, there is a risk that the "_utc" variable can be renamed and this will break pickle compatibility.

> there is a risk that the "_utc" variable can be renamed
> and this will break pickle compatibility.

I think we will still need to maintain _utc global indefinitely to keep the old pickles readable.

On the other hand, it looks like support for qualnames is only available with pickle protocol 4, so I don't see any downside from from storing the full qualname other than an 8-byte increase in the size of the pickle.

I guess my position on this will be +0:  I'll accept a patch if someone will submit it, but I am unlikely to do it myself.

> I think we will still need to maintain _utc global indefinitely to keep the old pickles readable.

We have no need to maintain _utc global indefinitely if don't add it.

> On the other hand, it looks like support for qualnames is only available with pickle protocol 4,

No, support for qualnames now is available with all pickle protocols.

Following patch adds tests for timezone pickling and copying. Note, that timezone.utc identity is preserved.

New changeset e48da8f01316 by Serhiy Storchaka in branch '3.4':
Issue #9051: Added tests for pickling and copying the timezone objects.
https://hg.python.org/cpython/rev/e48da8f01316

New changeset 634905e9628d by Serhiy Storchaka in branch '3.5':
Issue #9051: Added tests for pickling and copying the timezone objects.
https://hg.python.org/cpython/rev/634905e9628d

New changeset 9c31544d6a66 by Serhiy Storchaka in branch 'default':
Issue #9051: Added tests for pickling and copying the timezone objects.
https://hg.python.org/cpython/rev/9c31544d6a66

>>> import pickle, pickletools, datetime
>>> len(pickle.dumps([datetime.datetime.utcfromtimestamp(i) for i in range(10000)], 3)) / 10000
30.9283
>>> len(pickle.dumps([datetime.datetime.fromtimestamp(i, datetime.timezone.utc) for i in range(10000)], 3)) / 10000
32.936
>>> len(pickle.dumps([datetime.datetime.utcfromtimestamp(i) for i in range(10000)], 4)) / 10000
19.0074
>>> len(pickle.dumps([datetime.datetime.fromtimestamp(i, datetime.timezone.utc) for i in range(10000)], 4)) / 10000
21.0144

In best case the UTC timezone costs only 2 bytes per datetime instance (BINGET + 1-byte index). In worst case it can cost 5 bytes (LONG_BINGET + 4-bytes index). Saving few bytes for single timezone instance has significant effect only for small pickled data. But in this case it is more advantageous to save a datetime just as a timestamp. I suggest to close this issue.