classification
Title: datetime object fails to restore from reduction
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: belopolsky Nosy List: alexandre.vassalotti, belopolsky, jaraco, ned.deily, python-dev, serhiy.storchaka, tim.peters, vstinner
Priority: normal Keywords: patch

Created on 2016-11-20 15:55 by jaraco, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Files
File name Uploaded Description Edit
datetime-reduce.patch serhiy.storchaka, 2016-11-20 17:09 review
Pull Requests
URL Status Linked Edit
PR 552 closed dstufft, 2017-03-31 16:36
Messages (13)
msg281278 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-11-20 15:55
On Python 3.5, the datetime would reduce and restore cleanly.

$ python3.5 -c "import datetime; func, params = datetime.datetime.now().__reduce__(); func(*params)"

With Python 3.6.0b3, it now fails with a TypeError.

$ python3.6 -c "import datetime; func, params = datetime.datetime.now().__reduce__(); func(*params)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: an integer is required (got type bytes)
msg281279 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-11-20 15:57
Pickle still works, so pickle must be relying on a different protocol for serialization.

$ python
Python 3.6.0b3 (v3.6.0b3:8345e066c0ed, Oct 31 2016, 18:05:23) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import datetime
>>> pickle.loads(pickle.dumps(datetime.datetime.now()))
datetime.datetime(2016, 11, 20, 10, 56, 43, 264436)
msg281280 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-20 16:10
Now pickling of the datetime.datetime objects is implemented with __reduce_ex__. __reduce__ is not defined in datetime.datetime and is inherited from datetime.date.

>>> datetime.datetime.__reduce_ex__
<method '__reduce_ex__' of 'datetime.datetime' objects>
>>> datetime.datetime.__reduce__
<method '__reduce__' of 'datetime.date' objects>

__reduce_ex__ has higher priority and is called instead of __reduce__. It is weird that __reduce_ex__ and __reduce__ are not consistent. __reduce__ should be defined as

    def __reduce__(self):
        return self.__reduce_ex__(2)

or just

    __reduce__ = object.__reduce__

Other way is to define __reduce_ex__ instead of __reduce__ in datetime.date.
msg281284 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-20 17:09
Proposed patch restores the __reduce__() methods and makes Python and C implementations more consistent.
msg281287 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-11-20 17:19
> On Nov 20, 2016, at 11:10 AM, Serhiy Storchaka <report@bugs.python.org> wrote:
> 
> Other way is to define __reduce_ex__ instead of __reduce__ in datetime.date.

I would prefer this solution.
msg281288 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-20 17:34
> Other way is to define __reduce_ex__ instead of __reduce__ in datetime.date.

Sorry, I was wrong. This would wouldn't work with C implementation. And explicitly setting __reduce__ = object.__reduce__ doesn't work. The only way is to define both __reduce_ex__ and __reduce__ for time and datewtime.
msg281292 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-11-20 18:46
> On Nov 20, 2016, at 12:34 PM, Serhiy Storchaka <report@bugs.python.org> wrote:
> 
> The only way is to define both __reduce_ex__ and __reduce__ for time and datewtime.

OK. I'll review your patch and get it committed shortly.
msg281388 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-11-21 19:18
The patch LGTM.  I'll commit it tonight unless Serhiy beats me to it.
msg281395 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-11-21 22:25
If you can push this in the next hour or two, it can still make b4.
msg281396 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-11-21 22:28
> If you can push this in the next hour or two ...

I'll do it now.
msg281397 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-11-21 22:30
New changeset 0a2a0061e425 by Serhiy Storchaka in branch '3.6':
Issue #28752: Restored the __reduce__() methods of datetime objects.
https://hg.python.org/cpython/rev/0a2a0061e425

New changeset 23140bd66d86 by Serhiy Storchaka in branch 'default':
Issue #28752: Restored the __reduce__() methods of datetime objects.
https://hg.python.org/cpython/rev/23140bd66d86
msg281399 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2016-11-21 22:34
It looks like Serhiy has already committed it.  Thanks!
msg281521 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2016-11-22 21:53
Thanks all! So pleased to see this fixed.
History
Date User Action Args
2017-03-31 16:36:16dstufftsetpull_requests: + pull_request906
2016-11-22 21:53:21jaracosetmessages: + msg281521
2016-11-21 22:35:22belopolskysetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2016-11-21 22:34:59belopolskysetmessages: + msg281399
2016-11-21 22:30:46python-devsetnosy: + python-dev
messages: + msg281397
2016-11-21 22:28:02belopolskysetmessages: + msg281396
2016-11-21 22:25:50ned.deilysetmessages: + msg281395
2016-11-21 19:34:08serhiy.storchakasetnosy: + ned.deily
2016-11-21 19:18:15belopolskysetnosy: + tim.peters, vstinner, alexandre.vassalotti
messages: + msg281388
2016-11-21 19:12:28belopolskysetassignee: belopolsky
stage: patch review -> commit review
2016-11-20 18:46:45belopolskysetmessages: + msg281292
2016-11-20 17:34:51serhiy.storchakasetmessages: + msg281288
2016-11-20 17:19:31belopolskysetmessages: + msg281287
2016-11-20 17:09:22serhiy.storchakasetfiles: + datetime-reduce.patch
keywords: + patch
messages: + msg281284

stage: patch review
2016-11-20 16:10:56serhiy.storchakasetnosy: + serhiy.storchaka, belopolsky
messages: + msg281280

components: + Extension Modules
type: behavior
2016-11-20 15:57:38jaracosetmessages: + msg281279
2016-11-20 15:55:23jaracocreate