classification
Title: unpickling of a datetime object in 3.5 fails when pickled with 2.7
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: datetime.__setstate__ fails decoding python2 pickle
View: 22005
Assigned To: Nosy List: belopolsky, josh.r, p-ganssle, serhiy.storchaka, vadimf, xtreak
Priority: normal Keywords:

Created on 2019-04-01 23:02 by vadimf, last changed 2019-04-02 10:48 by josh.r. This issue is now closed.

Files
File name Uploaded Description Edit
pickle_unpickle.tar.gz vadimf, 2019-04-01 23:02 tgz with the source files needed to reproduce the problem
Messages (3)
msg339308 - (view) Author: Vadim (vadimf) Date: 2019-04-01 23:02
Unpickling fails when pickling is performed with 2.7 (pickledatetime.py) and unpickling is done with 3.5 (Tested on Ubuntu 16.04)
Please see detailed error description and workaround in the comments to the attached files.
msg339325 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-04-02 09:46
This seems to have been fixed with 8452ca15f41061c8a6297d7956df22ab476d4df4. I checked out your example locally and renamed pickle.py to another name to avoid module collision. Ran the below commands to pickle using Python 2 and to unpickle from master before and after the relevant commit. The fix is present in 3.7 and 3.6.

➜  bpo36499 python2 pickledatetime.py

# with 8452ca15f41061c8a6297d7956df22ab476d4df4
➜  bpo36499 ../cpython/python.exe unpickledatetime.py
Pickle:  {'timenode': datetime.datetime(2019, 4, 2, 15, 13, 2, 675110)}

# with 8452ca15f41061c8a6297d7956df22ab476d4df4~1
➜  bpo36499 ../cpython/python.exe unpickledatetime.py
Traceback (most recent call last):
  File "unpickledatetime.py", line 22, in <module>
    main()
  File "unpickledatetime.py", line 18, in main
    pkl = unpickler.load()
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/pickle.py", line 1081, in load
    dispatch[key[0]](self)
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/pickle.py", line 1429, in load_reduce
    stack[-1] = func(*args)
TypeError: an integer is required (got type str)

# pickledatetime.py

import io
import pickle
from datetime import datetime


def main():
    """
    Uses python 2.7 to create a pickle file with a datetime object inside a dictionary.
    """
    data = {}
    data['timenode'] = datetime.now()
    with io.open('written_by_py27.pickle', 'wb') as handle:
        pickle.dump(data, handle, protocol=pickle.HIGHEST_PROTOCOL)

if __name__ == "__main__":
    main()

# unpickledatetime.py

import io
import pickle


def main():
    """
    Attempts to unpickle a dictionary with a datatype object inside
    """
    with io.open('written_by_py27.pickle', 'rb') as handle:
        unpickler = pickle._Unpickler(handle)
        unpickler.encoding = 'latin1'
        pkl = unpickler.load()
        print("Pickle: ", pkl)

if __name__ == "__main__":
    main()
msg339328 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-04-02 10:48
As noted, this is fixed in 3.6+. 3.5 is in security fix only mode ( https://devguide.python.org/#status-of-python-branches ), which is why the fix for #22005 wasn't backported.
History
Date User Action Args
2019-04-02 10:48:52josh.rsetstatus: open -> closed

superseder: datetime.__setstate__ fails decoding python2 pickle

nosy: + josh.r
messages: + msg339328
resolution: duplicate
stage: resolved
2019-04-02 09:46:38xtreaksetnosy: + serhiy.storchaka, xtreak
messages: + msg339325
2019-04-01 23:14:00xtreaksetnosy: + belopolsky, p-ganssle
2019-04-01 23:02:43vadimfcreate