classification
Title: Restore loading of TYPE_INT64 in marshal
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: serhiy.storchaka
Priority: normal Keywords: patch

Created on 2017-11-12 17:50 by serhiy.storchaka, last changed 2017-11-15 16:06 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4381 merged serhiy.storchaka, 2017-11-12 18:50
PR 4405 merged python-dev, 2017-11-15 15:41
Messages (4)
msg306128 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-12 17:50
TYPE_INT64 is supported in the marshal module since Python 1.5. Since its use causes instability of marhalled files produced on different platforms, the marshal module in Python 3.3 no longer use it when write new files (see issue15466). And in Python 3.4 it stopped reading files containing the TYPE_INT64 code (see issue15480).

This is backward incompatible change. Python 2.7 still produce files containing the TYPE_INT64 code (though this code is rare, because integers in ranges -9223372036854775808..-2147483649 and 2147483648..9223372036854775807 are rare), and there may be marshal files produced by older versions of Python 3, Python 2 and Python 1. Formally Python 3 supports marshal formats produced by all previous versions, except the code objects and the TYPE_INT64 code.

Supporting loading the TYPE_INT64 code don't create problems like using it for saving data. But it is needed for backward compatibility.

I think that removing the TYPE_INT64 code was a mistake.
msg306130 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-12 18:54
Example:

$ python2 -c 'import sys, marshal; marshal.dump(1234567890, sys.stdout)' | python3 -c 'import sys, marshal; print(marshal.load(sys.stdin.buffer))'
1234567890

$ python2 -c 'import sys, marshal; marshal.dump(123456789012345, sys.stdout)' | python3 -c 'import sys, marshal; print(marshal.load(sys.stdin.buffer))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: bad marshal data (unknown type code)

 python2 -c 'import sys, marshal; marshal.dump(12345678901234567890, sys.stdout)' | python3 -c 'import sys, marshal; print(marshal.load(sys.stdin.buffer))'
12345678901234567890

PR 4381 reverts this change and simplifies the restored code.
msg306276 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-15 15:41
New changeset 00987f6230fcdbecc8d9ab4b2b9fae8f99a1a4a9 by Serhiy Storchaka in branch 'master':
bpo-32011: Revert "Issue #15480: Remove the deprecated and unused TYPE_INT64 code from marshal." (#4381)
https://github.com/python/cpython/commit/00987f6230fcdbecc8d9ab4b2b9fae8f99a1a4a9
msg306281 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-15 16:06
New changeset d15bb5fcad584e113836486d17c6abcbf2168a86 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-32011: Revert "Issue GH-15480: Remove the deprecated and unused TYPE_INT64 code from marshal." (GH-4381) (#4405)
https://github.com/python/cpython/commit/d15bb5fcad584e113836486d17c6abcbf2168a86
History
Date User Action Args
2017-11-15 16:06:19serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-11-15 16:06:00serhiy.storchakasetmessages: + msg306281
2017-11-15 15:41:13python-devsetpull_requests: + pull_request4354
2017-11-15 15:41:07serhiy.storchakasetmessages: + msg306276
2017-11-12 18:54:36serhiy.storchakasetmessages: + msg306130
2017-11-12 18:50:38serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request4329
2017-11-12 17:50:16serhiy.storchakacreate