This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: cPickle fails on large objects (still - 2011 and counting)
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: alexandre.vassalotti, robert@smithpierce.net, serhiy.storchaka, xiang.zhang
Priority: normal Keywords:

Created on 2016-08-27 19:16 by robert@smithpierce.net, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 662 merged serhiy.storchaka, 2017-03-13 21:08
Messages (5)
msg273795 - (view) Author: Robert Pierce (robert@smithpierce.net) Date: 2016-08-27 19:16
cPickle fails on large objects, throwing a SystemError exception which is cryptic. 

The issue was fixed for pickle in python 3 back in 2011 (http://bugs.python.org/issue11564), but never addressed in 2.7.  It seems to be a recurring complaint (e.g., http://bugs.python.org/issue11872), but always seems to be closed without being fixed or explained why it cannot be fixed.

Test case from 2011 still fails:

>>> import cPickle; cPickle.dumps('a' * (2 ** 31),-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: error return without exception set
msg289550 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-13 21:11
PR 662 presumably fixes this issue.

Could anyone please check that the test case is fixed? I have no enough memory for testing.
msg289563 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2017-03-14 04:13
>>> import cPickle
[45575 refs]
>>> cPickle.dumps('a' * (2 ** 31),-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: cannot serialize a string larger than 2 GiB
[45606 refs]
>>> 

:-)
msg289565 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-14 05:29
Thank you Xiang!
msg290189 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-24 22:20
New changeset 1aa1803b3df2af9481628b8896babbd686d314ff by Serhiy Storchaka in branch '2.7':
bpo-27880: Fixed integer overflow in cPickle when pickle large strings or (#662)
https://github.com/python/cpython/commit/1aa1803b3df2af9481628b8896babbd686d314ff
History
Date User Action Args
2022-04-11 14:58:35adminsetgithub: 72067
2017-03-24 22:20:37serhiy.storchakasetmessages: + msg290189
2017-03-14 05:30:23serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-03-14 05:29:08serhiy.storchakasetmessages: + msg289565
2017-03-14 04:13:04xiang.zhangsetnosy: + xiang.zhang
messages: + msg289563
2017-03-13 21:11:26serhiy.storchakasetnosy: + alexandre.vassalotti, serhiy.storchaka
messages: + msg289550

assignee: serhiy.storchaka
stage: patch review
2017-03-13 21:08:11serhiy.storchakasetpull_requests: + pull_request545
2016-08-27 19:16:24robert@smithpierce.netcreate