This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author goddard
Recipients goddard, loewis, pitrou, vstinner
Date 2009-03-25.07:02:46
SpamBayes Score 9.948153e-12
Marked as misclassified No
Message-id <1237964570.86.0.959754236313.issue5557@psf.upfronthosting.co.za>
In-reply-to
Content
I agree that having such large Python code files is a rare circumstance
and optimizing the byte-code compiler for that should be a low priority.

Thanks for the cpickle suggestion.  The Chimera session file Python code
is mostly large nested dictionaries and sequences.  I tested cPickle and
repr() to embed data structures in the Python code getting rather larger
file size because the 8-bit characters became 4 bytes in the text file
string (e.g. "\xe8").  Using cPickle, and base64 encoding dropped the
file size by about a factor of 2.5 and cPickle, bzip2 or zlib
compression, and base64 dropped the size another factor of 2.  The big
win is that the byte code compilation used 150 Mbytes and 5 seconds
instead of 2 Gbytes and 15 minutes of thrashing for a 40 Mbyte python
file.  I think our reason for not using pickled data originally in the
session files was because we like users to be able to look at and edit
the session files in a text editor.  (This is research software where
such hacks sometimes are handy.)  But the especially large data
structures in the sessions can't reasonably be meddled with by users so
pickling should be fine.  Pickling adds about 15% to the session save
time, and reduces session opening by about the same amount.  Compression
slows the save down another 15% and probably is not worth the factor of
2 reduction in file size in our case.
History
Date User Action Args
2009-03-25 07:02:51goddardsetrecipients: + goddard, loewis, pitrou, vstinner
2009-03-25 07:02:50goddardsetmessageid: <1237964570.86.0.959754236313.issue5557@psf.upfronthosting.co.za>
2009-03-25 07:02:48goddardlinkissue5557 messages
2009-03-25 07:02:46goddardcreate