Author gromgull
Recipients gromgull
Date 2014-02-03.14:27:33
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1391437653.41.0.935040025319.issue20501@psf.upfronthosting.co.za>
In-reply-to
Content
When reading large files with fileinput, it will work as expected and only process a line at a time when used normally, but if you add an hook_encoded openhook it will read the whole file into memory before returning the first line. 

Verify by running this program on a large text file: 

import fileinput

for l in fileinput.input(openhook=fileinput.hook_encoded('iso-8859-1')):
    raw_input()

and check how much memory it uses. Remove the openhook and memory usage goes down to nothing.
History
Date User Action Args
2014-02-03 14:27:33gromgullsetrecipients: + gromgull
2014-02-03 14:27:33gromgullsetmessageid: <1391437653.41.0.935040025319.issue20501@psf.upfronthosting.co.za>
2014-02-03 14:27:33gromgulllinkissue20501 messages
2014-02-03 14:27:33gromgullcreate