classification
Title: memory bloat in binary file upload (PR#381)
Type: enhancement Stage:
Components: Library (Lib) Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: barry Nosy List: barry, naris
Priority: normal Keywords:

Created on 2000-07-31 21:13 by anonymous, last changed 2000-11-06 18:39 by barry. This issue is now closed.

Messages (5)
msg357 - (view) Author: Nobody/Anonymous (nobody) Date: 2000-07-31 21:13
Jitterbug-Id: 381
Submitted-By: naris@ensim.com
Date: Mon,  3 Jul 2000 21:29:25 -0400 (EDT)
Version: 1.5.2
OS: RedHat 6.1


read_lines_to_outerboundary chews up memory.
there's a while (1) loop that does a readline(), which
is probably not good, as you could have a binary file that
doesn't have an endline til the end, and thus readline would
read the entire file into memory.  i did the following:
dd'd a 100MB /dev/zero file and used zope to upload it.
it chewed up a large amount of memory.

there seems to be bug reports about text files and windows
systems (and needing to use python -u), but this problem
is unrelated.  i am using a unix browser -> unix web server.



====================================================================
Audit trail:
Tue Jul 11 08:24:22 2000	guido	moved from incoming to open
msg358 - (view) Author: Naris Siamwalla (naris) Date: 2000-08-24 19:38
there's a bug i submitted to zope that deals with a second problem
with read_lines_to_outerboundary, which will also cause a memory
leak.

the read_lines_to_outerboundary has an unnecessary line 
self.lines.append(line).  removing will fix it this second problem.
msg359 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2000-09-25 22:06
Agreed that readlines() isn't the best thing to use in the face of large binary uploads.  I've added this feature request to PEP 42 (and am closing this bug report).

Removing self.lines might be more problematic, since although it's not documented as part of the public interface for FieldStorage, it is possible that existing code is using it, and that code would break.
msg360 - (view) Author: Naris Siamwalla (naris) Date: 2000-09-27 10:38
i feel the self.lines.append() is is a pretty serious problem...have you got any
second opinions on how to proceed?

if its not part of the public interface for FieldStorage, then wouldn't it be somewhat safe to remove it (you might anger some people, but they could always stick to using the old cgi.py)?

my argument is that if they changed the socket connect/bind functions to reject connect(host,port) in favor of connect((host,port)), then we should certainly be able to remove this self.lines.append() :-)

what do you think?
msg361 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2000-11-06 18:39
We've decided to remove the self.lines attribute for Python 2.1.  This part of the bug report is identical to 119806.  No one can remember why self.lines was there in the first place, so we'll remove it and see who complains.
History
Date User Action Args
2000-07-31 21:13:00anonymouscreate