classification
Title: cgi handling of POSTed files is broken
Type: behavior Stage: test needed
Components: Library (Lib) Versions: Python 3.2, Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: MLModel, georg.brandl, ggenellina, gvanrossum, orsenthil, quentel, r.david.murray, vstinner
Priority: normal Keywords: patch

Created on 2010-03-06 01:43 by MLModel, last changed 2012-05-03 08:43 by orsenthil.

Files
File name Uploaded Description Edit
cgi-post-broken.zip MLModel, 2010-03-06 01:43 zipped demo of problem
http-server.patch quentel, 2012-05-03 07:54 Patch for http.server.CGIHTTPRequestHandler review
Messages (5)
msg100509 - (view) Author: Mitchell Model (MLModel) Date: 2010-03-06 01:43
I am reluctant to post this because (a) I might have made some dumb mistake in my code, simple as it is, and (b) the problem might be well-known or even hopeless because the cgi module may not be WSGI compliant, but if this isn't going to be fixed then at least the problem should be described in the cgi module documentation.

I run an HTTPServer with a CGIHTTPRequestHandler.
I open an HTML file with a trivial form that simply POSTs an upload of a single file.
I browse, select a file, and click submit.
The web request never completes -- the browser just waits for a response.
Interrupting the server with ^C^C produces a backtrace that indicates it is stuck trying to decode the file contents.

The attached zip file contains:
    cgi-server.py
    cgi-post.html
    cgi-bin/cgi-test.py
    exception.txt
The latter is the backtrace output from when I interrupt the server.

This is on OS X 10.5 with either Python3.1 or Python3.2 from the repository.
msg101211 - (view) Author: Gabriel Genellina (ggenellina) Date: 2010-03-17 02:05
This doesn't look like a documentation bug to me - handling of uploaded files via CGI *should* work, even if CGI is not the best way to do that.
msg107405 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-06-09 16:57
The example works for me if I make this change:

--- Lib/cgi.py	(revision 81862)
+++ Lib/cgi.py	(working copy)
@@ -608,7 +608,7 @@
         parser = email.parser.FeedParser()
         # Create bogus content-type header for proper multipart parsing
         parser.feed('Content-Type: %s; boundary=%s\r\n\r\n' % (self.type, ib))
-        parser.feed(self.fp.read())
+        parser.feed(self.fp.read(self.length))
         full_msg = parser.close()
         # Get subparts
         msgs = full_msg.get_payload()


However this seems iffy to me because the content length presumably counts bytes whereas self.fp seems to be a text file, but since most HTTP clients don't close the connection, without some kind of boundary on the read() call it just hangs forever.

Also someone pointed out to me offline that this change may be needed, separately (though I haven't confirmed this yet):

--- Lib/cgi.py	(revision 81862)
+++ Lib/cgi.py	(working copy)
@@ -233,6 +233,7 @@
         lines = []
         while 1:
             line = fp.readline()
+            line = line.decode()
             if not line:
                 terminator = lastpart # End outer loop
                 break
msg107960 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2010-06-16 22:25
Could this be related to issue 4953?
msg159838 - (view) Author: Pierre Quentel (quentel) * Date: 2012-05-03 07:54
There are 2 different problems :
- handling of data by cgi.FieldStorage (issue 4953) : fixed since version 3.2
- in http.server.CGIHTTPRequestHandler, for POST requests on Windows, before opening the subprocess in run_cgi() all data is read by a *single* call to self.rfile.read(). If not all bytes are read by this single call, which is the case for a large file upload, only the read data are processed

The attached patch modifies http.server :
- if all data are read in the first call to self.rfile.read() (that is, if its length is equal to the Content-length header), process them
- if not, store all data in a temporary file (not in memory) and set the stdin argument of subprocess.Popen to this temporary file

With this patch, the tests provided by Mitchell all work on my PC (Windows XP Pro SP3)
History
Date User Action Args
2012-05-03 08:43:50orsenthilsetassignee: orsenthil

nosy: + orsenthil
2012-05-03 07:54:27quentelsetfiles: + http-server.patch
keywords: + patch
messages: + msg159838
2012-05-02 19:55:24quentelsetnosy: + quentel
2012-02-24 10:46:16ezio.melottisetstage: test needed
versions: + Python 3.3, - Python 3.1
2011-07-04 01:19:49vstinnersetnosy: + vstinner
2010-09-16 02:42:41r.david.murraysetnosy: + r.david.murray

title: urlparse -> cgi handling of POSTed files is broken
2010-09-15 21:41:09ncoghlansettitle: cgi handling of POSTed files is broken -> urlparse
2010-07-29 13:58:22georg.brandlsetassignee: georg.brandl -> (no value)
2010-06-16 22:25:33gvanrossumsetmessages: + msg107960
2010-06-09 16:57:13gvanrossumsetnosy: + gvanrossum
messages: + msg107405
components: - Documentation
2010-03-17 02:05:49ggenellinasetnosy: + ggenellina
messages: + msg101211
2010-03-06 01:43:39MLModelcreate