cgi.FieldStorage uses fp.readline(1 << 16) to read in POSTed file data if no content length has been specified. All HTTP clients I've looked at terminate the file body with CRLF and then the final MIME boundary. If the file body is 65,535 bytes, and doesn't contain \n or \r\n, then fp.readline(1 << 16) will return the original 65,535 bytes of the file plus the \r from the final \r\n sequence before the final boundary string. Since \r isn't considered a line ending, it gets considered as part of the POSTed file data, and you end up with an extra \r at the end of the file data.

This is a possible fix to this issue. It's not as clean as I'd like, but the simpler versions I tried could end up with the entire file contents in memory for degenerate (or malicious) inputs.

The trick is handling the case where the current line ends with \r. We can't know if this is just a normal character in the file, or represents the end of a line until we see the start of the next line.

To demonstrate how to hit this in a real use case, run the attached script which implements a simple http server that saves POSTed files to a local file "got_data". It returns the sha1sum of the POSTed file as the http response.

Then, create a test file consisting of 65,535 null bytes, and submit with curl.

-> % dd if=/dev/zero of=data bs=1 count=65535                                                                                           
65535+0 records in
65535+0 records out
65535 bytes (66 kB) copied, 0.0890487 s, 736 kB/s

-> % sha1sum data
391edab7225a1de662ebc3a1a670a20d8e6a226b  data

-> % curl -Fdata=@data http://localhost:8080/
8dd623ef130a8cd3e97086101a6e1255a91fb916%

Thank you for your report, but your patch looks overcomplicated, it fails on 'x'*65535+'\r'+'y'*65535 and hangs on 'x'*65535+'\r'.

Here is a simpler patch with tests.

Thanks, your patch is definitely much simpler!

I was worried about the case where you have interrupted \r\n that appears in the middle of the content. But that case is handled by the next readline(), which returns a single \n.

One question about the tests you've attached - would it be better to be explicit about the line endings in check()? Do triple quoted strings in python always use \n for EOL regardless of the source code EOL format?

> Do triple quoted strings in python always use \n for EOL regardless of the source code EOL format?

Python parser always interprets EOL as \n in string literals.

New changeset 63058453a4cc by Serhiy Storchaka in branch '2.7':
Issue #18167: cgi.FieldStorage no more fails to handle multipart/form-data
http://hg.python.org/cpython/rev/63058453a4cc

New changeset a48f65bac986 by Serhiy Storchaka in branch '3.3':
Issue #18167: cgi.FieldStorage no more fails to handle multipart/form-data
http://hg.python.org/cpython/rev/a48f65bac986

New changeset 17ec73a3a854 by Serhiy Storchaka in branch 'default':
Issue #18167: cgi.FieldStorage no more fails to handle multipart/form-data
http://hg.python.org/cpython/rev/17ec73a3a854