classification
Title: Inconsistency in cgi.FieldStorage() causes unicode/byte TypeError.
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: cgi.FieldStorage can't parse multipart part headers with Content-Length and no filename in Content-Disposition
View: 24764
Assigned To: Nosy List: Marcel Hellkamp, X-Istence, berker.peksag
Priority: normal Keywords: patch

Created on 2016-06-13 18:45 by Marcel Hellkamp, last changed 2016-09-27 03:38 by X-Istence. This issue is now closed.

Files
File name Uploaded Description Edit
foo.patch Marcel Hellkamp, 2016-06-13 19:23 review
Messages (5)
msg268460 - (view) Author: Marcel Hellkamp (Marcel Hellkamp) * Date: 2016-06-13 18:45
Discovered here: https://github.com/bottlepy/bottle/issues/856

If a multipart section has a "Content-Length" header, but no "filename" attribute in the "Content-Disposition" header, cgi.FieldStorage tries to write binary data to a temporary file opened in text-mode.

The problem here is that cgi.FieldStorage tries to decide if something is a binary file-upload or a unicode form-field, but it does so based on two different headers in two different places. If the headers contradict each other (form-fields usually don't have a Content-Length, file-uploads usually have a filename), parsing breaks with a TypeError.

Unfortunately, there are some HTTP client libraries out there that trigger this bug.



Here is what happens:

A "Content-Length" header causes `cgi.FieldStorage.length` to be is set (which is fine).
https://hg.python.org/cpython/file/3.4/Lib/cgi.py#l550

If `length` has a value, `read_binary()` is used instead of `read_lines()` (which is questionable).
https://hg.python.org/cpython/file/3.4/Lib/cgi.py#l733

`read_binary()` calls `make_file()` which creates the buffer file in text mode if it does not find a `filename` attribute in the "Content-Disposition" Header (which is somewhat okay).
https://hg.python.org/cpython/file/3.4/Lib/cgi.py#l515
https://hg.python.org/cpython/file/3.4/Lib/cgi.py#l893

The bug is triggered if the last two steps disagree on the bytes vs. text question.
msg268462 - (view) Author: Marcel Hellkamp (Marcel Hellkamp) * Date: 2016-06-13 19:23
This should fix the issue.
msg268466 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-06-13 19:28
Thanks for the patch. From a quick look, the patch looks fine but we need a test case. You can see Lib/test/test_cgi.py for existing tests.
msg268468 - (view) Author: Marcel Hellkamp (Marcel Hellkamp) * Date: 2016-06-13 19:57
Looks like this is a duplicate to #24764 and already fixed. Sorry for the noise.
msg277481 - (view) Author: Bert JW Regeer (X-Istence) * Date: 2016-09-27 03:38
This is not a duplicate of https://bugs.python.org/issue24764
History
Date User Action Args
2016-09-27 03:38:23X-Istencesetnosy: + X-Istence
messages: + msg277481
2016-06-13 20:03:47berker.peksagsetstatus: open -> closed
superseder: cgi.FieldStorage can't parse multipart part headers with Content-Length and no filename in Content-Disposition
resolution: duplicate
stage: patch review -> resolved
2016-06-13 19:57:52Marcel Hellkampsetmessages: + msg268468
2016-06-13 19:28:16berker.peksagsetmessages: + msg268466
stage: needs patch -> patch review
2016-06-13 19:23:05Marcel Hellkampsetfiles: + foo.patch
keywords: + patch
messages: + msg268462
2016-06-13 19:03:53berker.peksagsetnosy: + berker.peksag
stage: needs patch
type: crash -> behavior

versions: + Python 3.6, - Python 3.4
2016-06-13 18:46:07Marcel Hellkampsettitle: Inconsistency in cgi.FieldStorage() causes unicode/byte issue. -> Inconsistency in cgi.FieldStorage() causes unicode/byte TypeError.
2016-06-13 18:45:31Marcel Hellkampcreate