Title: cgi.FieldStorage constructor assumes all lines terminate with \n
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ian Craggs
Priority: normal Keywords:

Created on 2018-01-12 18:41 by Ian Craggs, last changed 2018-01-12 19:37 by ned.deily.

Messages (2)
msg309868 - (view) Author: Ian Craggs (Ian Craggs) Date: 2018-01-12 18:41
Using cgi.FieldStorage in an HTTP server in a subclass of BaseHTTPRequestHandler, parsing the request with:

form = cgi.FieldStorage(fp=self.rfile,

This has been working fine with clients using the Python requests library. Now processing requests from a Java library (org.apache.cxf.jaxrs.client.WebClient), the final line in a multipart request does not include the (\r)\n, which causes the final read to hang until a socket timeout.  The read in question is in, read_lines_to_outerboundary:

line = self.fp.readline(1<<16) # bytes

(line 824 in Python 3.6.2).  I changed this read to not assume the termination of the final line with \n:

    def read_line(self, last_boundary):
        line = self.fp.readline(len(last_boundary))
        if line != last_boundary and not line.endswith(b"\n"):
            line += self.fp.readline((1<<16) - len(last_boundary))
        return line

and the request worked.  The Java library is being used in tests against our production web server so I assume that is working correctly.  

Perhaps I am misusing the FieldStorage class, I don't know, I'm not expert on this.
msg309871 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-12 19:36
I'm removing the macOS tag and nosies because I think it highly unlikely that the behavior would be limited to macOS.  Without a reproducible test case, it's not easy to verify that or investigate further.  The cgi module doesn't get a lot of attention but perhaps someone with cgi experience will take a look.
Date User Action Args
2018-01-12 19:37:08ned.deilysetnosy: - ned.deily
2018-01-12 19:36:59ned.deilysetnosy: - ronaldoussoren
messages: + msg309871
components: - macOS
2018-01-12 18:41:57Ian Craggscreate