This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: cgi.FieldStorage constructor assumes all lines terminate with \n
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Ian Craggs
Priority: normal Keywords:

Created on 2018-01-12 18:41 by Ian Craggs, last changed 2022-04-11 14:58 by admin.

Messages (2)
msg309868 - (view) Author: Ian Craggs (Ian Craggs) Date: 2018-01-12 18:41
Using cgi.FieldStorage in an HTTP server in a subclass of BaseHTTPRequestHandler, parsing the request with:

form = cgi.FieldStorage(fp=self.rfile,
                        headers=self.headers,
                        environ={"REQUEST_METHOD":op.upper(),
                              "CONTENT_TYPE":self.headers['Content-Type'],})

This has been working fine with clients using the Python requests library. Now processing requests from a Java library (org.apache.cxf.jaxrs.client.WebClient), the final line in a multipart request does not include the (\r)\n, which causes the final read to hang until a socket timeout.  The read in question is in cgi.py, read_lines_to_outerboundary:

line = self.fp.readline(1<<16) # bytes

(line 824 in Python 3.6.2).  I changed this read to not assume the termination of the final line with \n:

    def read_line(self, last_boundary):
        line = self.fp.readline(len(last_boundary))
        if line != last_boundary and not line.endswith(b"\n"):
            line += self.fp.readline((1<<16) - len(last_boundary))
        return line


and the request worked.  The Java library is being used in tests against our production web server so I assume that is working correctly.  

Perhaps I am misusing the FieldStorage class, I don't know, I'm not expert on this.
msg309871 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-12 19:36
I'm removing the macOS tag and nosies because I think it highly unlikely that the behavior would be limited to macOS.  Without a reproducible test case, it's not easy to verify that or investigate further.  The cgi module doesn't get a lot of attention but perhaps someone with cgi experience will take a look.
History
Date User Action Args
2022-04-11 14:58:56adminsetgithub: 76722
2018-01-12 19:37:08ned.deilysetnosy: - ned.deily
2018-01-12 19:36:59ned.deilysetnosy: - ronaldoussoren
messages: + msg309871
components: - macOS
2018-01-12 18:41:57Ian Craggscreate