classification
Title: httplib doesn't specify content-length header for POST requests without data
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Arve.Knudsen, demian.brecht, eric.araujo, ezio.melotti, jcea, jimr, ned.deily, orsenthil, piotr.dobrogost, python-dev
Priority: normal Keywords: patch

Created on 2012-05-04 10:06 by Arve.Knudsen, last changed 2015-02-27 17:31 by demian.brecht. This issue is now closed.

Files
File name Uploaded Description Edit
httplib-2.7.patch Arve.Knudsen, 2012-05-04 19:34 Patch httplib in Python 2.7 to set 'content-length: 0' automatically review
Messages (18)
msg159915 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2012-05-04 10:06
httplib doesn't specify the HTTP header 'content-length' for POST requests without data. Conceptually this makes sense, considering the empty content. However, IIS (7.5) servers don't accept such requests and respond with a 411 status code. See this question on StackOverflow for reference: http://stackoverflow.com/questions/5915131/can-i-send-an-empty-http-post-webrequest-object-from-c-sharp-to-iis.

See also issue #223 of the Requests project, https://github.com/kennethreitz/requests/issues/223, which regards this problem in the context of the requests Python library.

The following code makes a data-less POST request to the HTTP sniffer Fiddler running on localhost:
import httplib
conn = httplib.HTTPConnection("localhost", 8888)
conn.request("POST", "/post", "", {})
conn.close()

Fiddler reports that it receives the following headers for the POST request, as you can see 'content-length' is not included:
POST http://localhost:8888/post HTTP/1.1
Host: localhost:8888
Accept-Encoding: identity
msg159935 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-05-04 14:21
Could you provide a patch?

Does this affect  3.x too?
msg159936 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2012-05-04 14:35
I can look into patch and 3.x tonight I think. Should I provide a test with an eventual patch?
msg159938 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-05-04 14:42
Patch with test, please :-).

I know it is a pain in the ass, but the result is having a higher quality python.
msg159942 - (view) Author: Piotr Dobrogost (piotr.dobrogost) Date: 2012-05-04 16:01
> Fiddler reports that it receives the following headers for the POST request

Python 3.2.3 on Windows Vista 64bit gives the same output for

import http.client
conn = http.client.HTTPConnection('localhost',8888)
conn.request("POST", "/post", "", {})
conn.close()
msg159946 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2012-05-04 17:23
Which HTTP methods should we auto-define content-length for? POST and PUT?  I noticed that my first attempt at a patch would define content-length also for GET requests, which broke a unit test (test_ipv6host_header).
msg159954 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2012-05-04 18:12
Actually, when inspecting the HTTP requests sent by Chrome for the different methods (a great little Chrome app called Postman let me fire requests manually), I found that content-length would be set for most methods. I could confirm that 'content-length: 0' would be set for the following methods: POST, PUT, PATCH, DELETE and HEAD.

I guess it should be good enough to model that behaviour in httplib?
msg159959 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-05-04 19:12
HEAD?. It doesn't make sense in HEAD if it doesn't make sense in GET.

Looking around, I found this, to mud the water a little bit more: <http://fixunix.com/tcp-ip/66198-http-rfc-related-question-content-length-0-get-request.html>.

Not being explicit about this is a "bug" in the specification, I think.
msg159964 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2012-05-04 19:34
Here's my initial proposal for a patch against httplib, based on the 2.7 branch of cpython repository. I've included a couple of tests, which check that when content is empty, content-length is set to 0 for certain methods (POST/PUT etc) and not specified for others.
msg159965 - (view) Author: Arve Knudsen (Arve.Knudsen) Date: 2012-05-04 19:34
Yes, I agree it doesn't make much sense for HEAD AFAICT, but Chrome does it. Maybe there's a reason?
msg161096 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-05-19 08:05
The rule for content-length seems, if there is a body for a request, even if the body is "" ( empty body), then you should send the Content-Length.

The mistake in the Python httplib was, the set_content_length was called with this condition.

if body and ('content-length' not in header_names):

If the body was '', this was skipped. The default for GET and methods which do not use body was body=None and that was statement for correct in those cases.

A simple fix which covers the applicable methods and follows the definition of content-length seems to me like this:

-        if body and ('content-length' not in header_names):
+        if body is not None and 'content-length' not in header_names:

I prefer this rather than checking for methods explicitly as it could go into unnecessary details. (Things like if you are not sending a body why are you sending a Content-Length?. This fails the definition of Content-Length itself). The Patch is fine, I would adopt that for the above check and commit it all the active versions.

Thanks Arve Knudsen, for the bug report and the patch.
msg161098 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-19 08:59
New changeset 57f1d13c2cd4 by Senthil Kumaran in branch '2.7':
Fix Issue14721: Send Content-length: 0 for empty body () in the http.request
http://hg.python.org/cpython/rev/57f1d13c2cd4

New changeset 6da1ab5f777d by Senthil Kumaran in branch '3.2':
Fix Issue14721: Send Content-length: 0 for empty body () in the http.client requests
http://hg.python.org/cpython/rev/6da1ab5f777d

New changeset 732d70746fc0 by Senthil Kumaran in branch 'default':
merge - Fix Issue14721: Send Content-length: 0 for empty body () in the http.client requests
http://hg.python.org/cpython/rev/732d70746fc0
msg161099 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-05-19 09:00
This is fixed in all the branches. Thanks!
msg161154 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2012-05-19 19:46
Too late for asking to keep the parenthesis :-). I hate to have to remember non-obvious precedence rules :-). Cognitive overhead.
msg236600 - (view) Author: James Rutherford (jimr) * Date: 2015-02-25 16:59
The fix for this still doesn't set Content-Length to zero when body is None, but I don't see any reason why this should be the case. For example, the following snippet would work for any 'empty' body:

if 'content-length' not in header_names:
    self._set_content_length(body if body is not None else '')

I'm happy to produce a patch if there's any chance it would be merged.
msg236686 - (view) Author: Demian Brecht (demian.brecht) * Date: 2015-02-26 16:41
> I'm happy to produce a patch if there's any chance it would be merged.

If the patch adheres to the RFC, then I see no reason why it shouldn't be merged. What makes this a little more tricky than the snippet that you included in your post though (which would include the Content-Length header for all HTTP methods) is the following from RFC 7230:

   A user agent SHOULD send a Content-Length in a request message when
   no Transfer-Encoding is sent and the request method defines a meaning
   for an enclosed payload body.  For example, a Content-Length header
   field is normally sent in a POST request even when the value is 0
   (indicating an empty payload body).  A user agent SHOULD NOT send a
   Content-Length header field when the request message does not contain
   a payload body and the method semantics do not anticipate such a
   body.

Currently, there is nothing in the http package that defines whether or not a given HTTP method expects a body (as far as I'm aware at any rate), although this would be a simple addition. I'd imagine that the result might look like this:

_METHODS_EXPECTING_BODIES = {'OPTIONS', 'POST', 'PUT', 'PATCH'}

if method.upper() in _METHODS_EXPECTING_BODIES and \
        'content-length' not in header_names:
    self._set_content_length(body)

I'd prefer to have the conversion from None to empty string done in the body of _set_content_length in order to ensure consistency should the call be made from elsewhere.
msg236731 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-02-27 08:19
James, Demian: this issue has been closed for almost three years and the changes released long ago: comments made here will likely be ignored.  Please open a new issue if you want them to be acted on.
msg236804 - (view) Author: Demian Brecht (demian.brecht) * Date: 2015-02-27 17:31
Thanks for the heads up Ned.

James: I've created #23539 in the event that you'd like to contribute a patch.
History
Date User Action Args
2015-02-27 17:31:57demian.brechtsetmessages: + msg236804
2015-02-27 08:19:28ned.deilysetnosy: + ned.deily

messages: + msg236731
versions: - Python 3.5
2015-02-26 16:41:19demian.brechtsetnosy: + demian.brecht

messages: + msg236686
versions: + Python 3.5
2015-02-25 16:59:18jimrsetnosy: + jimr
messages: + msg236600
2012-05-19 19:46:50jceasetmessages: + msg161154
2012-05-19 09:00:21orsenthilsetstatus: open -> closed
messages: + msg161099

assignee: orsenthil
resolution: fixed
stage: needs patch -> resolved
2012-05-19 08:59:16python-devsetnosy: + python-dev
messages: + msg161098
2012-05-19 08:05:01orsenthilsetmessages: + msg161096
2012-05-04 19:34:54Arve.Knudsensetmessages: + msg159965
2012-05-04 19:34:09Arve.Knudsensetfiles: + httplib-2.7.patch
keywords: + patch
messages: + msg159964
2012-05-04 19:12:29jceasetmessages: + msg159959
2012-05-04 18:12:08Arve.Knudsensetmessages: + msg159954
2012-05-04 17:23:52Arve.Knudsensetmessages: + msg159946
2012-05-04 16:37:22eric.araujosetnosy: + eric.araujo
2012-05-04 16:01:06piotr.dobrogostsetnosy: + piotr.dobrogost
messages: + msg159942
2012-05-04 15:20:49pitrousetnosy: + orsenthil
stage: test needed -> needs patch

versions: + Python 3.2, Python 3.3
2012-05-04 14:42:16jceasetmessages: + msg159938
2012-05-04 14:35:51Arve.Knudsensetmessages: + msg159936
2012-05-04 14:24:48ezio.melottisetnosy: + ezio.melotti

type: behavior
stage: test needed
2012-05-04 14:21:20jceasetnosy: + jcea
messages: + msg159935
2012-05-04 10:06:43Arve.Knudsencreate