classification
Title: [http.client] HTTPConnection.putrequest not support "chunked" Transfer-Encodings to send data
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Rotkraut, demian.brecht, harobed, orsenthil, petri.lehtinen, piotr.dobrogost, pitrou, whitemice
Priority: normal Keywords:

Created on 2011-06-12 10:47 by harobed, last changed 2014-08-28 13:54 by Rotkraut.

Files
File name Uploaded Description Edit
chunkedhttp.py Rotkraut, 2014-08-28 08:45 A custom module implementing upload with chunked transfer encoding for urllib
Messages (10)
msg138203 - (view) Author: harobed (harobed) Date: 2011-06-12 10:47
Hi,

HTTPConnection.putrequest not support "chunked" Transfer-Encodings to send data.

Exemple, I can't do PUT request with chunk transfert.

Regards,
Stephane
msg138242 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-06-13 13:01
What's the use case? Do you have an iterable that yields data whose size is unknown?

AFAIK, most web servers don't even support chunked uploads.

(Removing Python 2.7 from versions as this is clearly a feature request.)
msg138258 - (view) Author: harobed (harobed) Date: 2011-06-13 15:35
I use http.client in WebDAV client.

Mac OS X Finder WebDAV client perform all his request in "chunk" mode : PUT and GET.

Here, I use http.client to simulate Mac OS X Finder WebDAV client.

Regards,
Stephane
msg138296 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2011-06-14 06:15
harobed wrote:
> I use http.client in WebDAV client.
>
> Mac OS X Finder WebDAV client perform all his request in "chunk" mode : PUT and GET.
>
> Here, I use http.client to simulate Mac OS X Finder WebDAV client.

Now I'm confused. Per the HTTP specification, GET requests don't have
a body, so "Transfer-Encoding: chunked" doesn't apply to them.

Are you sure you don't confuse with the response that the server
sends? In responses, "Transfer-Encoding: chunked" is very common.
msg138357 - (view) Author: harobed (harobed) Date: 2011-06-15 07:39
> Now I'm confused. Per the HTTP specification, GET requests don't have
a body, so "Transfer-Encoding: chunked" doesn't apply to them.

> Are you sure you don't confuse with the response that the server
sends? In responses, "Transfer-Encoding: chunked" is very common.

Sorry, yes GET requests have "Transfer-Encoding: chunked" in server response.
PUT requests can send body data in transfer-encoding chunked mode.
msg138691 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2011-06-20 08:13
We had support for chunked transfer encoding for POST method recently, which is exposed via urllib2 wrapper function. PUT is not exposed via urllib2 and users should use httplib. This feature of chunked transfer can be added to PUT by taking the body of the message as iterable.
msg171268 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-09-25 13:14
> We had support for chunked transfer encoding for POST method recently, 
> which is exposed via urllib2 wrapper function.

I couldn't find what you're talking about.
If I look at AbstractHTTPHandler.do_request_, it actually mandates a Content-Length header for POST data (hence no chunked encoding).
msg226012 - (view) Author: Rolf Krahl (Rotkraut) Date: 2014-08-28 08:45
I'd like to support the request.  I have a use case where I definitely need this feature: I maintain a Python client for a scientific metadata catalogue, see [1] for details.  The client also features the upload of the data files.  The files may come in as a data stream from another process, so my client takes a file like object as input.  The files may be large (several GB), so buffering them is not an option, they must get streamed to the server as they come in.  Therefore, there is have no way to know the Content-length of the upload beforehand.

I implemented chunked transfer encoding in a custom module that monkey patches the library, see the attached file.  This works fine, but of course it's an awkward hack as I must meddle deeply into the internals of urllib and http.client to get this working.  This module is tested to work with Python 2.7, 3.1, 3.2, 3.3, and 3.4 (for Python 3 you need to pass it through 2to3 first).  I really would like to see this feature in the standard library in order to get rid of this hack in my package.  I would happy to transform my module into a patch to the library if such a patch would have a chance to get accepted.

[1]: https://code.google.com/p/icatproject/wiki/PythonIcat
msg226018 - (view) Author: Piotr Dobrogost (piotr.dobrogost) Date: 2014-08-28 11:28
@Rotkraut

The truth is http in stdlib is dead.
Your best option is to use 3rd party libs like requests or urllib3.
Authors of these libs plan to get rid of httplib entirely; see "Moving away from httplib" (https://github.com/shazow/urllib3/issues/58)
msg226024 - (view) Author: Rolf Krahl (Rotkraut) Date: 2014-08-28 13:54
Thanks for the notice!  As far as I read from the link you cite, getting rid of the current httplib in urllib3 is planned but far from being done.  Furthermore, I don't like packages with too many 3rd party dependencies.  Since my package is working fine with the current standard lib, even though using an ugly hack in one place, I'd consider switching to urllib3 as soon as the latter makes it into the standard lib.

I still believe that adding chunked transfer encoding to http.client and urllib in the current standard lib would only require a rather small change that can easily be done such that the lib remains fully compatible with existing code.  Still waiting for feedback if such a patch is welcome.
History
Date User Action Args
2014-08-28 13:54:29Rotkrautsetmessages: + msg226024
2014-08-28 11:28:08piotr.dobrogostsetmessages: + msg226018
2014-08-28 08:45:48Rotkrautsetfiles: + chunkedhttp.py
nosy: + Rotkraut
messages: + msg226012

2014-07-26 01:15:14demian.brechtsetnosy: + demian.brecht
2014-07-26 00:10:59whitemicesetnosy: + whitemice
2012-10-10 21:51:32piotr.dobrogostsetnosy: + piotr.dobrogost
2012-09-25 13:14:48pitrousetnosy: + pitrou
messages: + msg171268
2011-06-20 08:13:35orsenthilsetassignee: orsenthil
messages: + msg138691
stage: needs patch
2011-06-15 07:39:49harobedsetmessages: + msg138357
2011-06-14 13:03:49pitrousetnosy: + orsenthil
2011-06-14 06:15:57petri.lehtinensetmessages: + msg138296
2011-06-13 15:35:20harobedsetmessages: + msg138258
2011-06-13 13:01:35petri.lehtinensetversions: - Python 2.7
nosy: + petri.lehtinen

messages: + msg138242

type: enhancement
2011-06-12 10:47:13harobedcreate