This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: transparent gzip compression in urllib
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.5
Status: open Resolution:
Dependencies: Superseder:
Assigned To: orsenthil Nosy List: Jim.Jewett, abacabadabacaba, antialize, berker.peksag, demian.brecht, eric.araujo, jcea, jcon, jerub, jjlee, martin.panter, nadeem.vawda, orsenthil, rhettinger, ruseel, serhiy.storchaka, thomaspinckney3, vstinner
Priority: high Keywords: patch

Created on 2006-06-19 08:59 by antialize, last changed 2022-04-11 14:56 by admin.

File name Uploaded Description Edit
urllib2-gzip.patch antialize, 2006-06-19 09:26 urllib2-gzip.patch
issue1508475.diff orsenthil, 2010-11-25 08:49 review
http_client_gzip.patch vstinner, 2014-02-21 23:34 review
Messages (23)
msg50500 - (view) Author: Jakob Truelsen (antialize) Date: 2006-06-19 08:59
Some webservers support gzipping things before sending
them, this patch adds transparrent support for this in
urllib2 (documentation

This patach *requires* hash patch 914340 as a
prerequirement as this enabels stream support in the
gzip libary.. 
msg50501 - (view) Author: John J Lee (jjlee) Date: 2007-01-30 01:34
Looks good.

This needs tests and docs.  As a new feature, this could not be released until Python 2.6.

It would be nice to have support for managing content negotiation in general, but that wish isn't an obstacle to this patch.
msg114671 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-22 10:53
@Jakob could you provide an updated patch for py3k that includes unit test and doc changes?
msg114725 - (view) Author: Jakob Truelsen (antialize) Date: 2010-08-23 07:43
No, I have long since moved on to other things.
msg114726 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-08-23 07:47
Its okay, Jacab, we will take it forward.
msg122342 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-11-25 08:43
The transparent gzip Content-Encoding support should be done at the
http.client level code.

Before adding this feature, a question needs to be sorted out.

If we support the transparent gzip and wrap the file pointer to a
GzipFile filepointer, should reset the Content-Length value?

What if a user of urllib is relying on the Content-Length of response
to do something further?

I observed that google-chrome returns the uncompressed output (which
is correct for a browser), but has the Content-Length set the
compressed output length.
msg122343 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2010-11-25 08:49
Patch for py3k.
msg122351 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-25 11:21
msg158315 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-04-15 09:51
What if the gzip module is not available?

I think, with transparent decompression should delete headers Content-Encoding (to free the user from re-decompression) and Content-Length (which is wrong).
msg158380 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-04-15 23:52
In that case, transparent decompression should not be available. (
Request header should not be sent and response wont be compressed).
msg158400 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-04-16 05:41
The patch for py3k also has the disadvantage that the content is decoded even if the user has defined a Content-Encoding and he is going to process compressed response himself.
msg160355 - (view) Author: Tom Pinckney (thomaspinckney3) * Date: 2012-05-10 17:12
What if this gzip decompression was optional and controlled via a flag or handler instead of making it automagic?

It's not entirely trivial to implement so it is nice to have the option of this happening automatically if one wishes.

Then, the caller would be aware that Content-length / Accept-encoding / Content-encoding etc have been modified iff they requested gzip decompression.
msg160384 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-10 23:43
Enabled by default with a knob to turn it off sounds good.  Maybe the original headers could be preserved in some object.
msg163925 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-06-25 10:26
The first step is to answer on the fundamental question: on what level transparent decompression will work? On http.client level or on urllib level? Patch for first case will be much more difficult, but will benefit from compression in other http-based protocols.
msg163935 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2012-06-25 10:53
I think, the transparent compression should work at http.client level. I also agree with other points made by Serhiy:

- transparent decompression should delete headers Content-Encoding and Content-Length (this is as per RFC too)

- Should not do another compression if the user has a explicit specified intent of using Content-Encoding: gzip and is ready to do decompression himself.

- This transparent compression/decompression would require the availability gzip module, if not then the feature may be disabled and normal request-response cycle would proceed.

- I think, having it 'ON' with a flag to switch 'OFF' would be more desirable than having this feature via Handler. The reason being it can help in performance of any requests on servers that support it and browsers have adopted similar approach too.
msg211893 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-21 23:34
I updated issue1508475.diff for Python 3.4 and removed the change in urllib: http_client_gzip.patch. The patch only changes http.client to support server sending gzip data.

For example, the new website serves gzip data even if the Accept-Encoding header is not sent by the client: see the issue #20719.
msg212252 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-02-26 12:08
I have code that already handles an “gzip” encoded response from urlopen(). All three patches leave the Content-Encoding header intact, so I suspect my code would try to decompress the body a second time. Deleting this header (as already suggested) would work for me.
msg212473 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-02-28 21:20
Victor, the patch looks good and would be a welcome enhancement.

There should be an option for turning this on and off (perhaps, I want the zipped content and want to unzip later or in a different thread).

Consider adding support for "deflate" as well.
msg213982 - (view) Author: Jim Jewett (Jim.Jewett) * (Python triager) Date: 2014-03-18 15:44
This is an enhancement, so I am changing the affected version from 3.3 to 3.5.  

It is python-only, which works well with the cheeseshop.

That said, the patch is truly short; if that is really sufficient, it could almost go into the documentation as a recipe.  But I would prefer some more assurances that it actually does work; a quick skim suggests that it relies on a superclass happening to implement read via readinto.

Needs tests and documentation change.
msg226258 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-09-02 01:26
I think the patch is indeed a bit short, for instannce it looks like calling read() without a size limit could bypass the decoding.

Also, I wonder if Content-Encoding handling is better done at a higher level. What if someone wants to download a *.tar.gz file? They may not expect the tar file to be transparently decompressed. And I suspect this would blow up if you tried a partial range request.

Transfer-Encoding is meant to be the proper way to transparently compress HTTP messages at a low level, but it doesn’t seem to be used as much in the real world.
msg226399 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-09-05 05:45
Related: Issue 1243678, which includes a patch for “httplib” (now known as “http.client”?). That patch looks like it sets Accept-Encoding and decodes according to Content-Encoding. However I suspect it is also trying to be too “transparent” at the wrong level and would have many of the problems already mentioned here.
msg234363 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-01-20 09:32
The Lib/xmlrpc/ file appears to already support compression using “Content-Encoding: gzip”. Perhaps it could be leveraged for any work on this issue.
msg236013 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-02-15 00:18
I suggest resolving Issue 15955 first, then the GzipFile API could be used without fear of decompression bombs.
Date User Action Args
2022-04-11 14:56:18adminsetgithub: 43521
2015-05-27 22:41:12martin.panterlinkissue1243678 superseder
2015-03-07 02:32:39demian.brechtsetnosy: + demian.brecht
2015-02-15 00:18:02martin.pantersetmessages: + msg236013
2015-02-11 12:15:27berker.peksagsetnosy: + berker.peksag
2015-01-20 09:32:05martin.pantersetmessages: + msg234363
2014-09-05 05:45:04martin.pantersetmessages: + msg226399
2014-09-02 01:26:57martin.pantersetmessages: + msg226258
2014-03-19 06:05:02berker.peksagsetversions: + Python 3.5, - Python 3.3
2014-03-18 15:44:32Jim.Jewettsetnosy: + Jim.Jewett
messages: + msg213982
2014-02-28 21:20:55rhettingersetnosy: + rhettinger
messages: + msg212473
2014-02-26 12:08:10martin.pantersetnosy: + martin.panter
messages: + msg212252
2014-02-21 23:34:41vstinnersetfiles: + http_client_gzip.patch

nosy: + vstinner
messages: + msg211893

keywords: + patch
2013-01-10 04:33:47jconsetnosy: + jcon
2012-07-07 16:27:14jerubsetnosy: + jerub
2012-06-25 10:53:45orsenthilsetmessages: + msg163935
2012-06-25 10:26:50serhiy.storchakasetmessages: + msg163925
2012-06-25 05:38:14rhettingersetpriority: normal -> high
2012-06-17 14:30:41jceasetnosy: + jcea
2012-06-17 10:32:19pitroulinkissue15089 superseder
2012-05-10 23:43:07eric.araujosetkeywords: - patch, easy

messages: + msg160384
2012-05-10 17:12:05thomaspinckney3setnosy: + thomaspinckney3
messages: + msg160355
2012-04-16 05:41:12serhiy.storchakasetmessages: + msg158400
2012-04-15 23:52:40orsenthilsetmessages: + msg158380
2012-04-15 09:51:31serhiy.storchakasetmessages: + msg158315
2012-04-15 07:07:02serhiy.storchakasetnosy: + serhiy.storchaka

versions: + Python 3.3, - Python 3.2
2012-03-04 20:39:13abacabadabacabasetnosy: + abacabadabacaba
2010-11-25 11:21:10eric.araujosetmessages: + msg122351
2010-11-25 08:49:49orsenthilsetfiles: + issue1508475.diff

messages: + msg122343
stage: test needed -> patch review
2010-11-25 08:43:48orsenthilsetmessages: + msg122342
2010-11-23 21:26:28nadeem.vawdasetnosy: + nadeem.vawda
2010-11-23 04:39:56ruseelsetnosy: + ruseel
2010-11-20 20:23:25eric.araujosetnosy: + eric.araujo, - BreamoreBoy
components: - Extension Modules
title: transparent gzip compression in liburl2 -> transparent gzip compression in urllib
2010-11-20 20:17:57r.david.murraylinkissue9500 superseder
2010-08-23 07:47:06orsenthilsetassignee: orsenthil
messages: + msg114726
2010-08-23 07:43:26antializesetmessages: + msg114725
2010-08-22 10:53:38BreamoreBoysetnosy: + BreamoreBoy

messages: + msg114671
versions: + Python 3.2, - Python 2.7
2009-04-22 18:48:53ajaksu2setkeywords: + easy
2009-02-12 17:42:44ajaksu2setnosy: + orsenthil
stage: test needed
type: enhancement
components: + Library (Lib)
versions: + Python 2.7, - Python 2.4
2006-06-19 08:59:09antializecreate