New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
transparent gzip compression in urllib #43521
Comments
Some webservers support gzipping things before sending This patach *requires* hash patch 914340 as a |
Looks good. This needs tests and docs. As a new feature, this could not be released until Python 2.6. It would be nice to have support for managing content negotiation in general, but that wish isn't an obstacle to this patch. |
@jakob could you provide an updated patch for py3k that includes unit test and doc changes? |
No, I have long since moved on to other things. |
Its okay, Jacab, we will take it forward. |
The transparent gzip Content-Encoding support should be done at the Before adding this feature, a question needs to be sorted out. If we support the transparent gzip and wrap the file pointer to a What if a user of urllib is relying on the Content-Length of response I observed that google-chrome returns the uncompressed output (which |
Patch for py3k. |
What if the gzip module is not available? I think, with transparent decompression should delete headers Content-Encoding (to free the user from re-decompression) and Content-Length (which is wrong). |
In that case, transparent decompression should not be available. ( |
The patch for py3k also has the disadvantage that the content is decoded even if the user has defined a Content-Encoding and he is going to process compressed response himself. |
What if this gzip decompression was optional and controlled via a flag or handler instead of making it automagic? It's not entirely trivial to implement so it is nice to have the option of this happening automatically if one wishes. Then, the caller would be aware that Content-length / Accept-encoding / Content-encoding etc have been modified iff they requested gzip decompression. |
Enabled by default with a knob to turn it off sounds good. Maybe the original headers could be preserved in some object. |
The first step is to answer on the fundamental question: on what level transparent decompression will work? On http.client level or on urllib level? Patch for first case will be much more difficult, but will benefit from compression in other http-based protocols. |
I think, the transparent compression should work at http.client level. I also agree with other points made by Serhiy:
|
I updated bpo-1508475.diff for Python 3.4 and removed the change in urllib: http_client_gzip.patch. The patch only changes http.client to support server sending gzip data. For example, the new python.org website serves gzip data even if the Accept-Encoding header is not sent by the client: see the issue bpo-20719. |
I have code that already handles an “gzip” encoded response from urlopen(). All three patches leave the Content-Encoding header intact, so I suspect my code would try to decompress the body a second time. Deleting this header (as already suggested) would work for me. |
Victor, the patch looks good and would be a welcome enhancement. There should be an option for turning this on and off (perhaps, I want the zipped content and want to unzip later or in a different thread). Consider adding support for "deflate" as well. |
This is an enhancement, so I am changing the affected version from 3.3 to 3.5. It is python-only, which works well with the cheeseshop. That said, the patch is truly short; if that is really sufficient, it could almost go into the documentation as a recipe. But I would prefer some more assurances that it actually does work; a quick skim suggests that it relies on a superclass happening to implement read via readinto. Needs tests and documentation change. |
I think the patch is indeed a bit short, for instannce it looks like calling read() without a size limit could bypass the decoding. Also, I wonder if Content-Encoding handling is better done at a higher level. What if someone wants to download a *.tar.gz file? They may not expect the tar file to be transparently decompressed. And I suspect this would blow up if you tried a partial range request. Transfer-Encoding is meant to be the proper way to transparently compress HTTP messages at a low level, but it doesn’t seem to be used as much in the real world. |
Related: bpo-1243678, which includes a patch for “httplib” (now known as “http.client”?). That patch looks like it sets Accept-Encoding and decodes according to Content-Encoding. However I suspect it is also trying to be too “transparent” at the wrong level and would have many of the problems already mentioned here. |
The Lib/xmlrpc/client.py file appears to already support compression using “Content-Encoding: gzip”. Perhaps it could be leveraged for any work on this issue. |
I suggest resolving bpo-15955 first, then the GzipFile API could be used without fear of decompression bombs. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: