Some webservers support gzipping things before sending
them, this patch adds transparrent support for this in
urllib2 (documentation http://www.http-compression.com/)

This patach *requires* hash patch 914340 as a
prerequirement as this enabels stream support in the
gzip libary.. 

Looks good.

This needs tests and docs.  As a new feature, this could not be released until Python 2.6.

It would be nice to have support for managing content negotiation in general, but that wish isn't an obstacle to this patch.

@Jakob could you provide an updated patch for py3k that includes unit test and doc changes?

No, I have long since moved on to other things.

Its okay, Jacab, we will take it forward.

The transparent gzip Content-Encoding support should be done at the
http.client level code.

Before adding this feature, a question needs to be sorted out.

If we support the transparent gzip and wrap the file pointer to a
GzipFile filepointer, should reset the Content-Length value?

What if a user of urllib is relying on the Content-Length of response
to do something further?

I observed that google-chrome returns the uncompressed output (which
is correct for a browser), but has the Content-Length set the
compressed output length.

Patch for py3k.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

What if the gzip module is not available?

I think, with transparent decompression should delete headers Content-Encoding (to free the user from re-decompression) and Content-Length (which is wrong).

In that case, transparent decompression should not be available. (
Request header should not be sent and response wont be compressed).

The patch for py3k also has the disadvantage that the content is decoded even if the user has defined a Content-Encoding and he is going to process compressed response himself.

What if this gzip decompression was optional and controlled via a flag or handler instead of making it automagic?

It's not entirely trivial to implement so it is nice to have the option of this happening automatically if one wishes.

Then, the caller would be aware that Content-length / Accept-encoding / Content-encoding etc have been modified iff they requested gzip decompression.

Enabled by default with a knob to turn it off sounds good.  Maybe the original headers could be preserved in some object.

The first step is to answer on the fundamental question: on what level transparent decompression will work? On http.client level or on urllib level? Patch for first case will be much more difficult, but will benefit from compression in other http-based protocols.

I think, the transparent compression should work at http.client level. I also agree with other points made by Serhiy:

- transparent decompression should delete headers Content-Encoding and Content-Length (this is as per RFC too)

- Should not do another compression if the user has a explicit specified intent of using Content-Encoding: gzip and is ready to do decompression himself.

- This transparent compression/decompression would require the availability gzip module, if not then the feature may be disabled and normal request-response cycle would proceed.

- I think, having it 'ON' with a flag to switch 'OFF' would be more desirable than having this feature via Handler. The reason being it can help in performance of any requests on servers that support it and browsers have adopted similar approach too.