classification
Title: urllib(2) should allow automatic decoding by charset
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Add a "decode to declared encoding" version of urlopen to urllib
View: 4733
Assigned To: Nosy List: ajaksu2, edemaine, eric.araujo, jjlee, loewis, mastrodomenico, orsenthil
Priority: normal Keywords: easy

Created on 2006-11-19 19:47 by edemaine, last changed 2010-11-18 02:26 by eric.araujo. This issue is now closed.

Messages (3)
msg61263 - (view) Author: Erik Demaine (edemaine) Date: 2006-11-19 19:47
Currently, urllib.urlopen(...).read() returns a string, not a unicode object.  Ditto for urllib2.  No attempt is made to decode the data using the charset encoding specified in the header ....info()['Content-Type'].

Is it fair to assume that, in Python 3K, urllib....read() will return (Unicode) strings instead of bytes, automatically decoding according to the charset?

Do you think we could expose this futuristic functionality in Python 2?  I doubt we could change read() without breaking a lot of existing code that already does this decoding (e.g., http://zesty.ca/python/scrape.py), but perhaps a 'uread()' method could return a unicode object instead of a string.
msg61264 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-11-22 06:57
I don't think urlopen(...).read() should return strings in Py3k, but instead it should return bytes - in general, resources retrieved are byte sequences (many are application/octet-stream).

Making the return type depend on the resource being fetched is also unintuitive.

It might be reasonable to have the user specified "binary" or "text" on urlopen() (just like regular open()).
msg81429 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-02-09 00:29
There's an attempt to implement this behavior, for 3.1, in issue 4733.
Maybe having a parallel in 2.7 could help the 2.x 3.x transition for
some users?
History
Date User Action Args
2010-11-18 02:33:31eric.araujounlinkissue4733 dependencies
2010-11-18 02:26:19eric.araujosetstatus: open -> closed
dependencies: - Add a "decode to declared encoding" version of urlopen to urllib
superseder: Add a "decode to declared encoding" version of urlopen to urllib
versions: + Python 3.2, - Python 3.1, Python 2.7
nosy: + eric.araujo

resolution: duplicate
stage: test needed -> resolved
2010-01-27 23:47:10mastrodomenicosetnosy: + mastrodomenico
2009-04-22 17:26:15ajaksu2setkeywords: + easy
2009-02-13 01:20:26ajaksu2setnosy: + jjlee
2009-02-12 18:24:23ajaksu2linkissue4733 dependencies
2009-02-12 18:23:53ajaksu2setnosy: + orsenthil
dependencies: + Add a "decode to declared encoding" version of urlopen to urllib
stage: test needed
2009-02-09 00:29:25ajaksu2setnosy: + ajaksu2
messages: + msg81429
versions: + Python 3.1, Python 2.7
2006-11-19 19:47:30edemainecreate