classification
Title: Remove mimetools usage from the stdlib
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: barry Nosy List: barry, bboissin, benjamin.peterson, brett.cannon, eric.araujo, hdiogenes, rhettinger
Priority: normal Keywords: patch

Created on 2008-05-14 00:59 by brett.cannon, last changed 2010-07-17 22:23 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
remove_mimetools_from_urllib.patch hdiogenes, 2008-06-04 06:00 Remove mimetools from (test_)urllib (updated)
email.parser-small_fix.patch hdiogenes, 2008-06-06 23:11 Simple patch to remove outdated workaround from email Parser
replace_getheaders_by_get.patch hdiogenes, 2008-06-07 01:49 Start using methods compatible with both mimetools and email.Message
rename_HTTPResponse.getheaders.patch hdiogenes, 2008-06-07 02:07 Suggestion: rename HTTPResponse.getheader() to .get_header()
remove_mimetools_from_http.client.patch hdiogenes, 2008-06-07 03:28 Breaks tests because FeedParser closes the socket.
remove_mimetools.patch hdiogenes, 2008-06-11 13:31 Completely removes dependencies on mimetools (but breaks two tests).
applied_mimetools.patch barry, 2008-06-12 04:07
Messages (23)
msg66809 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-05-14 00:59
The mimetools module has been deprecated for ages, but it is still used in 
multiple places (a quick grep lists ``cgi``, ``httplib``, ``urllib``, 
``urllib2``, ``test_cookielib``, ``test_multifile``, ``test_urllib``, 
``test_urllib2``, ``test_urllib2net``, ``test_urllib_localnet``, 
``test_urllibnet``, ``test_xmlrpc``). All uses need to be removed before 
the module can go from Py3K.
msg66973 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-05-16 23:32
mimetools.Message is compatible with email.message.Message, right?
msg67176 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-05-22 02:40
I don't know how compatible it is.
msg67177 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-05-22 03:10
The APIs are bit different, but it should be possible to migrate from 
the old to the new.  All the key functionality is there.

Barry, any thoughts on transitioning away from the deprecated modules?
msg67681 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-04 03:50
The only use of mimetools inside cgi is in parse_multipart, which is 
untested and has huge warnings like these ones:

"""
XXX This does not parse nested multipart parts -- use FieldStorage for
that.

XXX This should really be subsumed by FieldStorage altogether -- no
point in having two implementations of the same parsing algorithm.
Also, FieldStorage protects itself better against certain DoS attacks
by limiting the size of the data read in one chunk.  The API here
does not support that kind of protection.  This also affects parse()
since it can call parse_multipart().
"""
msg67682 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-04 04:58
I'm sending a patch that removes mimetools from urllib and test_urllib. It 
now returns an email.message.Message instead of mimetools.Message.

I also moved some imports to the top of the file. If that's a problem, I 
can send another patch.
msg67684 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-04 06:43
It's relatively easy to replace mimetools in most places: it's just a 
matter of changing mimetools.Message to email.message_from_file, but it 
gets a lot more complicated when there's inheritance involved.

The problem is that mimetools.Message is both a parser and a Message 
representation, and AFAIK there's no single class in ``email`` which does 
both -- the functionality is split between the Parser and Message classes.

Inheritance happens in two places: ``pydoc.Message`` and 
``http.client.HTTPMessage``, and I couldn't find a solution for those 
cases yet. Any help would be appreciated.
msg67686 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-06-04 07:37
One idea:  in http.client.HTTPMessage, inherit from email.Message, 
modify the addheaders/addcontinue/readheaders methods to use the 
methods in Message.   Then, change the instantiation in HTTPResponse to 
self.msg = email.parser.Parser(_class=HTTPMessage).parsestr(io.BytesIO
()).  

Something similar may apply to pydoc.  FYI, I believe there is a 
pending discussion about whether to keep pydoc in Py3.0.
msg67791 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-07 01:48
Attached a patch to rename all calls to the mimetools-specific 
.getheader() to .get(), which works on both libraries (mimetools and 
email.Message). That eases transition without breaking any tests.
msg67793 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-07 02:07
As we're moving away from using .getheader() everywhere (file 10538), I 
think it's a good time to make HTTPResponse conform to PEP8 and define 
.get_header (with underscore) instead.
msg67794 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-06-07 02:14
-1 on the get_header() change.  I would like this patch to be as 
minimal as possible.  The new spelling is a bit awkward and it 
introduces an unnecessary change that could break code.
msg67797 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-07 03:28
Raymond, thanks for the parser solution! It worked -- just broke some 
tests because of the nature of FeedParser. While rfc822.Message can read 
the file and stop at some point in the middle (meaning that self.fp can 
still be read), FeedParser will always read the whole file *and* close 
it, even if setting _headersonly to True. This is the same problem that 
I had to work around on issue 2849. I'm attaching a patch that 
demonstrates it: lots of tests giving "I/O operation on closed file."

Right now I can think of two options:
 * Working around it, again (don't know how)
 * Implement some flag in FeedParser to make it consume only the headers 
and leave the file open.

Any other idea?

Oh, and if you think it's better not to mess with the name of other 
methods like getheader, that's fine by me. It was only a suggestion, 
after all. ;)
msg67970 - (view) Author: Humberto Diogenes (hdiogenes) Date: 2008-06-11 13:31
mimetools removal is almost complete, with the exception of only two 
broken tests: test_urllib2_localnet and test_xmlrpc. I'm not sure if it 
can make it to this beta, but at least it's really close.
msg67971 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-06-11 13:40
Barry, can you take a look at this?
Ideally, it should go it before the beta
so it can get thoroughly exercised.
msg68050 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 03:35
This patch works for me in Python 3.0.  It's basically Humberto's patch
with the two failing tests fixed.
msg68051 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 04:07
Actually, try this one.
msg68052 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 04:09
r64164 in Python 3.0.  This doesn't apply cleanly to Python 2.6; could
someone please back port it?
msg68058 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-12 12:56
On Wed, Jun 11, 2008 at 11:09 PM, Barry A. Warsaw
<report@bugs.python.org> wrote:
>
> Barry A. Warsaw <barry@python.org> added the comment:
>
> r64164 in Python 3.0.  This doesn't apply cleanly to Python 2.6; could
> someone please back port it?

Why does it need to be in 2.6? mimetools is still there.
msg68060 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2008-06-12 13:05
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Jun 12, 2008, at 8:57 AM, Benjamin Peterson wrote:

> Benjamin Peterson <musiccomposition@gmail.com> added the comment:
>
> On Wed, Jun 11, 2008 at 11:09 PM, Barry A. Warsaw
> <report@bugs.python.org> wrote:
>>
>> Barry A. Warsaw <barry@python.org> added the comment:
>>
>> r64164 in Python 3.0.  This doesn't apply cleanly to Python 2.6;  
>> could
>> someone please back port it?
>
> Why does it need to be in 2.6? mimetools is still there.

I guess you're right, it doesn't.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSFEfD3EjvBPtnXfVAQKrBwQAoOOjFenHlq0DVEtfeVSuG5/rPWHrzfQw
OwoSlYWa5zkqWqjFdRG780rElbfAjgtz3J7Od8HENxUfsHBvH3bFM0PZCLSQRZMI
UVP8dM1KIcBDOZoQPMoahM1Q7l16iCsayhauWEnJP2MIVje7D/rXO9jjpkQixRyi
3wkfmkNin0k=
=gAB6
-----END PGP SIGNATURE-----
msg68093 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2008-06-12 21:26
This is done.
msg77153 - (view) Author: Benoit Boissinot (bboissin) Date: 2008-12-06 17:35
> Why does it need to be in 2.6? mimetools is still there.

Because it outputs a warning with python2.6 -3, it's annoying for people
trying to port their application to python3k
msg77179 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-12-06 22:39
Benoit has a good point, so I have opened this issue again so we can get
2.7 even closer to 3.x compatibility.
msg110608 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-07-17 22:23
Closing as outdated since 2.7 has been release with some mimetools usage left.
History
Date User Action Args
2010-07-17 22:23:17eric.araujosetstatus: open -> closed
versions: + Python 2.7, - Python 2.6, Python 3.0
messages: + msg110608

resolution: out of date
stage: resolved
2009-09-10 23:26:05eric.araujosetnosy: + eric.araujo
2009-02-28 16:53:57benjamin.petersonsetpriority: critical -> normal
2008-12-06 22:39:00brett.cannonsetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg77179
2008-12-06 17:35:11bboissinsetnosy: + bboissin
messages: + msg77153
2008-06-12 21:28:46benjamin.petersonunlinkissue2775 dependencies
2008-06-12 21:26:32benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg68093
2008-06-12 13:05:23barrysetmessages: + msg68060
2008-06-12 12:56:56benjamin.petersonsetmessages: + msg68058
2008-06-12 04:09:45barrysetpriority: release blocker -> critical
messages: + msg68052
2008-06-12 04:07:37barrysetfiles: + applied_mimetools.patch
messages: + msg68051
2008-06-12 04:06:56barrysetfiles: - applied_mimetools.patch
2008-06-12 03:36:03barrysetfiles: + applied_mimetools.patch
messages: + msg68050
2008-06-11 13:40:50rhettingersetmessages: + msg67971
2008-06-11 13:32:17hdiogenessetfiles: + remove_mimetools.patch
messages: + msg67970
2008-06-07 03:28:47hdiogenessetfiles: + remove_mimetools_from_http.client.patch
messages: + msg67797
2008-06-07 02:14:05rhettingersetmessages: + msg67794
2008-06-07 02:07:02hdiogenessetfiles: + rename_HTTPResponse.getheaders.patch
messages: + msg67793
2008-06-07 01:49:11hdiogenessetfiles: + replace_getheaders_by_get.patch
messages: + msg67791
2008-06-06 23:11:08hdiogenessetfiles: + email.parser-small_fix.patch
2008-06-04 07:37:46rhettingersetmessages: + msg67686
2008-06-04 06:43:14hdiogenessetmessages: + msg67684
2008-06-04 06:01:10hdiogenessetfiles: - remove_mimetools_from_urllib.patch
2008-06-04 06:00:45hdiogenessetfiles: + remove_mimetools_from_urllib.patch
2008-06-04 04:58:05hdiogenessetfiles: + remove_mimetools_from_urllib.patch
keywords: + patch
messages: + msg67682
2008-06-04 03:50:58hdiogenessetnosy: + hdiogenes
messages: + msg67681
2008-05-22 03:10:25rhettingersetassignee: barry
messages: + msg67177
nosy: + rhettinger
2008-05-22 02:40:41brett.cannonsetmessages: + msg67176
2008-05-16 23:33:07benjamin.petersonsetnosy: + barry, benjamin.peterson
messages: + msg66973
2008-05-16 04:43:08brett.cannonsetpriority: critical -> release blocker
2008-05-14 00:59:53brett.cannonlinkissue2775 dependencies
2008-05-14 00:59:34brett.cannoncreate