classification
Title: FD leak in urllib2
Type: resource usage Stage: resolved
Components: IO, Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: ResourceWarning in urllib.request
View: 12133
Assigned To: Nosy List: Valery.Khamenya, eric.araujo, ezio.melotti, orsenthil, terry.reedy, vstinner
Priority: normal Keywords: patch

Created on 2011-10-08 12:25 by Valery.Khamenya, last changed 2019-10-24 10:29 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
leak.py Valery.Khamenya, 2011-10-08 12:25
urllib2.patch Valery.Khamenya, 2011-10-08 19:25
Messages (7)
msg145166 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 12:25
Explanation from dablitz's comment at https://bugs.pypy.org/issue867 :

<quote>
urllib2 in the stdlib leaks fd's if an exception is raised while opening a connection. The issue occurs due to a socket being opened then an exception being raised before an object with the socket is returned, leaving no way to explicitly close the object. On cpython this would not be an issue as the object would lose all references almost immediately however it lingers around with a proper GC causing FD's to build up if the same condition happens repeatedly (eg a loop/web crawling)

The file enclosed is a script to generate the leakage, to run invok it as follows <python> leak.py

pypy should start leaking FD's and can be see in /proc/<pid of leak.py>/fd

</quote>

Related issues:
http://bugs.python.org/issue3066
http://bugs.python.org/issue1208304
http://bugs.python.org/issue1601399
msg145169 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 13:23
by the way, timeout parameter should be set to 0.2 as for my 13Mbit ADSL line. With 0.002 it is not reproducible for my environment
msg145183 - (view) Author: Valery Khamenya (Valery.Khamenya) Date: 2011-10-08 19:25
the problem seems to be fixed with the patch attached. 
Thanks go to fijal@freenode
msg145184 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-10-08 19:50
Pay attention not to introduce regressions like the one in #12576 while fixing this.  I'm not sure there are similar tests for urllib2 -- if not they should be added.
msg145568 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-10-14 22:34
Can you test both the leak and fix (if needed) on 3.2? 
(I am on Win7).
msg145668 - (view) Author: √Čric Araujo (eric.araujo) * (Python committer) Date: 2011-10-17 11:40
Instead of except+raise, you could use finally.
msg355318 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-24 10:29
This issue is a duplicate of bpo-12133 which has been fixed in Python 2.7 by:

commit c74a6ba2d6c1f331896cf8dacc698b0b88c7e670
Author: Victor Stinner <victor.stinner@haypocalc.com>
Date:   Fri Jun 17 14:06:27 2011 +0200

    Issue #12133: AbstractHTTPHandler.do_open() of urllib.request closes the HTTP
    connection if its getresponse() method fails with a socket error. Patch written
    by Ezio Melotti.
History
Date User Action Args
2019-10-24 10:29:32vstinnersetstatus: open -> closed

superseder: ResourceWarning in urllib.request

nosy: + vstinner
messages: + msg355318
resolution: duplicate
stage: patch review -> resolved
2011-10-17 11:40:28eric.araujosetnosy: + eric.araujo
messages: + msg145668
2011-10-14 22:34:48terry.reedysetnosy: + terry.reedy
messages: + msg145568
2011-10-08 19:50:42ezio.melottisetnosy: + ezio.melotti

messages: + msg145184
stage: patch review
2011-10-08 19:45:18pitrousetnosy: + orsenthil
2011-10-08 19:25:49Valery.Khamenyasetfiles: + urllib2.patch
keywords: + patch
messages: + msg145183
2011-10-08 13:23:21Valery.Khamenyasetmessages: + msg145169
2011-10-08 12:25:52Valery.Khamenyacreate