classification
Title: urllib.request memory leak / overflow
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: ssl.enum_certificates() regression
View: 35941
Assigned To: Nosy List: Ofer Sadan, christian.heimes, hongweipeng, kakshay, steve.dower, xtreak
Priority: normal Keywords:

Created on 2019-09-22 11:29 by Ofer Sadan, last changed 2019-09-23 10:34 by Ofer Sadan. This issue is now closed.

Messages (10)
msg352968 - (view) Author: Ofer Sadan (Ofer Sadan) Date: 2019-09-22 11:29
running `urllib.request.urlopen` multiple times causes the memory usage to increase with each run, even after calling `close()` on the request or using `del` on the result

To recreate this problem, run code:

    import urllib.request
    def ip():
        r = urllib.request.urlopen('https://api.ipify.org')
        b = r.read()
        r.close()
        print(len(b))
        return b

    for i in range(1000):
        result = ip()
        del result

Even though `len(b)` has a maximum value of 15 (for this url at least), the memory increases with each run by 200KB - 1MB
msg352969 - (view) Author: Ofer Sadan (Ofer Sadan) Date: 2019-09-22 11:35
Just to note: I'm using Python 3.7.4 on Windows 10 (64bit), the issue exists on both the 32bit/64bit python versions I could test on.
msg352975 - (view) Author: Kumar Akshay (kakshay) * Date: 2019-09-22 12:19
JFYI, Not reproducible on macOS.
msg352976 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-09-22 12:31
Is this related to https://bugs.python.org/issue35941 given that it's related to windows. Similar report https://bugs.python.org/issue37498
msg352978 - (view) Author: Kumar Akshay (kakshay) * Date: 2019-09-22 12:42
Yes, https://bugs.python.org/issue37498 is indeed the same issue
msg352989 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-22 22:15
Closing as a duplicate of issue35941
msg352996 - (view) Author: hongweipeng (hongweipeng) * Date: 2019-09-23 09:17
My Windows 10 (64bit). There is no such problem in 3.6.1, but 3.7.0 has. And when requesting `http://...`, both of them are normal.

By the way, the related issues are all closed, is it supposed to reopen one?
msg353000 - (view) Author: Ofer Sadan (Ofer Sadan) Date: 2019-09-23 09:58
I can confirm that testing with http instead of https this problem doesn't exist, on 3.7.4.

Are the issues closed because this problem is fixed in the upcoming 3.7.5, or 3.8.0?
msg353005 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-09-23 10:25
Yes, the problem will be fixed by 3.7.5.

In the mean time you can work around the issue by creating a single SSLContext object with ssl.create_default_context() and reusing it for all requests. It's also much more efficient to reuse a single context for all requests.
msg353006 - (view) Author: Ofer Sadan (Ofer Sadan) Date: 2019-09-23 10:34
I appreciate the suggestion and the update, thank you!
History
Date User Action Args
2019-09-23 10:34:31Ofer Sadansetmessages: + msg353006
2019-09-23 10:25:50christian.heimessetnosy: + christian.heimes
messages: + msg353005
2019-09-23 09:58:54Ofer Sadansetmessages: + msg353000
2019-09-23 09:17:07hongweipengsetmessages: + msg352996
2019-09-22 22:15:46steve.dowersetstatus: open -> closed
superseder: ssl.enum_certificates() regression
messages: + msg352989

resolution: duplicate
stage: resolved
2019-09-22 14:57:20hongweipengsetnosy: + hongweipeng
2019-09-22 12:42:55kakshaysetmessages: + msg352978
2019-09-22 12:31:49xtreaksetnosy: + steve.dower
2019-09-22 12:31:15xtreaksetnosy: + xtreak
messages: + msg352976
2019-09-22 12:19:37kakshaysetnosy: + kakshay
messages: + msg352975
2019-09-22 11:35:21Ofer Sadansetmessages: + msg352969
2019-09-22 11:29:48Ofer Sadancreate