Issue40727
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2020-05-22 09:29 by Recursing, last changed 2022-04-11 14:59 by admin.
Messages (6) | |||
---|---|---|---|
msg369573 - (view) | Author: (Recursing) * | Date: 2020-05-22 09:29 | |
Minimal code to reproduce: ``` import ssl import certifi import gc import asyncio ca_path = certifi.where() async def make_async_context() -> None: context = ssl.SSLContext(ssl.PROTOCOL_TLS) context.load_verify_locations(ca_path) await asyncio.sleep(1) async def main(n: int) -> None: await asyncio.wait([make_async_context() for _ in range(n)]) gc.collect() asyncio.run(main(2000)) input("Finished run, still using lots of memory :(") gc.collect() input("gc.collect() does not help :(") ``` Running this code on several linux machines (with python from 3.6.9 to 3.9.0a5, and openSSL from 1.1.1 11 Sep 2018 to 1.1.1g 21 Apr 2020) causes a significant memory leak, while on windows memory usage peaks around 1 GB but gets freed |
|||
msg369578 - (view) | Author: Christian Heimes (christian.heimes) * ![]() |
Date: 2020-05-22 09:40 | |
Does it also leak without asyncio? |
|||
msg369582 - (view) | Author: (Recursing) * | Date: 2020-05-22 09:55 | |
Removing the `await asyncio.sleep(1)` removes the leak, while changing it to `await asyncio.sleep(0)` seems to keep it |
|||
msg369583 - (view) | Author: Christian Heimes (christian.heimes) * ![]() |
Date: 2020-05-22 10:09 | |
Without asyncio memory consumption stays low and stable for me: $ ./python -m venv venv $ ./venv/bin/pip install psutil $ ./venv/bin/python >>> ssl.OPENSSL_VERSION 'OpenSSL 1.1.1g FIPS 21 Apr 2020' >>> import psutil, ssl, os >>> p = psutil.Process(os.getpid()) >>> cafile = ssl.get_default_verify_paths().cafile >>> p.memory_info() pmem(rss=14811136, vms=237223936, shared=8138752, text=2125824, lib=0, data=6701056, dirty=0) >>> for i in range(1000): ... ssl.SSLContext(ssl.PROTOCOL_TLS).load_verify_locations(cafile) ... >>> p.memory_info() pmem(rss=17489920, vms=238170112, shared=9863168, text=2125824, lib=0, data=7647232, dirty=0) >>> for i in range(1000): ... ssl.SSLContext(ssl.PROTOCOL_TLS).load_verify_locations(cafile) ... >>> p.memory_info() pmem(rss=17489920, vms=238170112, shared=9863168, text=2125824, lib=0, data=7647232, dirty=0) |
|||
msg369584 - (view) | Author: Christian Heimes (christian.heimes) * ![]() |
Date: 2020-05-22 10:17 | |
When I run your example, RSS jumps from 20 MB to about 1,600 MB. There is almost no increase when I run the look several more times. >>> p.memory_info() pmem(rss=19902464, vms=240513024, shared=10014720, text=2125824, lib=0, data=9887744, dirty=0) >>> asyncio.run(main(2000)) <stdin>:2: DeprecationWarning: The explicit passing of coroutine objects to asyncio.wait() is deprecated since Python 3.8, and scheduled for removal in Python 3.11. >>> p.memory_info() pmem(rss=1608568832, vms=1829105664, shared=10014720, text=2125824, lib=0, data=1598480384, dirty=0) >>> asyncio.run(main(2000)) >>> p.memory_info() pmem(rss=1608835072, vms=1829367808, shared=10014720, text=2125824, lib=0, data=1598742528, dirty=0) >>> asyncio.run(main(2000)) >>> p.memory_info() pmem(rss=1608601600, vms=1829367808, shared=10014720, text=2125824, lib=0, data=1598742528, dirty=0) Why are you creating so many SSLContext objects any way? It's very inefficient and really not necessary. I recommend that you create one context in your application and reuse for all connection. You only ever need additional contexts for different configuration (protocol, verification, trust anchors, ...). |
|||
msg369586 - (view) | Author: (Recursing) * | Date: 2020-05-22 10:32 | |
> Without asyncio memory consumption stays low and stable for me Same for me > RSS jumps from 20 MB to about 1,600 MB. That is the memory consumption I observe as well, the issue is that it doesn't get freed on Linux > There is almost no increase when I run the look several more times. Same for me, but of course only if I exit the "async" context between runs > Why are you creating so many SSLContext objects any way? It's very inefficient and really not necessary. The original issue was observed in a very long running process (months), that occasionally needed a context and it was convenient to just create one every time (actually it creates an AsyncClient context https://github.com/encode/httpx/issues/978) even if it is relatively inefficient, it didn't really matter, but memory usage unexpectedly slowly grew to 1 GB which was very unexpected |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:31 | admin | set | github: 84904 |
2020-05-22 10:32:24 | Recursing | set | messages: + msg369586 |
2020-05-22 10:17:45 | christian.heimes | set | assignee: christian.heimes -> messages: + msg369584 |
2020-05-22 10:09:47 | christian.heimes | set | messages: + msg369583 |
2020-05-22 09:55:53 | Recursing | set | messages: + msg369582 |
2020-05-22 09:40:48 | christian.heimes | set | messages: + msg369578 |
2020-05-22 09:29:10 | Recursing | create |