classification
Title: Memory leak in asyncio server
Type: behavior Stage:
Components: asyncio Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: EmperorBale, asvetlov, valid22, yselivanov
Priority: normal Keywords:

Created on 2020-09-16 05:28 by EmperorBale, last changed 2020-09-17 16:52 by EmperorBale.

Files
File name Uploaded Description Edit
test_server.py EmperorBale, 2020-09-16 05:41
Messages (3)
msg376975 - (view) Author: Bale (EmperorBale) Date: 2020-09-16 05:41
In the file I attached below is a pretty simple asyncio server, it is identical to the echo server on the documentation except it does not disconnect after echoing. 

When a connection is received, it creates an object called "test object" and the only reference to that object is on the local scope. If all is working properly, when the handler function returns the test object should also be deleted, but I noticed the __del__ method is not being called, indicating python is not deleting it. It seems like a reference to the frame of the handler is being kept in a list (not sure where) and it is not being removed from it.

Here is example code of how to trigger the bug:
s = socket.socket();s.connect(("127.0.0.1", 8888));s.send(b"test\0")
s.close()

By sending data and closing afterwards WITHOUT using s.recv() causes the issue, what's interesting is that if we call s.close() immediately after sending, the object is deleted by the server and everything works as intended. Calling s.close() a few seconds after, causes the bug to happen. It should also be noted that if I call s.recv() before s.close(), the bug does not happen.

What's more interesting is that when I run the above code in my python3.5 interpreter the bug does not get triggered, no matter what I do. The bug does also not happen when I run it in a python3.8 interpreter on my windows computer. When I run on the code on ubuntu on a python3.8 interpreter, the bug is triggered.
msg376976 - (view) Author: Bale (EmperorBale) Date: 2020-09-16 05:55
Also, if this is intended behavior and there is something that I am missing, you should really make it more clear in the documentation. In every project I found on github that uses asyncio.start_server, the bug exists
msg377062 - (view) Author: Bale (EmperorBale) Date: 2020-09-17 16:52
Ok since nobody is replying, the bug seems to be that if the asyncio server sends data, and I do not use s.recv() to clear the buffer, and I use s.close() instead, a reference to handle_echo is kept and is never deleted.

Using gc.collect() does delete the leaked object. 

The socket that you use to connect to the server seems to also determine whether the bug works, like how I can not do it when I create the connection on my windows computer, but the bug is triggered when I do it on my ubuntu server. I should also mention that I am running the asyncio server on my ubuntu server, on python 3.8.

Here is output of the bug happening.
FROM CLIENT ON MY WINDOWS COMPUTER ON PY3.8:
>>> s = socket.socket()
>>> s.connect(("x.x.x.x", 8888))
>>> s.send(b"test")
4
# Wait for server to send data, but dont use s.recv(), just assume
>>> s.close()
Server output:
Received 'test' from ('x.x.x.x', 42241)
Send: 'test'
connection lost
DELETED OBJECT

After connection was lost, DELETED OBJECT was printed, so everything worked fine. But when I create the socket in a different environment, such as on ubuntu python 3.8:
Client side code (exact same as before):
>>> s = socket.socket()
>>> s.connect(("x.x.x.x", 8888))
>>> s.send(b"test")
4
# Wait for server to send data, but dont use s.recv(), just assume
>>> s.close()
Server side output:
Received 'test' from ('x.x.x.x', 42249)
Send: 'test'
connection lost

We can see that connection lost was printed, but DELETED OBJECT was not. This is evidence that there is a memory leak bug with asyncio.start_server
History
Date User Action Args
2020-09-17 16:52:12EmperorBalesetmessages: + msg377062
2020-09-16 05:55:24EmperorBalesetmessages: + msg376976
2020-09-16 05:41:28EmperorBalesetfiles: + test_server.py

messages: + msg376975
2020-09-16 05:28:08EmperorBalecreate