classification
Title: Thread pool return ref hold memory
Type: resource usage Stage: resolved
Components: asyncio Versions: Python 3.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Tianshu Gao, asvetlov, pablogsal, yselivanov
Priority: normal Keywords:

Created on 2019-08-21 20:55 by Tianshu Gao, last changed 2019-09-09 10:52 by asvetlov. This issue is now closed.

Messages (2)
msg350108 - (view) Author: Tianshu Gao (Tianshu Gao) Date: 2019-08-21 20:55
This is very similar to issue35715. But this is happen for thread.

After the func in thread finished, the memory is still hold and accumulate.

import asyncio
import time
import concurrent
import threading

loop = asyncio.get_event_loop()

def prepare_a_giant_list():
    m = []
    for i in range(1000*1000):
        m.append("There's a fat fox jump over a sheep" + str(i))

    th_num = threading.active_count()
    print("Thread number is {}".format(th_num))
    return m

@asyncio.coroutine
def main():
    global loop
    global counter
    async_executor = concurrent.futures.ThreadPoolExecutor(max_workers=20)
    loop.run_in_executor(async_executor, prepare_a_giant_list)
    time.sleep(15)
    loop.run_in_executor(async_executor, prepare_a_giant_list)
    time.sleep(15)
    loop.run_in_executor(async_executor, prepare_a_giant_list)
    time.sleep(15)
    loop.run_in_executor(async_executor, prepare_a_giant_list)
    time.sleep(15)

if __name__ == "__main__":
    loop.run_until_complete(main())
    loop.close()
msg350523 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-08-26 10:51
In asyncio code please use non-blocking code and await a future returned by run_until_complete.

The following code doesn't leak:

import asyncio
import concurrent
import threading


def prepare_a_giant_list():
    m = []
    for i in range(1000*1000):
        m.append("There's a fat fox jump over a sheep" + str(i))

    th_num = threading.active_count()
    print("Thread number is {}".format(th_num))
    return m


async def main():
    loop = asyncio.get_running_loop()
    async_executor = concurrent.futures.ThreadPoolExecutor(max_workers=20)
    await loop.run_in_executor(async_executor, prepare_a_giant_list)
    await asyncio.sleep(15)
    await loop.run_in_executor(async_executor, prepare_a_giant_list)
    await asyncio.sleep(15)
    await loop.run_in_executor(async_executor, prepare_a_giant_list)
    await asyncio.sleep(15)
    await loop.run_in_executor(async_executor, prepare_a_giant_list)
    await asyncio.sleep(15)


if __name__ == "__main__":
    asyncio.run(main())
History
Date User Action Args
2019-09-09 10:52:43asvetlovsetstatus: open -> closed
resolution: not a bug
stage: resolved
2019-08-26 10:51:15asvetlovsetmessages: + msg350523
2019-08-21 20:58:40rhettingersetnosy: + pablogsal
2019-08-21 20:55:03Tianshu Gaocreate