This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author pitrou
Recipients MultiSosnooley, bquinlan, pitrou
Date 2018-05-08.17:45:33
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1525801533.5.0.682650639539.issue33444@psf.upfronthosting.co.za>
In-reply-to
Content
Yes, this looks surprising, but there is no memory leak here, just memory fragmentation in the glibc allocator.  This is the program I used to diagnose it:
https://gist.github.com/pitrou/6c5310d4c721436165666044e3c31158

At the end the program prints glibc allocator stats as returned by mallinfo() (see http://man7.org/linux/man-pages/man3/mallinfo.3.html). On my system, the process takes 480 MB RSS and "fordblks" (the total number of bytes in free blocks) is 478 MB. However, "keepcost" (the releasable free space) is only 30 MB.  The rest is probably interspersed with internal interpreter structures that have stayed alive.

The fragmentation seems to depend on the number of threads.  If you start the executor with only one thread, memory consumption is much lower.  This makes sense: by ensuring all operations happen in order with little concurrency, we minimize the chance that short-lived data gets interspersed with longer-lived data.
History
Date User Action Args
2018-05-08 17:45:33pitrousetrecipients: + pitrou, bquinlan, MultiSosnooley
2018-05-08 17:45:33pitrousetmessageid: <1525801533.5.0.682650639539.issue33444@psf.upfronthosting.co.za>
2018-05-08 17:45:33pitroulinkissue33444 messages
2018-05-08 17:45:33pitroucreate