This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author vstinner
Recipients The Compiler, Xavion, r.david.murray, vstinner, ztane
Date 2016-09-20.07:43:52
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1474357433.05.0.895180277395.issue28165@psf.upfronthosting.co.za>
In-reply-to
Content
Test-2.py has issues:

* it doesn't call Timer.join()
* it uses a weak synchronization between the main thread and the Timer thread: see msg276990 for an example using Event

If you use a better synchronization code, call timer.join() and call gc.collect(), the memory usage is very stable even after creating more than 100 000 threads + subprocesses.

Xavion: "As I noted before the problem is not with subprocess leaking 4K memory *always*. The issue comes from the fact that subprocess seems to leak 4K memory per individual thread."

I'm unable to reproduce a *leak*.

Test-3.py output:
----
497 thread+subprocess
VmRSS:	    9584 kB
986 thread+subprocess
VmRSS:	    9596 kB      <== after the warmup, the usage seems stable
1490 thread+subprocess
VmRSS:	    9596 kB
(...)
10361 thread+subprocess
VmRSS:	    9596 kB
(...)
30282 thread+subprocess
VmRSS:	    9596 kB
30695 thread+subprocess
VmRSS:	    9672 kB
31160 thread+subprocess
VmRSS:	    9684 kB      <=== memory usage decreases :-)
(...)
60768 thread+subprocess
VmRSS:	    9684 kB
^C
---

If you really want to say that something is wrong: I don't understand why we must call gc.collect() to keep the memory usage stable. But I guess that the GC is not always called for performance.

Without the GC it's not that bad:
---
1083 thread+subprocess
VmRSS:	    9764 kB
2097 thread+subprocess
VmRSS:	    9888 kB
3136 thread+subprocess
VmRSS:	    9888 kB
(...)
11750 thread+subprocess
VmRSS:	    9888 kB
12668 thread+subprocess
VmRSS:	    9940 kB
13705 thread+subprocess
VmRSS:	    9940 kB
(...)
70948 thread+subprocess
VmRSS:	    9940 kB
^C
---

There is no such "4k leak per function call".

I close the issue. It's a bug in your code, not in Python.
History
Date User Action Args
2016-09-20 07:43:53vstinnersetrecipients: + vstinner, r.david.murray, ztane, The Compiler, Xavion
2016-09-20 07:43:53vstinnersetmessageid: <1474357433.05.0.895180277395.issue28165@psf.upfronthosting.co.za>
2016-09-20 07:43:53vstinnerlinkissue28165 messages
2016-09-20 07:43:52vstinnercreate