This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author aeros
Recipients aeros, asvetlov, benjamin.peterson, vstinner, yselivanov
Date 2019-11-15.00:41:10
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1573778471.04.0.421523294016.issue38591@roundup.psfhosted.org>
In-reply-to
Content
> You have to account also for the thread stack size. I suggest to look at RSS memory instead.

Ah, good point. I believe get_size() only accounts for the memory usage of the thread object, not the amount allocated in physical memory from the thread stack. Thanks for the clarification. 

> I measured the RSS memory per thread: it's around 13.2 kB/thread. Oh, that's way lower than what I expected.

On Python 3.8 and Linux kernel 5.3.8, I received the following result:

# Starting mem
VmRSS:      8408 kB
# After initializing and starting 1k threads:
VmRSS:     21632 kB

~13224kB for 1k threads, which reflects the ~13.2kB/thread estimate. 

Also, as a sidenote, I think you could remove the "for thread in threads: thread.started_event.wait()" portion for our purposes. IIUC, waiting on the threading.Event objects wouldn't affect memory usage.

> To be clear: I mean that FastChildWatcher is safe only if all process's code spaws subprocesses by FastChildWatcher.
> If ProcessPoolExecutor or direct subprocess calls are used the watcher is unsafe.
> If some C extension spawns new processes on its own (e.g. in a separate thread) -- the watcher is unsafe.

> I just think that this particular watcher is too dangerous.

So are we at least in agreement for starting with deprecating FastChildWatcher? If a server is incredibly tight on memory and it can't spare ~13.2kB/thread, SafeChildWatcher would be an alternative to ThreadedChildWatcher.

Personally, I still think this amount is negligible for most production servers, and that we can reasonably deprecate SafeChildWatcher as well. But I can start with FastChildWatcher.
History
Date User Action Args
2019-11-15 00:41:11aerossetrecipients: + aeros, vstinner, benjamin.peterson, asvetlov, yselivanov
2019-11-15 00:41:11aerossetmessageid: <1573778471.04.0.421523294016.issue38591@roundup.psfhosted.org>
2019-11-15 00:41:11aeroslinkissue38591 messages
2019-11-15 00:41:10aeroscreate