Message356639
> You have to account also for the thread stack size. I suggest to look at RSS memory instead.
Ah, good point. I believe get_size() only accounts for the memory usage of the thread object, not the amount allocated in physical memory from the thread stack. Thanks for the clarification.
> I measured the RSS memory per thread: it's around 13.2 kB/thread. Oh, that's way lower than what I expected.
On Python 3.8 and Linux kernel 5.3.8, I received the following result:
# Starting mem
VmRSS: 8408 kB
# After initializing and starting 1k threads:
VmRSS: 21632 kB
~13224kB for 1k threads, which reflects the ~13.2kB/thread estimate.
Also, as a sidenote, I think you could remove the "for thread in threads: thread.started_event.wait()" portion for our purposes. IIUC, waiting on the threading.Event objects wouldn't affect memory usage.
> To be clear: I mean that FastChildWatcher is safe only if all process's code spaws subprocesses by FastChildWatcher.
> If ProcessPoolExecutor or direct subprocess calls are used the watcher is unsafe.
> If some C extension spawns new processes on its own (e.g. in a separate thread) -- the watcher is unsafe.
> I just think that this particular watcher is too dangerous.
So are we at least in agreement for starting with deprecating FastChildWatcher? If a server is incredibly tight on memory and it can't spare ~13.2kB/thread, SafeChildWatcher would be an alternative to ThreadedChildWatcher.
Personally, I still think this amount is negligible for most production servers, and that we can reasonably deprecate SafeChildWatcher as well. But I can start with FastChildWatcher. |
|
Date |
User |
Action |
Args |
2019-11-15 00:41:11 | aeros | set | recipients:
+ aeros, vstinner, benjamin.peterson, asvetlov, yselivanov |
2019-11-15 00:41:11 | aeros | set | messageid: <1573778471.04.0.421523294016.issue38591@roundup.psfhosted.org> |
2019-11-15 00:41:11 | aeros | link | issue38591 messages |
2019-11-15 00:41:10 | aeros | create | |
|