Message410654
Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python.
Many Repo users have encountered resource leak warnings with Python 3.9.
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5
I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.
To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )
```
import multiprocessing as mp
global_resource = mp.Semaphore()
def submain(): pass
if __name__ == '__main__':
p = mp.Process(target=submain)
p.start()
p.join()
```
Tested on macOS with Python 3.9.7
> python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.
This bug will 100% reproduce when then main module uses named resources as global variables and uses `spawn` context, which is the case of Repo on macOS.
This is caused by multiprocess::BaseProcess::_bootstrap.
When a new process is started with multiprocessing.Process.start() in `spawn` context.
1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.
When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.
And I uploaded a MR. |
|
Date |
User |
Action |
Args |
2022-01-15 18:04:57 | milestonejxd | set | recipients:
+ milestonejxd |
2022-01-15 18:04:57 | milestonejxd | set | messageid: <1642269897.29.0.322439501259.issue46391@roundup.psfhosted.org> |
2022-01-15 18:04:57 | milestonejxd | link | issue46391 messages |
2022-01-15 18:04:57 | milestonejxd | create | |
|