This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author milestonejxd
Recipients milestonejxd
Date 2022-01-15.18:04:57
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1642269897.29.0.322439501259.issue46391@roundup.psfhosted.org>
In-reply-to
Content
Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python. 

Many Repo users have encountered resource leak warnings with Python 3.9. 
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )

```
import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()
```

Tested on macOS with Python 3.9.7
> python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.

This bug will 100% reproduce when then main module uses named resources as global variables and uses `spawn` context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in `spawn` context.
1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.

And I uploaded a MR.
History
Date User Action Args
2022-01-15 18:04:57milestonejxdsetrecipients: + milestonejxd
2022-01-15 18:04:57milestonejxdsetmessageid: <1642269897.29.0.322439501259.issue46391@roundup.psfhosted.org>
2022-01-15 18:04:57milestonejxdlinkissue46391 messages
2022-01-15 18:04:57milestonejxdcreate