This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Library multiprocess leaks named resources.
Type: resource usage Stage: patch review
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BarkingBad, arhadthedev, benjamin.peterson, davin, milestonejxd, pitrou, sbt, yselivanov
Priority: normal Keywords: patch

Created on 2022-01-15 18:04 by milestonejxd, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
screen.png milestonejxd, 2022-01-24 09:31 screenshot
Pull Requests
URL Status Linked Edit
PR 30617 open milestonejxd, 2022-01-15 18:38
Messages (5)
msg410654 - (view) Author: XD Trol (milestonejxd) * Date: 2022-01-15 18:04
Repo is the standard tool that Google uses to build Android, Chrome OS, Chromium, etc, written in Python. 

Many Repo users have encountered resource leak warnings with Python 3.9. 
https://bugs.chromium.org/p/gerrit/issues/detail?id=14934&q=component%3Arepo&can=5

I did some work and found that the problem is not caused by the code of Repo, but a bug of the Python library, multiprocess.

To make it simple, the Python script below leaks named resource even when exiting normally. (And be unlinked by resource_tracker with a warning. )

```
import multiprocessing as mp

global_resource = mp.Semaphore()

def submain(): pass

if __name__ == '__main__':
    p = mp.Process(target=submain)
    p.start()
    p.join()
```

Tested on macOS with Python 3.9.7
> python test.py
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown.

This bug will 100% reproduce when then main module uses named resources as global variables and uses `spawn` context, which is the case of Repo on macOS.

This is caused by multiprocess::BaseProcess::_bootstrap.

When a new process is started with multiprocessing.Process.start() in `spawn` context.
1. The main module is reloaded in the subprocess (for pickle) in multiprocessing::spawn::_main.
2. Named resources (such as the semaphore above) in the main module resister their _cleanup into multiprocessing::util::_finalizer_registry, which unlink themselves.
3. multiprocess::BaseProcess::_bootstrap then clears _finalizer_registry.

When a subprocess is spawned, it is no need to clear util::_finalizer_registry (and no need to call util::_run_after_forkers). Disable clearing _finalizer_registry (and disable call to _run_after_forkers) should fix this bug without breaking anything else.

And I uploaded a MR.
msg410657 - (view) Author: Oleg Iarygin (arhadthedev) * Date: 2022-01-15 19:38
I added core devs related to multiprocessing into a nosy list so they got a notification and the PR will be evaluated and merged faster. FYI, the devs are Davin Potts and Antoine Pitrou (as per <https://devguide.python.org/experts/>).
msg410853 - (view) Author: XD Trol (milestonejxd) * Date: 2022-01-18 11:34
loop file owner.
msg411456 - (view) Author: XD Trol (milestonejxd) * Date: 2022-01-24 09:31
update,

Confirmed to affect all versions above 3.8+.

It's really annoying. 

Command line tools like Repo get a warning after every (success) command. 
And mess up with shell.

-------------------------------------------------------------------

$ repo status .
project ksbox/                                  branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo sync .
Fetching: 100% (1/1), done in 1.184s
Garbage collecting: 100% (1/1), done in 0.013s
repo sync has finished successfully.
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

$
$
$
$ repo status .
project ksbox/                                  branch dev
$ /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
msg413504 - (view) Author: Andrzej Ratajczak (BarkingBad) Date: 2022-02-18 19:39
Also hitting this problem. Would be grateful for fixing it, since there is already PR created
History
Date User Action Args
2022-04-11 14:59:54adminsetgithub: 90549
2022-02-18 19:39:09BarkingBadsetnosy: + BarkingBad
messages: + msg413504
2022-01-24 09:31:45milestonejxdsetfiles: + screen.png

messages: + msg411456
versions: + Python 3.8, Python 3.10, Python 3.11
2022-01-19 02:25:49milestonejxdsetnosy: + yselivanov
2022-01-18 11:34:34milestonejxdsetnosy: + benjamin.peterson, sbt
messages: + msg410853
2022-01-15 19:38:47arhadthedevsetnosy: + pitrou, davin, arhadthedev
messages: + msg410657
2022-01-15 18:38:06milestonejxdsetkeywords: + patch
stage: patch review
pull_requests: + pull_request28821
2022-01-15 18:04:57milestonejxdcreate