This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing default fork child process will not free object, which is inherited from parent process
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eleven.xiang
Priority: normal Keywords:

Created on 2021-09-28 07:57 by eleven.xiang, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
test.py eleven.xiang, 2021-09-28 07:57 test script
test1.cpp eleven.xiang, 2021-09-28 07:57 the source file to build the shared library
Messages (8)
msg402758 - (view) Author: eleven xiang (eleven.xiang) Date: 2021-09-28 07:57
Hi, 

Here I did an experiment, and detail as below
1. There is library, which has global object and has its class method
2. Use multiprocessing module to fork the child process, parent process call the global object's method first, and then child process called the global object method again to modify it
3. After that when script exit, parent will free the object, but child process will not free it.
4. if we change the start method to 'spawn', the child process will free it
msg402759 - (view) Author: eleven xiang (eleven.xiang) Date: 2021-09-28 08:00
you could use below command to build test.so from test1.cpp

g++ test1.cpp -shared -fPIC --std=c++11 -o test.so
msg402760 - (view) Author: eleven xiang (eleven.xiang) Date: 2021-09-28 08:15
update test platform

Ubuntu 16.04
msg402769 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-28 12:17
That's the principle of fork. You should use a different spawn method if you don't want to inherit resources inherited from the parent process, like spawn or forkserver:
https://docs.python.org/dev/library/multiprocessing.html#contexts-and-start-methods
msg402818 - (view) Author: eleven xiang (eleven.xiang) Date: 2021-09-29 02:56
I still have some questions about is, so re-open it.

You mean it is the fork principle, but if change the script to use os.fork(), the child process will inherit from parent, and also free it.

You could refer below change:


from ctypes import cdll
import multiprocessing as mp
import os

def foo():
    handle = cdll.LoadLibrary("./test.so")
    handle.IncA()

if __name__ == '__main__':
    foo()
    #p = mp.Process(target = foo, args = ())
    #p.start()
    #print(p.pid)
    #p.join()

    pid = os.fork()
    if pid == 0:
        foo()
    else:
        os.waitpid(pid, 0)
msg402819 - (view) Author: eleven xiang (eleven.xiang) Date: 2021-09-29 03:00
Below is the log:

// parent process call and increase it from 10 to 11.
process 71481, this = 0x7f4f271b5054, m_val = 10, A constructor called!!!
process 71481, this = 0x7f4f271b5054, m_val = 11, INC called!!!

// child process inherit with 11, and then also increase from 11 to 12
process 71482, this = 0x7f4f271b5054, m_val = 12, INC called!!!

// child process free it.
process 71482, this = 0x7f4f271b5054, m_val = 12, A destructor called!!!
// parent process free it.
process 71481, this = 0x7f4f271b5054, m_val = 11, A destructor called!!!
msg402823 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-29 07:27
> I still have some questions about is

The Python bug tracker is not a forum for asking help how to use Python. Please stop reopening this issue.
msg402835 - (view) Author: eleven xiang (eleven.xiang) Date: 2021-09-29 09:27
I am not asking for your help to use python.
What I post here is to want to report a potential issue about multiprocessing module.

This issue is the default start method fork will not free the object, but other methods works normally.
When I change to use os.fork() APIs, it also could free object in child process.

So I think it is a bug that multiprocessing fork method doesn't free object in child process.
History
Date User Action Args
2022-04-11 14:59:50adminsetgithub: 89471
2021-09-29 09:31:53vstinnersetnosy: - vstinner
2021-09-29 09:27:47eleven.xiangsetmessages: + msg402835
2021-09-29 07:27:56vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg402823
2021-09-29 03:01:01eleven.xiangsetresolution: not a bug -> (no value)
2021-09-29 03:00:02eleven.xiangsetmessages: + msg402819
2021-09-29 02:56:18eleven.xiangsetstatus: closed -> open

messages: + msg402818
2021-09-28 12:17:43vstinnersetstatus: open -> closed

nosy: + vstinner
messages: + msg402769

resolution: not a bug
stage: resolved
2021-09-28 08:15:30eleven.xiangsetmessages: + msg402760
2021-09-28 08:00:54eleven.xiangsetmessages: + msg402759
2021-09-28 07:57:49eleven.xiangsetfiles: + test1.cpp
2021-09-28 07:57:03eleven.xiangcreate