This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Disallow fork in a subinterpreter affects multiprocessing plugin
Type: behavior Stage:
Components: C API Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, franku
Priority: normal Keywords:

Created on 2021-03-16 14:41 by franku, last changed 2022-04-11 14:59 by admin.

Messages (5)
msg388841 - (view) Author: Frank Ueberschar (franku) Date: 2021-03-16 14:41
Related to this issue https://bugs.python.org/issue34651, our Bareos libcloud plugin cannot be run with Python > 3.7. We are using subprocesses in a C-subinterpreter environment. 

Is there a way to circumvent rewriting our code completely?
msg388842 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-16 14:47
Could you please post the error message and either post a minimal example or give us a link to your code?
msg388876 - (view) Author: Frank Ueberschar (franku) Date: 2021-03-16 20:17
Here is part of the gdb backtrace:

[franku@franku py3plug-fd-libcloud]$ sbin/bareos_fd-py3plug-fd-libcloud -f -c etc/bareos       
Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
Python runtime state: initialized

Current thread 0x00007f0dc2ab7640 (most recent call first):
  File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 66 in _launch
  File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib64/python3.9/multiprocessing/context.py", line 277 in _Popen
  File "/usr/lib64/python3.9/multiprocessing/context.py", line 224 in _Popen
  File "/usr/lib64/python3.9/multiprocessing/process.py", line 121 in start
  File "/home/franku/git/bareos/master/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py", line 102 in __init__
  File "/home/franku/git/bareos/master/core/src/plugins/filed/python/libcloud/BareosFdPluginLibcloud.py", line 267 in start_backup_job
  File "/home/franku/git/bareos/master/core/src/plugins/filed/python/pyfiles/BareosFdPluginBaseclass.py", line 285 in handle_plugin_event
  File "/home/franku/git/bareos/master/core/src/plugins/filed/python/pyfiles/BareosFdWrapper.py", line 38 in handle_plugin_event
Fatal Python error: _PyInterpreterState_DeleteExceptMain: not main interpreter
Python runtime state: initialized
msg388877 - (view) Author: Frank Ueberschar (franku) Date: 2021-03-16 20:32
Initialization of the Python interpreter in C-code is here: 
https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/python-fd.cc#L189

The actual Python code that starts the child processes is here:
https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py#L110

Where this is one of the actual Processes: 
https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/bareos_libcloud_api/bucket_explorer.py#L45

Inherited from this base class: 
https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/bareos_libcloud_api/process_base.py#L36
msg388879 - (view) Author: Frank Ueberschar (franku) Date: 2021-03-16 20:48
These lines correspond (due to dirty working copy): 

File "/home/franku/git/bareos/master/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py", line 102 in __init__

The actual Python code that starts the child processes is here:
https://github.com/bareos/bareos/blob/fb76608092ba204ce43cd7c262619e01f9d6a2d6/core/src/plugins/filed/python/libcloud/BareosLibcloudApi.py#L110
History
Date User Action Args
2022-04-11 14:59:42adminsetgithub: 87680
2021-03-16 20:48:00frankusetmessages: + msg388879
2021-03-16 20:32:38frankusetmessages: + msg388877
2021-03-16 20:17:22frankusetmessages: + msg388876
2021-03-16 14:47:35christian.heimessetnosy: + christian.heimes
messages: + msg388842
2021-03-16 14:41:07frankucreate