This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Lambda can't be pickled with "spawn" and only "fork"
Type: behavior Stage: resolved
Components: Interpreter Core, macOS Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: kyle.smith, ned.deily, ronaldoussoren, steven.daprano
Priority: normal Keywords:

Created on 2022-02-27 02:58 by kyle.smith, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (9)
msg414137 - (view) Author: Kyle Smith (kyle.smith) Date: 2022-02-27 02:58
The below code works on versions 3.5.2 to 3.8.10. Higher versions tested, such as 3.9.12 and 3.10.2 result in the error:
 "AttributeError: Can't pickle local object".


from multiprocessing import Lock
from multiprocessing.managers import AcquirerProxy, BaseManager, DictProxy

def get_shared_state(host, port, key):
    shared_dict = {}
    shared_lock = Lock()
    manager = BaseManager((host, port), key)
    manager.register("get_dict", lambda: shared_dict, DictProxy)
    manager.register("get_lock", lambda: shared_lock, AcquirerProxy)
    try:
        manager.get_server()
        manager.start()
    except OSError:  # Address already in use
        manager.connect()
    return manager.get_dict(), manager.get_lock()

HOST = "127.0.0.1"
PORT = 35791
KEY = b"secret"
shared_dict, shared_lock = get_shared_state(HOST, PORT, KEY)

shared_dict["number"] = 0
shared_dict["text"] = "Hello World"


This code was pulled from this article: https://stackoverflow.com/questions/57734298/how-can-i-provide-shared-state-to-my-flask-app-with-multiple-workers-without-dep/57810915#57810915


I looked around and couldn't find any open or closed bugs for this, so I'm sorry in advance if this is new expected behavior.
msg414139 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2022-02-27 03:45
Works for me in Python 3.10.0 on Linux.

After running your code, I get shared_dict is a DictProxy:

>>> shared_dict
<DictProxy object, typeid 'get_dict' at 0x7f092ccd6530>
>>> list(shared_dict.items())
[('number', 0), ('text', 'Hello World')]

and shared_lock an AcquirerProxy object.

Please double-check that the code you posted is the actual code that is failing, and copy and paste the full traceback you receive, not just a one-line summary.

Even if the error is reproducible, I doubt that the cause is what you state in the title of this issue:

BaseManager.register no longer supports lambda callable

Lambdas are just functions, they aren't a different type of callable. So the register method cannot distinguish between a lambda argument written directly in place, and a named def defined earlier then passed by name. So whatever error might be happening on your system, I doubt it has anything to do with the use of lambda syntax
msg414153 - (view) Author: Kyle Smith (kyle.smith) Date: 2022-02-27 13:56
Interesting. I did try between MacOS (12.2) and Ubuntu since I have servers with older python versions. For scoping then, this appears to be related only to MacOS since you were able to confirm it working correctly on Linux. 

The code is the same, but here's the entire traceback. All below examples were run on MacOS:

bash-3.2$ python3 --version
Python 3.9.6
bash-3.2$ python3 main.py
Traceback (most recent call last):
  File "/Volumes/Workspace/co/router/main.py", line 21, in <module>
    shared_dict, shared_lock = get_shared_state(HOST, PORT, KEY)
  File "/Volumes/Workspace/co/router/main.py", line 12, in get_shared_state
    manager.start()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/managers.py", line 553, in start
    self._process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'get_shared_state.<locals>.<lambda>'


Running this with 3.7.9 works:
bash-3.2$ python3.7 --version
Python 3.7.9
bash-3.2$ python3.7 main.py
{'number': 0, 'text': 'Hello World'}


If it's of interest, 3.10.2 is also failing on Mac.

bash-3.2$ /usr/local/opt/python\@3.10/bin/python3 --version
Python 3.10.2
bash-3.2$ /usr/local/opt/python\@3.10/bin/python3 main.py
Traceback (most recent call last):
  File "/Volumes/Workspace/co/router/main.py", line 21, in <module>
    shared_dict, shared_lock = get_shared_state(HOST, PORT, KEY)
  File "/Volumes/Workspace/co/router/main.py", line 12, in get_shared_state
    manager.start()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/managers.py", line 562, in start
    self._process.start()
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/python@3.10/3.10.2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'get_shared_state.<locals>.<lambda>'
msg414155 - (view) Author: Kyle Smith (kyle.smith) Date: 2022-02-27 14:07
Since you were able to help me scope that this is probably an MacOS specific issue, I was able to find this report:
https://github.com/pytest-dev/pytest-flask/issues/104

Setting `multiprocessing.set_start_method("fork")` _BEFORE_ the code is called resolves this.

I don't understand enough of MP to understand why this is a problem, and why I need to do this, but alas, at least I have a path forward. 


Since this behavior changed, and it _doesn't_ work cleanly anymore with MacOS, I would say this is a bug (from my casual understanding of python), because it's mentioned in the documentation that using "fork" can be problematic and lead to crashes outlined in bpo-33725.
> Changed in version 3.8: On macOS, the spawn start method is now the default.
> The fork start method should be considered unsafe as it can lead to crashes 
> of the subprocess. See bpo-33725.
msg414196 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2022-02-28 13:54
The default start method for multirprocessing was changed from "fork" to "spawn" on macOS. This was done because the "fork" method can easily be triggered into causing hard crashes (on macOS), in particular when the parent proces has called higher-level systemen APIs.

The "spawn" method requires pickling the data and callable passed to the child proces, and that's not supported for lambda's.
msg414253 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2022-03-01 10:06
This is not a bug in CPython, at best this can be a feature request to make it possible to pickle lambda's (which IMHO is unlikely to happen).

The "fork" and "spawn" start methods result in different behaviour in how data is copied into the new worker proces and there's nothing we can do about this. Note that the same difference in behaviour can be observed on Windows, which does not support the "fork" method at all.
msg414255 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2022-03-01 10:27
> The "spawn" method requires pickling the data and callable passed to 
> the child proces, and that's not supported for lambda's.

Ah, today I learned something. Kyle, looks like you were right about it 
being due to the lambdas.
msg414257 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2022-03-01 11:16
Oops, replying by email reopens the ticket. Back to pending you go!
msg414270 - (view) Author: Kyle Smith (kyle.smith) Date: 2022-03-01 14:54
I think that's fair, thanks for the conversation at least. I understand python mp a little bit more now...
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 91027
2022-03-01 14:54:14kyle.smithsetstatus: open -> closed
2022-03-01 14:54:11kyle.smithsetstatus: pending -> open

messages: + msg414270
2022-03-01 11:16:39steven.dapranosetstatus: open -> pending

messages: + msg414257
2022-03-01 10:27:29steven.dapranosetstatus: pending -> open

messages: + msg414255
2022-03-01 10:06:34ronaldoussorensetstatus: open -> pending
resolution: not a bug
messages: + msg414253

stage: resolved
2022-02-28 13:54:49ronaldoussorensetnosy: + ronaldoussoren, ned.deily
messages: + msg414196
components: + macOS
2022-02-27 20:25:43kyle.smithsettitle: BaseManager.register no longer supports lambda callable 3.8.12+ -> Lambda can't be pickled with "spawn" and only "fork"
2022-02-27 14:07:17kyle.smithsetmessages: + msg414155
2022-02-27 13:56:09kyle.smithsetmessages: + msg414153
2022-02-27 03:45:24steven.dapranosetnosy: + steven.daprano
messages: + msg414139
2022-02-27 02:58:03kyle.smithcreate