While writing a program using the multiprocessing library I stumbled upon what appears to be a bug with how different platforms deal with private methods.

When a class has a private method which is the target for a multiprocessing process, this name is correctly resolved on Linux (20.04.1-Ubuntu running Python 3.8.10) but fails to be resolved correctly on MacOS (Python 3.8.2 and 3.8.8) or Windows 10 (Python 3.9.6).


import multiprocessing

class Test(object):
    def __init__(self):
        self.a = 1
        self._b = 2
        self.__c = 3
        self.run1()
        self.run2()
    def _test1(self, conn):
        conn.send(self._b)
    def __test2(self, conn):
        conn.send(self.__c)
    def run1(self):
        print("Running self._test1()")
        parent, child = multiprocessing.Pipe()
        process = multiprocessing.Process(target=self._test1, args=(child, ))
        process.start()
        print(parent.recv())
        process.join()
    def run2(self):
        print("Running self.__test2()")
        parent, child = multiprocessing.Pipe()
        process = multiprocessing.Process(target=self.__test2, args=(child, ))
        process.start()
        print(parent.recv())
        process.join()

if __name__ == "__main__":
    t = Test()


On Linux, this has the intended behavior of printing:
Running self._test1()
2
Running self.__test2()
3

However, on Windows 10, this results in an Exception being raised:
Running self._test1()
2
Running self.__test2()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: 'Test' object has no attribute '__test2'

A similar exception is also raised on MacOS for this code.


It would therefore appear that there is different behavior for resolving class attributes starting with `__` on different platforms (at least within multiprocessing). It is my understanding that because multiprocessing.Process is called within the class, the private method should be within scope and so should resolve correctly.
I'm aware that Python doesn't have strict private methods, and instead renames them (Test.__test2 becomes Test._Test__test2) - explaining why on Windows it cannot find the attribute with that name. 

My question really is, which platform is correct here, and is the inconsistency intentional? I'd suggest Linux is most correct here as the process is spawned from within the object so the method should be in scope, but either way, the inconsistency between platforms may cause some unintended issues.

The issue seems to be that there was a change between 3.7.7 and 3.8.6, - multiprocessing started using pickle dump / load which cannot handle double underscore methods.

See #33007 for a reproducer with pickle.

I'm using MacOS and this multiprocessing example in OP works fine in 3.7.7 and raises the same error as OP in 3.8.6. So from my side at least it doesn't appear to be a cross-platform issue (but I'm not sure why it worked on Linux on 3.8.10 for OP).

I've also reproduced it on the latest 3.11.

So this appears to be a regression and if there's an easy fix, it's probably worth fixing.

More details and a suggested fix here: #37852 .

If we are not going to fix this issue in pickling at this time, I think it makes sense to raise an error with a good, clear explanation in multiprocessing.Process(target=self.__method), which is affected by this and caused some confusion in this issue: #44675 .

Josh: adding you since you reported this issue in another ticket, hope that's okay.