classification
Title: Cross-platform issues with private methods and multiprocessing
Type: behavior Stage:
Components: Library (Lib), macOS, Windows Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ned.deily, paul.moore, ronaldoussoren, steve.dower, tim.golden, ymerej, zach.ware
Priority: normal Keywords:

Created on 2021-07-19 15:49 by ymerej, last changed 2021-07-19 15:49 by ymerej.

Messages (1)
msg397810 - (view) Author: Jeremy (ymerej) Date: 2021-07-19 15:49
While writing a program using the multiprocessing library I stumbled upon what appears to be a bug with how different platforms deal with private methods.

When a class has a private method which is the target for a multiprocessing process, this name is correctly resolved on Linux (20.04.1-Ubuntu running Python 3.8.10) but fails to be resolved correctly on MacOS (Python 3.8.2 and 3.8.8) or Windows 10 (Python 3.9.6).


import multiprocessing

class Test(object):
    def __init__(self):
        self.a = 1
        self._b = 2
        self.__c = 3
        self.run1()
        self.run2()
    def _test1(self, conn):
        conn.send(self._b)
    def __test2(self, conn):
        conn.send(self.__c)
    def run1(self):
        print("Running self._test1()")
        parent, child = multiprocessing.Pipe()
        process = multiprocessing.Process(target=self._test1, args=(child, ))
        process.start()
        print(parent.recv())
        process.join()
    def run2(self):
        print("Running self.__test2()")
        parent, child = multiprocessing.Pipe()
        process = multiprocessing.Process(target=self.__test2, args=(child, ))
        process.start()
        print(parent.recv())
        process.join()

if __name__ == "__main__":
    t = Test()


On Linux, this has the intended behavior of printing:
Running self._test1()
2
Running self.__test2()
3

However, on Windows 10, this results in an Exception being raised:
Running self._test1()
2
Running self.__test2()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: 'Test' object has no attribute '__test2'

A similar exception is also raised on MacOS for this code.


It would therefore appear that there is different behavior for resolving class attributes starting with `__` on different platforms (at least within multiprocessing). It is my understanding that because multiprocessing.Process is called within the class, the private method should be within scope and so should resolve correctly.
I'm aware that Python doesn't have strict private methods, and instead renames them (Test.__test2 becomes Test._Test__test2) - explaining why on Windows it cannot find the attribute with that name. 

My question really is, which platform is correct here, and is the inconsistency intentional? I'd suggest Linux is most correct here as the process is spawned from within the object so the method should be in scope, but either way, the inconsistency between platforms may cause some unintended issues.
History
Date User Action Args
2021-07-19 15:49:31ymerejcreate