classification
Title: Cross-platform issues with private methods and multiprocessing
Type: behavior Stage:
Components: Library (Lib), macOS, Windows Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: andrei.avk, josh.r, ned.deily, paul.moore, ronaldoussoren, steve.dower, tim.golden, ymerej, zach.ware
Priority: normal Keywords:

Created on 2021-07-19 15:49 by ymerej, last changed 2021-11-29 14:23 by andrei.avk.

Messages (4)
msg397810 - (view) Author: Jeremy (ymerej) Date: 2021-07-19 15:49
While writing a program using the multiprocessing library I stumbled upon what appears to be a bug with how different platforms deal with private methods.

When a class has a private method which is the target for a multiprocessing process, this name is correctly resolved on Linux (20.04.1-Ubuntu running Python 3.8.10) but fails to be resolved correctly on MacOS (Python 3.8.2 and 3.8.8) or Windows 10 (Python 3.9.6).


import multiprocessing

class Test(object):
    def __init__(self):
        self.a = 1
        self._b = 2
        self.__c = 3
        self.run1()
        self.run2()
    def _test1(self, conn):
        conn.send(self._b)
    def __test2(self, conn):
        conn.send(self.__c)
    def run1(self):
        print("Running self._test1()")
        parent, child = multiprocessing.Pipe()
        process = multiprocessing.Process(target=self._test1, args=(child, ))
        process.start()
        print(parent.recv())
        process.join()
    def run2(self):
        print("Running self.__test2()")
        parent, child = multiprocessing.Pipe()
        process = multiprocessing.Process(target=self.__test2, args=(child, ))
        process.start()
        print(parent.recv())
        process.join()

if __name__ == "__main__":
    t = Test()


On Linux, this has the intended behavior of printing:
Running self._test1()
2
Running self.__test2()
3

However, on Windows 10, this results in an Exception being raised:
Running self._test1()
2
Running self.__test2()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\<USER>\AppData\Local\Programs\Python\Python39\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
AttributeError: 'Test' object has no attribute '__test2'

A similar exception is also raised on MacOS for this code.


It would therefore appear that there is different behavior for resolving class attributes starting with `__` on different platforms (at least within multiprocessing). It is my understanding that because multiprocessing.Process is called within the class, the private method should be within scope and so should resolve correctly.
I'm aware that Python doesn't have strict private methods, and instead renames them (Test.__test2 becomes Test._Test__test2) - explaining why on Windows it cannot find the attribute with that name. 

My question really is, which platform is correct here, and is the inconsistency intentional? I'd suggest Linux is most correct here as the process is spawned from within the object so the method should be in scope, but either way, the inconsistency between platforms may cause some unintended issues.
msg407246 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-11-29 06:04
The issue seems to be that there was a change between 3.7.7 and 3.8.6, - multiprocessing started using pickle dump / load which cannot handle double underscore methods.

See #33007 for a reproducer with pickle.

I'm using MacOS and this multiprocessing example in OP works fine in 3.7.7 and raises the same error as OP in 3.8.6. So from my side at least it doesn't appear to be a cross-platform issue (but I'm not sure why it worked on Linux on 3.8.10 for OP).

I've also reproduced it on the latest 3.11.

So this appears to be a regression and if there's an easy fix, it's probably worth fixing.
msg407247 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-11-29 06:10
More details and a suggested fix here: #37852 .
msg407278 - (view) Author: Andrei Kulakov (andrei.avk) * (Python triager) Date: 2021-11-29 14:23
If we are not going to fix this issue in pickling at this time, I think it makes sense to raise an error with a good, clear explanation in multiprocessing.Process(target=self.__method), which is affected by this and caused some confusion in this issue: #44675 .

Josh: adding you since you reported this issue in another ticket, hope that's okay.
History
Date User Action Args
2021-11-29 14:23:42andrei.avksetnosy: + josh.r
messages: + msg407278
2021-11-29 06:10:12andrei.avksetmessages: + msg407247
2021-11-29 06:04:39andrei.avksetnosy: + andrei.avk
messages: + msg407246
2021-07-19 15:49:31ymerejcreate