As of CPython 3.8.0b1:

If one pickles an object that is defined in the __main__ module, sends it to a subinterpreter as bytes, and then tries unpickling it there, it fails saying that __main__ doesn't define it.


import _xxsubinterpreters as interpreters
import pickle


class C:
    pass


c = C()

interp_id = interpreters.create()
c_bytes = pickle.dumps(c)
interpreters.run_string(
    interp_id,
    "import pickle; pickle.loads(c_bytes)",
    shared={"c_bytes": c_bytes},
)


If the above is executed directly with the python command-line, it fails. If it's imported from another module, it works.
One would expected behaviour compatible with sub-processes spawned with the spawn method, where the__main__ of the parent process is visible to the subprocess too.

Workarounds:
1 - define everything that must be pickled in an imported module
2 - use CloudPickle

Just to move the conversation from the subinterpreters project repo to here...

I'm going to take a look at how this is done by subprocess using the example provided by Guido:

import os
from concurrent.futures import ProcessPoolExecutor
from multiprocessing import get_context

class C:
    def __getstate__(self):
        print("pickled in %d" % os.getpid())
        return {}

    def __setstate__(self, state):
        print("unpickled in %d" % os.getpid())

    def hello(self):
        print("Hello world")


if __name__ == "__main__":
    with ProcessPoolExecutor(mp_context=get_context("spawn")) as ex:
        ex.submit(C().hello).result()

Output:

pickled in 23480
unpickled in 23485
Hello world

The relevant code for the multiprocessing example seems to be in Lib/multiprocessing/spawn.py. I think I get what it's doing, but I'm not sure whether we actually need something similar for subinterpreters. Any thoughts @eric.snow?

Yeah, the case of pickle + __main__ is an awkward one. [1]  However, the approach taken by multiprocessing isn't the right one for subinterpreters.

Multiprocessing operates under 2 design points that do not apply to subinterpreters:

* every process is running in the same __main__ module (sans the "script" part)
* pickle is a critical part of data-passing

For spawn, multiprocessing automatically runs the original __main__ module in each newly spawned process. [2]  Note that it runs it there with __name__ set to __mp_main__ (rather than __main__), to keep the "script" part from running.  
Subinterpreters could be made to work like this [3] but in reality they are more like processes created using the subprocess module.

I do not expect that to change.  However, there is room for add opt-in support for rerunning __main__ in a subinterpreter, or helpers to accomplish the same.  We can address such opt-in support or helpers in a separate issue later.  For now we are focusing on the fundamentals (at least in the context of PEP 554).


[1] Note that the problem is only with the __main__ module.  For other modules pickle does the right thing.
[2] https://github.com/python/cpython/blob/master/Lib/multiprocessing/spawn.py#L287
[3] I expect we will see subinterpreters supported in the multiprocessing module just like threads are supported.

In the meantime that leaves the workarounds that @crusaderky originally identified.  You could also:

* manually run __main__ in the subinterpreter first (sort of like multiprocessing does automatically); this works because the namespace of __main__ is not reset for each run_string() call
* (for -m) update __module__ of the relevant objects to be the actual module name rather than "__main__"

In the case of that second point, it relates to PEP 499 (which will ensure that the module is added to sys.modules in the -m case).  However, that PEP doesn't say anything about updating __module__ for objects.  I'll bring that up there.  With that solution the problem in this issue would go away.

Note that it won't help for objects in the __main__ of subinterpreters, since they do not correspond to executed modules.  Hmm, maybe it could still work...

Regardless, I'll open issues over on https://github.com/ericsnowcurrently/multi-core-python to track these possible future enhancements.

There's a reason multiprocessing in spawn mode jumps through the hoops that it does: it's the only way to get __main__ pickling to work when you're not forking the entire process.

You also don't want to naively re-run __main__ in the subprocess for the same reason multiprocessing doesn't: doing so would also re-run the script parts, not just the class and function definition parts.

So while I don't think it should be implicit the way it is for multiprocessing spawn mode, I do think it would make sense to offer a way to explicitly opt-in to re-running __main__ in a child interpreter as "__si_main__", aliasing the subinterpreter's __main__ module to that, and also aliasing "__si_main__" to "__main__" in the parent interpreter.