classification
Title: [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
Type: Stage: patch review
Components: Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.snow Nosy List: Johan Dahlin, emilyemorehouse, eric.snow, nascheme, ncoghlan, pmpp, serhiy.storchaka, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2018-05-22 19:34 by eric.snow, last changed 2019-01-18 20:43 by eric.snow.

Pull Requests
URL Status Linked Edit
PR 9334 closed eric.snow, 2018-09-15 18:25
PR 11617 open eric.snow, 2019-01-18 20:43
PR 11617 open eric.snow, 2019-01-18 20:43
PR 11617 open eric.snow, 2019-01-18 20:43
Messages (6)
msg317333 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2018-05-22 19:34
In order to keep subinterpreters properly isolated, objects
from one interpreter should not be used in C-API calls in
another interpreter.  That is relatively straight-forward
except in one case: indicating that the other interpreter
doesn't need the object to exist any more (similar to
PyBuffer_Release() but more general).  I consider the
following solutions to be the most viable.  Both make use
of recounts to protect cross-interpreter usage (with incref
before sharing).

1. manually switch interpreters (new private function)
  a. acquire the GIL
  b. if refcount > 1 then decref and release the GIL
  c. switch
  d. new thread (or re-use dedicated one)
  e. decref
  f. kill thread
  g. switch back
  h. release the GIL
2. change pending call mechanism (see Py_AddPendingCall) to
   per-interpreter instead of global (add "interp" arg to
   signature of new private C-API function)
  a. queue a function that decrefs the object
3. new cross-interpreter-specific private C-API function
  a. queue the object for decref (a la Py_AddPendingCall)
     in owning interpreter

I favor #2, since it is more generally applicable.  #3 would
probably be built on #2 anyway.  #1 is relatively inefficient.
With #2, Py_AddPendingCall() would become a simple wrapper
around the new private function.
msg317334 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2018-05-22 19:36
As a lesser (IMHO) alternative, we could also modify Py_DECREF
to respect a new "shared" marker of some sort (perhaps relative
to #33607), but that would probably still require one of the
refcount-based solutions (and add a branch to Py_DECREF).
msg317344 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-22 20:39
"That is relatively straight-forward except in one case: indicating that the other interpreter doesn't need the object to exist any more (similar to PyBuffer_Release() but more general)"

Why an interpreter would access an object from a different interpreter? Each interpreter should have its own object space, no?
msg317404 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-05-23 13:17
Adding a low level callback based mechanism to ask another interpreter to do work seems like a good way to go to me.

As an example of why that might be needed, consider the case of sharing a buffer exporting object with another subinterpreter: when the memoryview in the subinterpreter is destroyed, it needs to request that the buffer view be released in the source interpreter that actually owns the original object.
msg325481 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2018-09-16 11:54
I would suggest that sharing of objects between interpreters should be stamped out.  Could we have some #ifdef debug checking that would warn or assert so this doesn't happen?  I know currently we share a lot of objects.  However, in the long term, that does not seem like the correct design.  Instead, each interpreter should have its own objects and any passing or sharing should be tightly controlled.
msg325548 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2018-09-17 15:41
@Neil, we're definitely on the same page.  In fact, in a world where subinterpreters do not share a GIL, we can't ever use an object in one interpreter that was created in another (due to thread safety on refcounts).  The role of "tightly controlling" passing/sharing objects (for a very loose definition of "sharing") falls to the channels described in PEP 554. [1]

However, there are several circumstances where interpreters may collaborate that involves one holding a reference (but not using it) to an object owned by the other.  For instance, see PyBuffer_Release(). [2]  This issue is about addressing that situation safely.  It is definitely not about safely using objects from other interpreters.

[1] The low-level implementation, including channels, already exists in Modules/_xxsubinterpretersmodule.c.
[2] https://docs.python.org/3/c-api/buffer.html#c.PyBuffer_Release
History
Date User Action Args
2019-01-18 20:43:41eric.snowsetpull_requests: + pull_request11362
2019-01-18 20:43:26eric.snowsetpull_requests: + pull_request11361
2019-01-18 20:43:08eric.snowsetpull_requests: + pull_request11360
2018-11-10 18:26:45Johan Dahlinsetnosy: + Johan Dahlin
2018-09-17 15:41:58eric.snowsetassignee: eric.snow
messages: + msg325548
2018-09-16 11:54:57naschemesetnosy: + nascheme
messages: + msg325481
2018-09-15 18:25:04eric.snowsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request8758
2018-06-22 22:48:34eric.snowsetnosy: + emilyemorehouse
2018-05-23 13:17:57ncoghlansetmessages: + msg317404
2018-05-22 20:57:59vstinnersettitle: Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed. -> [subinterpreters] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.
2018-05-22 20:42:16pmppsetnosy: + pmpp
2018-05-22 20:39:54vstinnersetmessages: + msg317344
2018-05-22 19:36:39eric.snowsetmessages: + msg317334
versions: + Python 3.8, - Python 3.4
2018-05-22 19:34:41eric.snowcreate