classification
Title: [PATCH] stm.atomic
Type: enhancement Stage:
Components: Interpreter Core Versions: Python 3.3
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: arigo, asvetlov, barry, daniel.urban, georg.brandl, jcea, nadeem.vawda, ncoghlan, pitrou, terry.reedy
Priority: normal Keywords: patch

Created on 2011-08-29 09:25 by arigo, last changed 2011-12-01 17:04 by arigo. This issue is now closed.

Files
File name Uploaded Description Edit
stm.diff arigo, 2011-08-29 09:25 Patch to the head of CPython 'default' review
stm2.diff arigo, 2011-08-30 14:03 review
stm3.diff arigo, 2011-09-02 06:20 review
Messages (13)
msg143132 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-08-29 09:25
Here is (attached) a minimal patch to the core trunk CPython to allow extension modules to take over control of acquiring and releasing the GIL, as proposed here:

http://mail.python.org/pipermail/python-dev/2011-August/113248.html

With this patch, it is possible to write an independent extension module that offers the basic STM "with atomic" blocks:

https://bitbucket.org/arigo/cpython-withatomic/raw/default/stm

As Guido hints in the mail above, more experimentation is needed before we know if the idea can really fly, notably if there are annoying issues with existing locks causing random deadlocks.
msg143134 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-08-29 09:46
NB. I know that my stmmodule.c contains a gcc-ism: it uses a __thread global variable.  I plan to fix this in future versions :-)
msg143136 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-08-29 11:43
Rather than exposing the function pointers directly to the linker, I'd be happier with a function based API, with the pointer storage then being made static inside ceval.c.

/* Each function returns the old func, or NULL on failure */
_PyEval_GIL_func _PyEval_replace_take_GIL(_PyEval_GIL_func take_gil); _PyEval_GIL_func _PyEval_replace_drop_GIL(_PyEval_GIL_func drop_gil); 


The redirection code (sans error checking) would then look like:

old_take_gil = _PyEval_replace_take_GIL(stm_take_gil);
old_drop_gil = _PyEval_replace_drop_GIL(stm_drop_gil);

Currently they'd just replace the statics and would never fail, but it provides looser coupling regardless.
msg143137 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-08-29 11:49
I suppose I'm fine either way, but do you have a reason for not exposing the variables to the linker?  Some Windows-ism were such exposed variables are slower to access than static ones, maybe?  The point is that they are kind of "internal use only" anyway, so designing nice APIs around them looks overkill to me.
msg143208 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-08-30 11:11
Nothing specific, just a reflexive C++ induced dislike for linker-accessible globals in general.

However, while I slightly prefer the function driven API, I wouldn't actively oppose direct linker access if someone else wanted to check it in :)
msg143211 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-08-30 14:03
Ok, I followed Nick's suggestion, and I finally found out how to write the code in order to avoid all (or most?) deadlocks without any change in the rest of CPython.  It requires a way to be sure that some callback function is invoked _at the next cross-bytecode point_, and not (or not much) later, which I also include here.  Here is the updated patch.
msg143341 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-09-01 18:27
I'm worried that by allowing users to pass function pointers here, we are allowing all kinds of uses that we will have to more or less support later. What exactly "taking the GIL" means and when exactly it is done in the execution process is an implementation detail.

Armin, is there a way to implement what you want to do with a more constraining API?
msg143379 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-09-02 05:33
Antoine: we could take two lines from the current implementation of these hook from stm/transactionmodule.c, and move them to the interpreter core.  CPython would end up with containing the core logic for transactions.  A possible API would look like that:

/* stops all threads different from 'tstate' at their next
   bytecode, at which point the given callback is invoked */
_PyEval_SetThreadStopper(PyThreadState *tstate, void (*callback)(void));

Such a function can also be tested in CPython, e.g. with ctypes.  Still maybe a different name would be better, like _PyEval_InternalThreadStopper().  Or, if that's still a callback too much:

/* stops all threads different from 'tstate' at their next bytecode,
   at which point they wait for the 'lock' to be available */
_PyEval_SetThreadStopper(PyThreadState *tstate, PyThread_lock_type lock);
msg143382 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2011-09-02 06:02
I quite like the last idea. Something like:

_PyEval_SuspendOtherThreads(PyThreadState *tstate, PyThread_lock_type lock);

  All threads other than tstate will be prevented from executing further interpreter bytecodes until "lock" is released.

Offering this API might pose a problem for various "superinstruction" concepts in the future, though.
msg143383 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-09-02 06:20
That's why I don't really know which concept is the best: the core of transactionmodule.c is just about one page in length, so there are only so many ways to split this code between CPython and the module...

Attached the latest suggestion.  I am also fine if this function is not called _PyEval_SuspendOtherThreads() but _PyEval_InternalUseOnlyYouShouldReallyNotUseThis()...
msg143439 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-09-02 21:41
Semi-seriously, _PyEval_ForArminRigoOnly_
/* This function does whatever Armin Rigo wants it to do.
He may change it at any time. Do not use it. */

This would let you experiment with the boundary as you will (with review, of course) and relieve us of any obligation to anyone else who uses it.

CPython got the Don Beaudry metaclass hook sometime before 1.5. (I have no idea of its internal name or implementation.) It was also used, at his own risk, by Jim Fulton. Much later, in 2.2, it because the basis for new-style classes, which are now simply Python 3 classes.
http://www.python.org/doc/essays/metaclasses/
msg143453 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-09-03 07:05
:-)
msg148728 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-12-01 17:04
Closing the request for this patch.  It is unsatisfactory that it only offers the basic user-level STM feature of transaction, but not, say, abort_and_retry() or any other feature generally found in real-world STM implementations.  This goes against the idea, which was (in part) to let python-dev people play with it to know which features seem to make sense in Python.
History
Date User Action Args
2011-12-01 17:04:38arigosetstatus: open -> closed
resolution: rejected
messages: + msg148728
2011-09-06 06:37:27georg.brandlsetnosy: + georg.brandl
2011-09-06 04:33:30jceasetnosy: + jcea
2011-09-03 07:05:03arigosetmessages: + msg143453
2011-09-02 21:41:05terry.reedysetnosy: + terry.reedy
messages: + msg143439
2011-09-02 06:20:11arigosetfiles: + stm3.diff

messages: + msg143383
2011-09-02 06:02:17ncoghlansetmessages: + msg143382
2011-09-02 05:33:01arigosetmessages: + msg143379
2011-09-01 18:27:08pitrousetnosy: + pitrou
messages: + msg143341
2011-08-31 10:18:09asvetlovsetnosy: + asvetlov
2011-08-30 14:03:35arigosetfiles: + stm2.diff

messages: + msg143211
2011-08-30 11:11:05ncoghlansetmessages: + msg143208
2011-08-30 09:55:37daniel.urbansetnosy: + daniel.urban
2011-08-29 12:46:11barrysetnosy: + barry
2011-08-29 11:49:53arigosetmessages: + msg143137
2011-08-29 11:43:28ncoghlansetnosy: + ncoghlan
messages: + msg143136
2011-08-29 09:46:40arigosetmessages: + msg143134
2011-08-29 09:32:19nadeem.vawdasetnosy: + nadeem.vawda
2011-08-29 09:25:40arigocreate