Here is (attached) a minimal patch to the core trunk CPython to allow extension modules to take over control of acquiring and releasing the GIL, as proposed here:

http://mail.python.org/pipermail/python-dev/2011-August/113248.html

With this patch, it is possible to write an independent extension module that offers the basic STM "with atomic" blocks:

https://bitbucket.org/arigo/cpython-withatomic/raw/default/stm

As Guido hints in the mail above, more experimentation is needed before we know if the idea can really fly, notably if there are annoying issues with existing locks causing random deadlocks.

NB. I know that my stmmodule.c contains a gcc-ism: it uses a __thread global variable.  I plan to fix this in future versions :-)

Rather than exposing the function pointers directly to the linker, I'd be happier with a function based API, with the pointer storage then being made static inside ceval.c.

/* Each function returns the old func, or NULL on failure */
_PyEval_GIL_func _PyEval_replace_take_GIL(_PyEval_GIL_func take_gil); _PyEval_GIL_func _PyEval_replace_drop_GIL(_PyEval_GIL_func drop_gil); 


The redirection code (sans error checking) would then look like:

old_take_gil = _PyEval_replace_take_GIL(stm_take_gil);
old_drop_gil = _PyEval_replace_drop_GIL(stm_drop_gil);

Currently they'd just replace the statics and would never fail, but it provides looser coupling regardless.

I suppose I'm fine either way, but do you have a reason for not exposing the variables to the linker?  Some Windows-ism were such exposed variables are slower to access than static ones, maybe?  The point is that they are kind of "internal use only" anyway, so designing nice APIs around them looks overkill to me.

Nothing specific, just a reflexive C++ induced dislike for linker-accessible globals in general.

However, while I slightly prefer the function driven API, I wouldn't actively oppose direct linker access if someone else wanted to check it in :)

Ok, I followed Nick's suggestion, and I finally found out how to write the code in order to avoid all (or most?) deadlocks without any change in the rest of CPython.  It requires a way to be sure that some callback function is invoked _at the next cross-bytecode point_, and not (or not much) later, which I also include here.  Here is the updated patch.

I'm worried that by allowing users to pass function pointers here, we are allowing all kinds of uses that we will have to more or less support later. What exactly "taking the GIL" means and when exactly it is done in the execution process is an implementation detail.

Armin, is there a way to implement what you want to do with a more constraining API?

Antoine: we could take two lines from the current implementation of these hook from stm/transactionmodule.c, and move them to the interpreter core.  CPython would end up with containing the core logic for transactions.  A possible API would look like that:

/* stops all threads different from 'tstate' at their next
   bytecode, at which point the given callback is invoked */
_PyEval_SetThreadStopper(PyThreadState *tstate, void (*callback)(void));

Such a function can also be tested in CPython, e.g. with ctypes.  Still maybe a different name would be better, like _PyEval_InternalThreadStopper().  Or, if that's still a callback too much:

/* stops all threads different from 'tstate' at their next bytecode,
   at which point they wait for the 'lock' to be available */
_PyEval_SetThreadStopper(PyThreadState *tstate, PyThread_lock_type lock);

I quite like the last idea. Something like:

_PyEval_SuspendOtherThreads(PyThreadState *tstate, PyThread_lock_type lock);

  All threads other than tstate will be prevented from executing further interpreter bytecodes until "lock" is released.

Offering this API might pose a problem for various "superinstruction" concepts in the future, though.

That's why I don't really know which concept is the best: the core of transactionmodule.c is just about one page in length, so there are only so many ways to split this code between CPython and the module...

Attached the latest suggestion.  I am also fine if this function is not called _PyEval_SuspendOtherThreads() but _PyEval_InternalUseOnlyYouShouldReallyNotUseThis()...

Semi-seriously, _PyEval_ForArminRigoOnly_
/* This function does whatever Armin Rigo wants it to do.
He may change it at any time. Do not use it. */

This would let you experiment with the boundary as you will (with review, of course) and relieve us of any obligation to anyone else who uses it.

CPython got the Don Beaudry metaclass hook sometime before 1.5. (I have no idea of its internal name or implementation.) It was also used, at his own risk, by Jim Fulton. Much later, in 2.2, it because the basis for new-style classes, which are now simply Python 3 classes.
http://www.python.org/doc/essays/metaclasses/

:-)

Closing the request for this patch.  It is unsatisfactory that it only offers the basic user-level STM feature of transaction, but not, say, abort_and_retry() or any other feature generally found in real-world STM implementations.  This goes against the idea, which was (in part) to let python-dev people play with it to know which features seem to make sense in Python.