contextvars provide APIs to manage, store, and access context-local state.

Unfortunately, if Python is used as a frontend for a native libray (eg accessed via ctypes), and in case that the state of interest is managed in the native library, contextvar API is insufficient.

To support native libraries, instead of simply exposing the current state via `contextvar.get()`, contextvar API could allow specification of callbacks to update the state in the native library.

> Unfortunately, if Python is used as a frontend for a native libray (eg accessed via ctypes), and in case that the state of interest is managed in the native library, contextvar API is insufficient.

Please elaborate with a clearer example.

Consider a `lib = ctypes.CDLL(lib_path, ctypes.RTLD_LOCAL)` which provides APIs `lib.compute`, `lib.set_state` and a Python interface wrapping both in `def compute`, `def set_state`. Let's assume `lib.set_state` sets a thread-local state in the library implementation and further  that the behaviour of `lib.compute` depends on the last call to `lib.set_state`. 

Users of the Python interface may call `set_state` and `compute` concurrently in coroutines. Thereby one coroutine may change the state in the library, prior to another coroutine calling `compute`. This leads to a wrong result.

If `set_state` and `compute` were pure python functions, `contextvars` could be used to implement set_state and provide context-local state.

Is there any existing API that can be used to call `lib.set_state` on context changes? If so, please let me know. If not, would it make sense to extend `contexvars` to allow users to configure that `lib.set_state` is called on context change?

> Is there any existing API that can be used to call `lib.set_state` on context changes?

No, but there's C API that you can use to get/set contextvars. If a C library is hard coded to use threadlocals I'm afraid there's nothing we can do about it except fixing their C code to make state storage pluggable.

What library do you have in mind?

> If not, would it make sense to extend `contexvars` to allow users to configure that `lib.set_state` is called on context change?

Theoretically yes, for debug purposes at least.  But I still fail to see how you could use that API even if it existed.

> No, but there's C API that you can use to get/set contextvars. If a C library
> is hard coded to use threadlocals I'm afraid there's nothing we can do about
> it except fixing their C code to make state storage pluggable.

I agree the C library could check the Python state. On the other hand, the Python frontend may be just one of multiple supported language frontends of the library. So it seems preferable if Python could tell the library when to change state.

> What library do you have in mind?

MXNet has such a thread-local C API to enable / disable recording for autograd. https://github.com/apache/incubator-mxnet/blob/9b38df0622fec677e6b4891dfc4c10295359996d/include/mxnet/c_api.h#L1260-L1266

> Theoretically yes, for debug purposes at least. But I still fail to see how
> you could use that API even if it existed.

Consider we are in "context 1". When we change the state in the C library by calling `old_state = lib.set_state(new_state)`, we get to know the prior state. When switching to "context 2", `lib.set_state(old_state)` should be called. When switching back to "context 1", `lib.set_state(new_state)` should be called again. It's analogous to contextvars.

Would there be too much overhead if allowing specification of a python function that contextvars calls on context changes?

> Would there be too much overhead if allowing specification of a python function that contextvars calls on context changes?

Potentially yes, especially if we allow more than one context change callback.  Allowing just one makes the API inflexible (what if you want to use two libraries from PyPI that both want to use the callback).  Allowing multiple context change callbacks leads to complicated API.

For extra context: context switches occur on every callback invocation in asyncio and there can be thousands of them per seconds (or even more). Adding any extra code to context switching code will noticeably degrade the performance.

In general, I'd suggest patching the C library to make state management customizable (like CPython allows you to customize which memory allocator to use)