When asyncio event loop is created in a non-main thread, it needs to initialize a so called ChildWatcher for it (a helper object to intercept subprocesses exits).  Doing that requires to run code in the main thread.

I propose to add a 'sys.addpendingcall' function that will simply expose the already existing Py_AddPendingCall to the pure Python land.

Note that I'm working on making pending calls per-interpreter (see issue #33608 and https://github.com/python/cpython/pull/12360 (since reverted)).

As to exposing Py_AddPendingCall() as sys.addpendingcall, that might be opening a can of worms.  Injecting code into the eval loop at some arbitrary ("soon") future time requires care and the code isn't well exercised historically (much like subinterpreters).  By making it easier to use the pending calls API (e.g. from Python code) we may be introducing an attractive nuisance.  It also adds burden on other Python implementations.

My point is, let's think this through before adding sys.addpendingcall(). :)  Is there another way this could be done that doesn't open a can of worms?

Also, at the very least it should probably be a "private" function (i.e sys._addpendingcall).

> As to exposing Py_AddPendingCall() as sys.addpendingcall, that might be opening a can of worms.  Injecting code into the eval loop at some arbitrary ("soon") future time requires care and the code isn't well exercised historically (much like subinterpreters).

Well, it's absolutely the same mechanism that signal handlers use.  It looks like this can of works has been open for a pretty long time now :)

The whole point of adding this API is to make it possible for asyncio to work with the "signals" module not from the main thread.  I expect 99.9% of other use cases to be either about signals or other super low-level stuff that needs the main thread (off the top of my head I can't name any :))

> It also adds burden on other Python implementations.

Python signal processing *requires* users callbacks to be executed in the main thread (whereas Unix can deliver the signal to any thread, potentially). Therefore I think that any alternative Python implementation should have a mechanism to schedule code execution in the main thread.

> My point is, let's think this through before adding sys.addpendingcall(). :)  Is there another way this could be done that doesn't open a can of worms?

Maybe. One of such ways is to enable signal.signal calls from non-main threads.  But I'm not sure this is possible for all platforms we support. Maybe Victor or Andrew can shed more light on this.

> Also, at the very least it should probably be a "private" function (i.e sys._addpendingcall).

Maybe.  The problem is that asyncio must work the same on PyPy, which means that the sys API it uses should be available on PyPy too.  In which case, they will kind of have to add it, which means that there's no point in making it semi-private.  OTOH, I wouldn't mind sys._addpendingcall(), as long as PyPy implements it.

> Well, it's absolutely the same mechanism that signal handlers use.

Antoine changed signals a while back so they no longer use the pending calls machinery.

If this is about signals, it isn't enough just to run on the main thread.  It also has to be the main interpreter.

I dislike pending calls. It is a weird beast which complicates ceval.c. I
would prefer to remove it. It is no longer needed to handle signals!

At least, it sounds like a bad idea to me to expose it in Python in a
public API.

If you really need it, hide it better ;-)

Eric, regarding signals no longer using pending calls -- interesting, I'll take a look. Thanks for pointing this out.

Sent from my iPhone

There is not clear rationale to justify the addition of the function, so I reject the feature. If you are still interesting by the function, please elaborate the rationale.

> There is not clear rationale to justify the addition of the function

Yeah, with the new threaded watcher being the default we don't need this anymore.


> so I reject the feature

NP, here, but, hm, can you unilaterally reject features now? :)

> NP, here, but, hm, can you unilaterally reject features now? :)

Eric Snow and me are basically against the idea of exposing the "pending calls" mechanism in Python, since it's fragile and dangerous. Andrew Sveltov wrote "@vstinner had objections, perhaps this PR should be rejected." So I don't think that it's an unilaterally decision :)

There are multiple unanswered questions which have been asked here. For example, the ratione to add this feature seems to be based on a wrong assumption (signal handling using the pending call mechanism).

I'm not strongly against the feature. I first proposed to expose it, but make it private. Almost one year later, the PR was not updated. So I just closed the PR and the issue.

The problem is not only asynico, it's how this feature can be abused in general, and users who may have wrong expectations on how it works.


> Yeah, with the new threaded watcher being the default we don't need this anymore.

Handling subprocesses in asyncio also seem to be a can of worms. bpo-38323 is a good example :-(

> I'm not strongly against the feature. I first proposed to expose it, but make it private. Almost one year later, the PR was not updated. So I just closed the PR and the issue.

All clear, Victor. Let's keep this closed. The reason I didn't reply is because we are no longer need it and this is a very low-level piece of functionality that I myself feel uneasy about. It can have potential problems to implement in PyPy/other interpreters in a compatible with CPython fashion.

I'll reopen if there's a need for this in asyncio. But even then, we can make this a "private" and undocumented function in the sys module (of course that would mean that we only use it to issue warnings / add extra debug functionality).