This is a reincarnation of previous issues such as
- older https://bugs.python.org/issue21595 which partially (with ack on that) addressed the issue awhile back
- more recent https://bugs.python.org/issue38104 which was closed as "wont fix" since "the provided example finishes without any warning" on 3.8 (verified -- true for me with 3.8.3rc1); and with the explanation that "You spawn too many subprocesses that finish virtually at the same time. It leads to wakeup_fd overrun."
- additional similar reports could be found online, e.g. https://stackoverflow.com/a/52391791/1265472 .
In our project we are slowly introducing use of asyncio and have a mix of execution with asyncio and regular subprocess.Popen. We do run lots of short lived processes serially, and I think it should be Ok, i.e. it should not cause underlying libraries to spit out some output to ignore unless we indeed just using them incorrectly somehow.
If we recreate the SelectorEventLoop for every separate execution via asyncio -- no ignored exception messages are displayed. But if we start to reuse the same loop -- they eventually emerge. If I enable asyncio debug and log it along with our own debug messages, the strange thing that they come around the points where we run using regular subprocess.Popen, not asyncio. See https://github.com/datalad/datalad/pull/4527#issuecomment-629289819 for more information.
Unfortunately I do not have (yet) any short script to reproduce it, but I would appreciate possible hints on how to figure out what is actually causing them in our particular case. May be additional logging within asyncio could assist?
|
➜
