When stress testing my code in a process-limited environment I found that despite throwing an exception, it appears the process still executes. Attempting to catch/retry results in a duplicate.
Attached is a script that I was able to repro the problem on Linux. I cannot get it to behave similarly on MacOS.
The exception looks like:
/bin/sh: fork: retry: Resource temporarily unavailable
Traceback (most recent call last):
File "/home/colsen/async/./example", line 16, in run
proc = await asyncio.create_subprocess_exec('/bin/sh','-c', f'/bin/echo {_id} > {_uuid}.out')
File "/home/utils/Python/3.9/3.9.7-20211101/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File "/home/utils/Python/3.9/3.9.7-20211101/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec
transport = await self._make_subprocess_transport(
File "/home/utils/Python/3.9/3.9.7-20211101/lib/python3.9/asyncio/unix_events.py", line 202, in _make_subprocess_transport
watcher.add_child_handler(transp.get_pid(),
File "/home/utils/Python/3.9/3.9.7-20211101/lib/python3.9/asyncio/unix_events.py", line 1381, in add_child_handler
thread.start()
File "/home/utils/Python/3.9/3.9.7-20211101/lib/python3.9/threading.py", line 892, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
So, this script ended up producing 21 output files (duplicate on iteration 18).
I need a way to catch these errors, pause, and retry when they are recoverable.
|