I guess you are mainly complaining about when __exit__() implementation is built into the interpreter. E.g., given demo.py:

with open("/dev/full", "wb") as file:
    file.write(b"data")
    print("All is well")
print("Not executed")

the output looks like

All is well
Traceback (most recent call last):
  File "demo.py", line 3, in <module>
    print("All is well")
OSError: [Errno 28] No space left on device

When __exit__() is implemented natively in Python, you get more of a hint:

Traceback (most recent call last):
  File "demo.py", line 4, in <module>
    print("All is well")
  File "/home/proj/python/cpython/Lib/_pyio.py", line 457, in __exit__
    self.close()
  File "/home/proj/python/cpython/Lib/_pyio.py", line 773, in close
    self.flush()
  File "/home/proj/python/cpython/Lib/_pyio.py", line 1210, in flush
    self._flush_unlocked()
  File "/home/proj/python/cpython/Lib/_pyio.py", line 1217, in _flush_unlocked
    n = self.raw.write(self._write_buf)
  File "/home/proj/python/cpython/Lib/_pyio.py", line 1614, in write
    return os.write(self._fd, b)
OSError: [Errno 28] No space left on device

Maybe another option would be to insert a virtual frame in the built-in version of the traceback:

Traceback (most recent call last):
  File "demo.py", line 3, in <module>
    print("All is well")
  File "<built in>", line ???, in <context manager exit>
OSError: [Errno 28] No space left on device