You'd have to do that anyway if we implemented a delete=False constructor argument, since you want it deleted if there are any errors, and that's not what a delete=False API would do.  

If it were me, I'd write it (untested)

  @contextlib.contextmanager
  def open_for_atomic_replace(fn):
    try:
        fd, name = tempfile.mkstemp()
        with io.open(fd) as fff:
            yield fff
        fff.flush()
        os.fdatasync(fff)
        os.rename(name, fn)
    except BaseException:
        os.unlink(name)
        raise

which would make your code simpler than it is now.

Naming it 'open_for_atomic_replace' reminded me of issue 8604, which is what you really want, not delete=False.