_filename_to_unicode returns unicode names as is.  In 3.x, are filenames ever not unicode?  They come from either a file dialog (via tk, hence unicode), or sys.argv. I do not know about the latter, which is possibly OS-dependent.

This function is used in three places in idlelib, all within EditorWindow:
1.    def update_recent_files_list(self, new_file=None): ...
                ufile_name = self._filename_to_unicode(file_name)
2.    def short_title(self):  # reduce filename to basename
        return self._filename_to_unicode(filename)
3.    def long_title(self):
        return self._filename_to_unicode(self.io.filename or "")

The _f2u output is not saved to disk or used to open files; it is display only.  So replacing astral chars with either \Unnnnn escapes or the BMP box char should be fine.

1. The callback associated with each ufile_name encloses the original file_name, which is used to open the file.  The original filename is also saved back to disk before the _f2u call.

2&3. The titles are display only.  WindowList.py displays the long name for editor windows, but the callback is a wakeup function tied to the Window itself.