Thanks, Ned, for finding my mistake which you generously called a typo.
I have fixed the inequality in the Tk fossil repository.  Incidentally, there is now a core-8-6-12-rc branch of Tk, also containing the fix.  So it should not be too long before 8.6.12 is released.

I will look at the code and check whether I see the UFO that you report.  As you know, all that Tk does is to open an NSOpenPanel.  Once that is open it is completely handled by the OS.  Tk sees no events from the modal interaction and does not have any way of controlling any aspect of the dialog.  So the only hope would be that it is possible to set some configuration option before opening the NSOpenPanel which would somehow cause the OS to assign focus to the correct widget.  I will try to investigate what options are available.