> I am not fully sure why -O is essentially required for running .pyo
> files: why not have the Python interpreter handle everything
> automatically based on the extension?

In part because it would take work to make it happen and apparently no one feels strongly enough about adding that functionality, or convinced enough that it's appropriate, to do the work.  If you're interested a good first step would be to write a PEP 302 metapath hook (finder that gets inserted to sys.metapath) that makes import of .pyo files work even when -O is not used.  Maybe you'd also want to have .pyc files work even when -O *is* used.

Keep in mind that there may be other reasons why such functionality would not go into the interpreter.  However, the beauty of import hooks is that it wouldn't matter once you have yours in hand.