So, you're complaining about something which works, kind of:

$ touch héhé
$ LANG=C python3 -c "import os; print(os.listdir())"
['h\udcc3\udca9h\udcc3\udca9']

> This makes robustly working with non-ascii filenames on different
> platforms needlessly annoying, given no modern nix should have problems
> just using UTF-8 in these cases.

So why don't these supposedly "modern" systems at least set the appropriate environment variables for Python to infer the proper character encoding?
(since these "modern" systems don't have a well-defined encoding...)

Answer: because they are not modern at all, they are antiquated, inadapted and obsolete pieces of software designed and written by clueless Anglo-American people. Please report bugs against these systems. The culprit is not Python, it's the Unix crap and the utterly clueless attitude of its maintainers ("filesystems are just bytes", yeah, whatever...).