Author vstinner
Recipients Sworddragon, a.badger, ezio.melotti, loewis, ncoghlan, r.david.murray, vstinner
Date 2013-12-13.17:08:27
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1386954507.8.0.569298594543.issue19977@psf.upfronthosting.co.za>
In-reply-to
Content
os.fsencode(text) always fail if text cannot be encoded to sys.getfilesystemencoding(). surrogateescape doesn't help here.

Your example is "artificial", you should not get 'ä'. All OS data is decoded from the filesystem encoding using the surrogateescape error handler (except on Windows, where strict is used, but it's a different story, Python uses Unicode functions when available so don't worry). So all these data can always be encoded back to bytes using os.fsencode().

More generally, os.fsencode(os.fsdecode(read_data)) == read_data is always true on Unix, with any filesystem (locale) encoding.

You may get Unicode data from other sources like files or a GUI, but I don't see what can be done here.
History
Date User Action Args
2013-12-13 17:08:27vstinnersetrecipients: + vstinner, loewis, ncoghlan, ezio.melotti, a.badger, r.david.murray, Sworddragon
2013-12-13 17:08:27vstinnersetmessageid: <1386954507.8.0.569298594543.issue19977@psf.upfronthosting.co.za>
2013-12-13 17:08:27vstinnerlinkissue19977 messages
2013-12-13 17:08:27vstinnercreate