This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author kadler
Recipients kadler
Date 2020-09-30.17:26:22
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1601486783.26.0.906458411993.issue41894@roundup.psfhosted.org>
In-reply-to
Content
If a native module fails to load, the dynload code will call PyUnicode_FromString on the error message to give back to the user. This can cause a UnicodeDecodeError if the locale is not a UTF-8 locale and the error message contains non-ASCII code points.

While Linux systems almost always use a UTF-8 locale by default nowadays, AIX systems typically use non-UTF-8 locales by default. We encountered an issue where a customer did not have libbz2 installed, causing a load failure when bz2 tried to import _bz2 when running in an Italian locale:

$ LC_ALL=it_IT python3 -c 'import bz2'        
Traceback (most recent call last): 
 File "<string>", line 1, in <module> 
 File "/QOpenSys/pkgs/lib/python3.6/bz2.py", line 21, in <module> 
   from _bz2 import BZ2Compressor, BZ2Decompressor 
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 161: invalid continuation byte

After switching to a UTF-8 locale, the problem goes away:

$ LC_ALL=IT_IT python3 -c 'import bz2'   
Traceback (most recent call last): 
 File "<string>", line 1, in <module> 
 File "/QOpenSys/pkgs/lib/python3.6/bz2.py", line 21, in <module> 
   from _bz2 import BZ2Compressor, BZ2Decompressor 
ImportError:    0509-022 Impossibile caricare il modulo /QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so. 
       0509-150   Il modulo dipendente libbz2.so non è stato caricato. 
       0509-022 Impossibile caricare il modulo libbz2.so. 
       0509-026 Errore di sistema: Un file o una directory nel nome percorso non esiste. 
       0509-022 Impossibile caricare il modulo /QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so. 
       0509-150   Il modulo dipendente /QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so non è stato caricato.


While this conceivably affects any Unix-like platform, the only system I can recreate it on is AIX and IBM i PASE. As far as I can tell, on Linux you will always get something like "error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory". Even though there seems to be some translations in GLIBC, I have been unable to get them to be used on either Fedora or Ubuntu.
History
Date User Action Args
2020-09-30 17:26:23kadlersetrecipients: + kadler
2020-09-30 17:26:23kadlersetmessageid: <1601486783.26.0.906458411993.issue41894@roundup.psfhosted.org>
2020-09-30 17:26:23kadlerlinkissue41894 messages
2020-09-30 17:26:22kadlercreate