LANG=ka_GE.georgianps /usr/bin/python3
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: unknown encoding: GEORGIAN-PS
Aborted (core dumped)

but with python-2.7.5 no crash...
LANG=ka_GE.georgianps /usr/bin/python2
Python 2.7.5 (default, Oct  8 2013, 12:19:40) 
[GCC 4.8.1 20130603 (Red Hat 4.8.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

(fedora 19)

This bug was initially reported in LibreOffice:
https://bugs.freedesktop.org/show_bug.cgi?id=68850

I found three georgian encodings:

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/charmaps/GEORGIAN-PS;h=64615ff4344d74ea0c70cfd7a6c6c8019afb884e;hb=HEAD

https://sourceware.org/git/?p=glibc.git;a=blob;f=localedata/charmaps/GEORGIAN-ACADEMY;h=9dc1bc9e782e9fe6092a00daf1a75274fd6dd738;hb=HEAD

http://tools.ietf.org/html/draft-giasher-geostd8-00

The first one ("GEORGIAN-PS") is probably the most accurate because it is the one included in the GNU libc.

Could you please try to copy attached georgian_ps.py file into /usr/lib64/python3.3/encodings/ (or /usr/lib/python3.3/encodings/ for 32-bit Linux)?

Then try to print georgian letters using:

   print(bytes(range(0xc0, 0xe6)).decode("GEORGIAN-PS"))

Please give me also your locale encoding:

   import locale; print(locale.getpreferredencoding())

@Caolán: Do you know the GEORGIAN-ACADEMY encoding? It doesn't look to be used by any glibc locale.

On my Fedora 18, I have 3 georgian locales:

* ka_GE.georgianps: locale encoding GEORGIAN-PS
* ka_GE: locale encoding GEORGIAN-PS
* ka_GE.utf8: locale encoding UTF-8

You can workaround this issue by switching your locale from ka_GE.georgianps to ka_GE.utf8.

With recent versions of Python (e.g. 3.9) this no longer causes a crash. Python apparently falls back to UTF-8, at least on my system:

$ LANG=ka_GE.georgianps python3.9
Python 3.9.7 (default, Sep  9 2021, 23:20:13) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale; print(locale.getpreferredencoding())
UTF-8

I'm marking this as fixed. If someone still has issues with this encoding, please open a new issue with up-to-date information.

Python uses UTF-8 if the locale is not supported:

$ LANG=xxx python3.9 -c "import sys; print(sys.flags.utf8_mode)"
1

On Fedora 34, the locale is still supported, and Python 3.11 still fails:

vstinner@apu$ LANG=ka_GE.georgianps locale
LANG=ka_GE.georgianps
LC_CTYPE="ka_GE.georgianps"
LC_NUMERIC="ka_GE.georgianps"
LC_TIME="ka_GE.georgianps"
LC_COLLATE="ka_GE.georgianps"
LC_MONETARY="ka_GE.georgianps"
LC_MESSAGES="ka_GE.georgianps"
LC_PAPER="ka_GE.georgianps"
LC_NAME="ka_GE.georgianps"
LC_ADDRESS="ka_GE.georgianps"
LC_TELEPHONE="ka_GE.georgianps"
LC_MEASUREMENT="ka_GE.georgianps"
LC_IDENTIFICATION="ka_GE.georgianps"
LC_ALL=

vstinner@apu$ LANG=ka_GE.georgianps python3.11 -c "import sys; print(sys.flags.utf8_mode)"
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  stdlib dir = '/home/vstinner/python/main/Lib'
  sys._base_executable = '/home/vstinner/python/main/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/home/vstinner/python/main/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/home/vstinner/python/main/Lib',
    '/home/vstinner/python/main/build/lib.linux-x86_64-3.11-pydebug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: GEORGIAN-PS

Current thread 0x00007ff89b81d2c0 (most recent call first):
  <no Python frame>

Possible solutions (they can be combined):

1. Add support for the GEORGIAN-PS charset and all other encodings used in libc (issue22679). The problem is that it is difficult to get the official information about these encodings.

2. Falls back to utf-8 or ascii+surrogateescape in case of unsupported locale encoding. But typos can slip unnoticed.

On 19.10.2021 10:44, Serhiy Storchaka wrote:
> 
> Possible solutions (they can be combined):
> 
> 1. Add support for the GEORGIAN-PS charset and all other encodings used in libc (issue22679). The problem is that it is difficult to get the official information about these encodings.

As with all encodings we add: there has to be a real need to support
them natively in Python (as opposed to installing codecs via PyPI)
and we need a definite source for the encoding, e.g. a standards
document from an official body.

IMO, we should not really add more encodings to the stdlib, but instead
point people to e.g. the iconv package:

https://pypi.org/project/python-iconv/

Perhaps we ought to make it easier for such packages to provide
additional codecs even during the startup phase, e.g. via a special
env var which points Python to a list of codec packages to load
prior to initializing the I/O encoding... not sure whether this is
possible, though.

> 2. Falls back to utf-8 or ascii+surrogateescape in case of unsupported locale encoding. But typos can slip unnoticed.

I think this would be a more general solution to such cases, provided
the startup logic issues a visible warning about the fallback.