classification
Title: Adding missing ISO 8859 codecs, especially Thai
Type: enhancement Stage:
Components: None Versions:
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: lemburg Nosy List: lemburg, loewis, peter_jacobi
Priority: normal Keywords:

Created on 2004-08-02 09:48 by peter_jacobi, last changed 2007-08-23 19:31 by lemburg. This issue is now closed.

Files
File name Uploaded Description Edit
8859_11_and_16.zip peter_jacobi, 2004-08-03 22:58 Mappings generated using gencodec.py
Messages (13)
msg54203 - (view) Author: Peter Jacobi (peter_jacobi) Date: 2004-08-02 09:48
As the missing ISO 8859 codecs, (11:Thai, 16:Romanian) 
can be automatically generated from the Unicode 
mapping files (via gencodec.py), I'd like to ask for 
inclusion in the next version.
msg54204 - (view) Author: Peter Jacobi (peter_jacobi) Date: 2004-08-02 10:16
Logged In: YES 
user_id=845149

In a thread on news://comp.lang.python I was asked by 
Martin v. Löwis to provide evidence on the correctness of the 
ISO 8859-11 Unicode mapping file, as found on 
ftp://ftp.unicode.org/Public/MAPPINGS/ISO8859/8859-11.TXT 
(due to the disclaimer boilerplate in these files).

So far I can provide these three points:
a) ISO 8859-n vs ISO-8859-n
If the information at http://en.wikipedia.org/wiki/ISO_8859-
1#ISO_8859-1_vs_ISO-8859-1 is correct, Python 8859-n 
codecs do implement the ISO standard charsets ISO 8859-n 
in the specialized IANA forms ISO-8859-n (and in agreement 
with the Unicode mapping files). So any difficult C0/C1 
wording in the original ISO standard can be disregarded.

b) libiconv ISO 8859-11
The implementation by Bruno Haible in libiconv does agree 
with the Unicode mapping file:
http://cvs.sourceforge.net/viewcvs.py/libiconv/libiconv/lib/

c) IBM ICU4C
The implementation in ICU4C does agree with the Unicode 
mapping file:
http://oss.software.ibm.com/cvs/icu/charset/data/ucm/

msg54205 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-08-02 10:30
Logged In: YES 
user_id=21627

Marc-Andre, should we add these?
msg54206 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2004-08-02 11:14
Logged In: YES 
user_id=38388

Martin, I think it's a good idea to add the codecs for
completeness. 

We should probably also review the mapping files posted on
the unicode.org site every now and then and update the
codecs in Python accordingly. Sticking to the Unicode
Consortium's view of things is a good way to assure
compatibility with other applications, IMO.
msg54207 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2004-08-03 14:34
Logged In: YES 
user_id=38388

Peter, could you attach the generated codecs to this report ?

Thanks.
msg54208 - (view) Author: Peter Jacobi (peter_jacobi) Date: 2004-08-03 22:58
Logged In: YES 
user_id=845149

Attached are the output if gencodec.py for ISO-8859-11, 
ISO-8859-16 and for reference also the original mapping files.

Peter
msg54209 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2004-08-05 11:15
Logged In: YES 
user_id=38388

Thank you. 

Please also provide suitable aliases (I couldn't find any on
the IANA site), then I'll add them to Python 2.4.
msg54210 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-08-05 11:41
Logged In: YES 
user_id=21627

The unfortunate problem is that ISO-8859-11 is not a
IANA-registered character set. For ISO-8859-16,

http://www.iana.org/assignments/character-sets

lists:

Name: ISO-8859-16
MIBenum: 112
Source: ISO
Alias: iso-ir-226
Alias: ISO_8859-16:2001
Alias: ISO_8859-16
Alias: latin10
Alias: l10 

I believe ISO-8859-11 does not have any aliases. Some people
may claim TIS-620 is an alias, but it is not (as it does not
contain \xa0).
msg54211 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2004-08-05 12:15
Logged In: YES 
user_id=38388

I found these references for iso-8859-11:

iso_8859-11:1992 (try searching for this in goole :-)
http://mnogosearch.kn.vutbr.cz/Download/snapshot/mnogosearch32/src/uconv-alias.c

windows-874
http://www.memecode.com/site/ver.php?id=94

thai
windows-874
tis-620
iso-8859-11:2001
http://de.wikipedia.org/wiki/ISO_8859-11

The lsat URL suggests that iso-8859-11 is the same as tis-620,
but only the "basis" for windows-874. It also quotes the
year 2001 as the last revision of the mapping which
corresponds to the header of the Unicode mapping file.

I think it's safe to add the alias for tis-620 even though
the iso mapping has one more character. According to Google
that encoding name is much more popular than the iso one.
msg54212 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2004-08-05 12:33
Logged In: YES 
user_id=38388

Nevermind. I'll also add a proper tis_620.py codec.
msg54213 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2004-08-05 12:44
Logged In: YES 
user_id=38388

Checking in Misc/NEWS;
/cvsroot/python/python/dist/src/Misc/NEWS,v  <--  NEWS
new revision: 1.1073; previous revision: 1.1072
done
Checking in Lib/encodings/aliases.py;
/cvsroot/python/python/dist/src/Lib/encodings/aliases.py,v 
<--  aliases.py
new revision: 1.27; previous revision: 1.26
done
RCS file:
/cvsroot/python/python/dist/src/Lib/encodings/iso8859_11.py,v
done
Checking in Lib/encodings/iso8859_11.py;
/cvsroot/python/python/dist/src/Lib/encodings/iso8859_11.py,v
 <--  iso8859_11.py
initial revision: 1.1
done
RCS file:
/cvsroot/python/python/dist/src/Lib/encodings/iso8859_16.py,v
done
Checking in Lib/encodings/iso8859_16.py;
/cvsroot/python/python/dist/src/Lib/encodings/iso8859_16.py,v
 <--  iso8859_16.py
initial revision: 1.1
done
RCS file:
/cvsroot/python/python/dist/src/Lib/encodings/tis_620.py,v
done
Checking in Lib/encodings/tis_620.py;
/cvsroot/python/python/dist/src/Lib/encodings/tis_620.py,v 
<--  tis_620.py
initial revision: 1.1
done
msg54214 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2004-08-05 13:02
Logged In: YES 
user_id=21627

Code page 874 differs from the 8859 one in the definition of
\x80..\x9f. 

http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/PC/CP874.TXT

says

0x80	0x20AC	#EURO SIGN
0x85	0x2026	#HORIZONTAL ELLIPSIS
0x91	0x2018	#LEFT SINGLE QUOTATION MARK
0x92	0x2019	#RIGHT SINGLE QUOTATION MARK
0x93	0x201C	#LEFT DOUBLE QUOTATION MARK
0x94	0x201D	#RIGHT DOUBLE QUOTATION MARK
0x95	0x2022	#BULLET
0x96	0x2013	#EN DASH
0x97	0x2014	#EM DASH

I assume the Thai version of Windows is likely to generate
"windows-874". Debian offers the th_TH locale, with TIS-620,
and a th_TH.UTF-8 locale (i.e. no ISO-8859-1 one).

If ISO 8859-11 is understood as published by ISO (i.e. no
control characters at all), then CP 874 is a strict
extension (adding C0, plus the characters above).

Google gives these frequencies:
tis-620 16,200
windows-874 7,290
iso-8859-11  5,880
msg55192 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2007-08-23 19:31
Not sure why this is still open. The patches were checked in a long time
ago.
History
Date User Action Args
2007-08-23 19:31:03lemburgsetstatus: open -> closed
messages: + msg55192
2004-08-02 09:48:38peter_jacobicreate