Please add encoding for the iso6937 charset. Many settopboxes (DVB-T/S) and relevant devices uses it for displaying EPG, videotext, etc.

I've wrote (please look at the attached file) the encoding/decoding conversion codec some years ago.

New encoding can be added only in new Python release (3.6).

I've rewrote the iso6937 codec into Python 3. 

Could someone check it please?

Hi John, thanks for your contribution,

Looks like your implementation is missing some codepoints, like "\t":

    >>> print("\t".encode(encoding='iso6937'))                                                                                     
    [...]
    UnicodeError: encoding with 'iso6937' codec failed (UnicodeError: Unacceptable utf-8 character)

Probably due to the "range(0x20, "…, why `0x20`?

You're having problems to decode multibytes sequences as you're not having the `else: … result += chr(c[0])` in this case. So typically decoding `\xc2\x20` will raise a `KeyError` as `\x20` is _not_ in your decoding table.

Also, please conform your contribution to the PEP8: you're missing spaces after comas and you're sometime indenting with 8 spaces instead of 4.

I implemented a simple checker based on glibc localedata, it show clearly your decoding problems step by step, and should be easily extended to check for your encoding function too, see attachment. It uses the ISO6937 found typically in the locales debian package or in an 'apt-get sourcee glibc'.

Another comment about coding style: please use \uXXXX hex code representations for the decoding map. The stdlib source code is normally kept ASCII compatible and, for codecs, the Unicode code point numbers make it easier to check the mappings for correctness.

Thanks.

PS: You will also have to sign a contributor agreement: https://www.python.org/psf/contrib/

Just as reference, here's the wikipedia page for the encoding:

https://en.wikipedia.org/wiki/ISO/IEC_6937

and this is the ISO document (as preview):

http://webstore.iec.ch/preview/info_isoiec6937%7Bed3.0%7Den.pdf

(from the German wikipedia page).

iso6937.py:

> # from utf-8 to iso6937
> def iso6937_encode(input,errors,encoding_map):

Wait, is this code for Python 3? Decode from UTF-8 and encode to ISO-6937 in the same function seems strange to me.

I expected that the codec only implements two functions: encode text (unicode) to ISO-6937 (bytes), decode bytes from ISO-6937 to text.

Since the encoding is non trivial (multibyte), if we decide to add it, I suggest to require unit tests. I would like to see unit tests on multibyte strings, to check how the error handler is handled.

--

In general, I would prefer to not embed too many codecs in Python, it has a little cost to maintain these codecs.

My rule is more to only added encodings used (in practice) as locale encodings.

I think the encoder can just use codecs.charmap_encode(). The decoder seems could be simpler too.

Would be nice to generate the ISO 6937 encoding file from external data (e.g. from glibc localedata) like they are generated for other encodings. Take Tools/unicode/ files as a pattern.

Tests are required.

A number of lists of encodings should be updated: Doc/library/codecs.rst, Lib/encodings/aliases.py, Lib/locale.py, Lib/test/test_unicode.py, Lib/test/test_codecs.py, Lib/test/test_xml_etree.py.

@Serhiy: Do you think that the encoding is popular enough to pay the
price of its maintainance?

It's already possible to register manually a new encoding in an
application. It was even already possible in Python 2.7 (and older).

@Serhiy @haypo: Popular enough or not, it may start as a lib on pypi, we'll see its usage from here.

On 14.11.2016 13:03, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> iso6937.py:
> 
>> # from utf-8 to iso6937
>> def iso6937_encode(input,errors,encoding_map):
> 
> Wait, is this code for Python 3? Decode from UTF-8 and encode to ISO-6937 in the same function seems strange to me.

The patch shows the file as UTF-8. In reality, it is decoding from
Unicode strings.

> I expected that the codec only implements two functions: encode text (unicode) to ISO-6937 (bytes), decode bytes from ISO-6937 to text.
> 
> Since the encoding is non trivial (multibyte), if we decide to add it, I suggest to require unit tests. I would like to see unit tests on multibyte strings, to check how the error handler is handled.

+1

> In general, I would prefer to not embed too many codecs in Python, it has a little cost to maintain these codecs.
> 
> My rule is more to only added encodings used (in practice) as locale encodings.

This encoding is used in EPG data of various DVB television
formats. As such it is in active use (even though it is very old).

I think "active use" is a better approach to restricting
ourselves to only locale encodings, since the latter are
slowly converging towards UTF-8 :-)

BTW: Once a charmap style codec is written, there is little
change, so the maintenance is minimal. Codecs which include
more active logic such as this one are different, of course,
and therefore may potentially add more maintenance burden.

> My rule is more to only added encodings used (in practice) as locale encodings.

Just for reference: issue19459, issue21081, issue22679, issue20087.

> @Serhiy: Do you think that the encoding is popular enough to pay the
price of its maintainance?

Yes, it seems to me that the encoding still in use. I found questions about decoding from ISO 6937 and implementations for different programming languages.

Ok. I'm not waiting for a simpler patch reusing existing charmap
functions to see the complexity of the codec ;-)

PEP8 compliant, added missing codepoints, utf-8 to \uXXXX rewrited

@mdk

Big thanks for the checker.

>Looks like your implementation is missing some codepoints, like "\t":
>
>    >>> >print("\t".encode(encoding='iso6937'))                                                    
>    [...]
>    UnicodeError: encoding with 'iso6937' codec failed (UnicodeError: Unacceptable utf-8 character)

The '\t'  character is undefined in the iso6937 table, like all chars within the range 0x00 - 0x1F. I don't know how to handle such input for conversion.

According to https://webstore.iec.ch/preview/info_isoiec6937%7Bed3.0%7Den.pdf:

> NOTE: The shaded positions 00/00 to 01/15 and 07/15 to 09/15 are outside the scope of this International Standard.

So it's clear to me that they are not undefined, they are just described elsewhere.

According to https://en.wikipedia.org/wiki/ISO/IEC_6937:
>ISO/IEC 6937:2001, [...] is a multibyte extension of ASCII

Also, the glibc charmap for ISO_6937 define them:

$ head -n 20 localedata/charmaps/ISO_6937
<code_set_name> ISO_6937
<comment_char> %
<escape_char> /
% version: 1.0
%  source: ECMA registry and ISO/IEC 6937:1992

% alias ISO-IR-156
% alias ISO_6937:1992
% alias ISO6937
CHARMAP
<U0000>     /x00         NULL (NUL)
<U0001>     /x01         START OF HEADING (SOH)
<U0002>     /x02         START OF TEXT (STX)
<U0003>     /x03         END OF TEXT (ETX)
<U0004>     /x04         END OF TRANSMISSION (EOT)
<U0005>     /x05         ENQUIRY (ENQ)
<U0006>     /x06         ACKNOWLEDGE (ACK)
<U0007>     /x07         BELL (BEL)
<U0008>     /x08         BACKSPACE (BS)
<U0009>     /x09         CHARACTER TABULATION (HT)

Finally, if we're not implementing this range, this mean we have _no_ way to encode a new line, which looks highly strange to me, newline being a commonly used character.

But I found _no_ line in the whole ISO/IEC6937 about its ASCII inheritance, I may have just missed it.

If I take the ISO_6937 file as a template for encoding table then
increasing the range 0x20..0x7f to 0x00..0xA0 is the simplest solution.

If I take the ISO_6937 file as a template for encoding table then
increasing the range 0x20..0x7f to 0x00..0xA0 is the simplest solution.

The codec code has a few (performance) issues:

 * nonspacing_diacritical_marks should be a set for fast lookup
 * ord(c) in range(0x00, 0xA0) should be rewritten using < and >=
 * result += bytes([ord(c)]) has exponential timing (it copies
   the whole bytes string for every single operation); better
   use a bytearray and convert this to bytes in one final step
 * the error messages should include more useful information
   about the cause and location of the error, instead of just
   UnicodeError("Unacceptable unicode character") and
   raise KeyError

Please also check whether it's not possible to reuse the charmap codec
functions we have. Thanks.

> Please also check whether it's not possible to reuse the charmap codec functions we have
 I've found nothing useful, maybe you (as the author) can find something really useful which can improve code readability or increase the performance.

Please look at the newest codec version, particularly on line:

tmp += bytearray(encoding_map[c], 'latin1', 'ignore')

It is about extended ascii inheritance. Is it reliable and fast enough?

Please ignore my previous question about:
tmp += bytearray(encoding_map[c], 'latin1', 'ignore')

The latest version don't needs such encoding ...

Performance issue resolved, more info on error added.

I've checked encoding and decoding on a two UTF-8 ~3MiB txt files. Except the first BOM mark (May I ignore it?) all seems work OK.

LGTM, for me it's time to release it as a package on pypi to check the adoption rate and see it it's worth adding it in Python and maybe close this issue.

John: You should probably package this as a pip module alongisde with a git repository, at least to measure qty of interested persones, and get some feedback / contributions.

Would you mind converting this patch to a Github PR John?

For the moment, I'm closing this issue as there's no activity on it I suspect it may no be that usefull.

I may be wrong, so if someone actually needs this, don't hesitate either to put it as a package on PyPI (it should probably go there anyway), either to reopen the issue.

Is there any way to contact John Helour? I would still very much like to put this package on github and pypi. And would like to ask him permission for licensing. Or is there some standard open source license under which all code uploaded to https://bugs.python.org/ can automatically be distributed?

https://www.python.org/about/legal/ seems to indicate so, but doesn't mention an explicit license just the things you can do with it.

Maarten, the code posted on bugs is copyrighted by the person who wrote it. We can only accept it for inclusion in Python after the CLA has been signed, since then we are allowed to relicense it.

As a result you can only take John's code and post it elsewhere, if John permits you to do so, since the files don't include a license.

Note: Creating a character map based codec is not hard using gencodec.py from the Tools/unicode/ dir and perhaps some added extra logic.

Hi Marc-Andre Lemburg,

Thanks for your reply. I tried using gencodec.py as could be downloaded from https://github.com/python/cpython/blob/main/Tools/unicode/gencodec.py as you mentioned. However the code in gencodec.py seems to be in a much worse shape than the iso6937.py attached here. The code in gencodec relies on being able to compare integers with tuples. This is caused by the lines:

mappings = sorted(map)

hinting that this code has never been run using python 3.

providing a decent sort key solves this issue. But after that other issues pop up. For example there seems to be some problems handling the 0x-001  by the not appropriately handling of items in the mapping that have MISSING_CODE resulting in things like:

    0x80: 0x-001

showing up in the generated code.

And then there is the issue that python_mapdef_code has as a side effect that it does 'del map["IDENTITY"]' causing "'IDENTITY' in map" in python_tabledef_code to always evaluate to False even when it should evaluate to True.

The problems above can be observed by just running gencodec.py on https://unicode.org/Public/MAPPINGS/VENDORS/APPLE/SYMBOL.TXT .

If gencodec.py was a trustworthy and well maintained piece of code, I would happily use it. However at the moment I don't see it as a valid option since debugging gencodec.py would cost me at least as much time as just writing its output myself instead of generating it. Additionally https://unicode.org/ doesn't seem to provide a mapping file for iso6937.

I do agree that using codecs.charmap_encode and codecs.charmap_decode is a much better solution then the one in iso6937.py. But I don't understand gencodec.py well enough to actually fix it.

The route via gencodec or more generally via charmap_encode and charmap_decode seems to be one that is not possible without some low level CPython code adjustments. The reason for this is that charmap_encode and charmap_decode only seem to support mappings where a single encoded byte corresponds to multiple unicode points.

However iso6937 is a mixed length encoding, meaning in this specific case that unicode characters sometimes need to be encoded as a single byte and sometimes with two bytes.

For example chr(0x00c0) should be encoded as b'\xc1' + b'A'.

Right, the charmap codec was built with the Unicode Consortium mappings in mind.

However, you may have some luck decoding the two byte chars in ISO 6937 using combining code points in Unicode. With some extra post processing you could also normalize the output into single code points.

If I find time, I may have a look at gencodec.py again and update it to more modern interfaces. I've long given up maintenance of Unicode in Python and only try to help by giving some guidance based on the original implementation design.

Hi Marc-Andre Lemburg,

Thanks for your responses and guidance. At least your pointers to charmap_encode and charmap_decode helped, since it shows at least what the general idea is on how to deal with these types of encodings.

In the mean time I did produce some successes. I wrote some python code that can create character mappings based on the table in http://webstore.iec.ch/preview/info_isoiec6937%7Bed3.0%7Den.pdf so that we can be sure that there are no human errors in generating the mappings.

I think my further approach is to write pure python versions of charmap_encode and charmap_decode that can handle the general case of multi byte encodings to unicode case. This won't be as fast as using the builtins written c. But at least gives maintainable and hopefully reusable code.

Maybe later the c-implementation can be updated as well.